Customer POC metrics for Secure
Problem to solve
Suggest POC metrics that refocus security teams away from total vulnerabilities found and more onto the value of finding vulns in dev. / the value of security embedded into CI (areas where GitLab can shine).
Some of these GL can measure today; some we cannot. Let's invite the customers to work with us and shape the metrics GL can provide.
- Vulnerabilities, by project, remaining upon merge acceptance (found in security dashboard) + also trended over time.
- Vulnerabilities found in the MR (ones resolved will not show up in security dashboard)
- Auto remediated vulnerabilities
- Percent of code scanned (e.g. 100% of code using GL CI with scanning, tracked by customer outside of GL)
Suggested metrics that are needed:
- Vulnerability remediation issues created
- Vulnerabilities found and remediated before merge with master branch - if code was scanned during commits we could get immediate feedback to the coder
- Accuracy of the results via Youden index (https://www.owasp.org/index.php/Benchmark) - a single value that combines sensitivity (based on True Positives) and specificity (based on False Positives).
- Number of dismissed findings as percent of total (ideally "by reason for dismissal" to separate true false positives from compensating controls - it's ok to dismiss risks based on specific security controls)
Potential ideas that require more thought on how to measure:
- Tracking of residual risk, (perhaps where exceptions to policies were granted?)
- Comparing residual risk to the risk threshold,
- Summary comparing total risk/accepted (vulns moved to production or escaped bugs?) risk/offloaded (?) risk/to-be-mitigated risk (issues created?).
- how security controls are performing against discovered vulnerabilities.
- Dollars saved by identifying vulns earlier in the dev life cycle?
- Categorizing where vulnerabilities are detected.