Customer POC metrics for Secure

Problem to solve

Suggest POC metrics that refocus security teams away from total vulnerabilities found and more onto the value of finding vulns in dev. / the value of security embedded into CI (areas where GitLab can shine).

Some of these GL can measure today; some we cannot. Let's invite the customers to work with us and shape the metrics GL can provide.

Intended users

Security professionals

Further details

Proposal

Available metrics:

• Vulnerabilities, by project, remaining upon merge acceptance (found in security dashboard) + also trended over time.
• Vulnerabilities found in the MR (ones resolved will not show up in security dashboard)
• Auto remediated vulnerabilities
• Percent of code scanned (e.g. 100% of code using GL CI with scanning, tracked by customer outside of GL)

Suggested metrics that are needed:

• Vulnerability remediation issues created
• Vulnerabilities found and remediated before merge with master branch - if code was scanned during commits we could get immediate feedback to the coder
• Accuracy of the results via Youden index (https://www.owasp.org/index.php/Benchmark) - a single value that combines sensitivity (based on True Positives) and specificity (based on False Positives).
• Number of dismissed findings as percent of total (ideally "by reason for dismissal" to separate true false positives from compensating controls - it's ok to dismiss risks based on specific security controls)

Potential ideas that require more thought on how to measure:

• Tracking of residual risk, (perhaps where exceptions to policies were granted?)
• Comparing residual risk to the risk threshold,
• Summary comparing total risk/accepted (vulns moved to production or escaped bugs?) risk/offloaded (?) risk/to-be-mitigated risk (issues created?).
• how security controls are performing against discovered vulnerabilities.
• Dollars saved by identifying vulns earlier in the dev life cycle?
• Categorizing where vulnerabilities are detected.

Permissions and Security

Documentation

Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Links / references