Secure - Metrics - SaaS - Identify Fixed Finding, with scanner, within MR
Problem to solve
Right now we have only very rudimentary metrics about number of jobs run and would like more granularity in our telemetry system. I, as a Product Manager, would like more granularity and visibility into the number of findings, and fixes, that are occurring during our scans. This will help show if we experience sudden changes, and also show if people are acting on our findings.
Implementation Plan
Scope: Saas users, self hosted users are out of scope
Categories: All of Secure, except fuzzing, if possible
-
Add a hook to CompareSecurityReportsService
to calculate the difference of vulnerabilities between theprevious_report
and thelatest_report
-
Send a single backend snowplow/ruby event using Gitlab::Tracking.self_describing_event
with the schema created in https://gitlab.com/gitlab-org/iglu/-/blob/master/public/schemas/com.gitlab/secure_context/jsonschema/1-0-0
Payload should include following information (please check latest schema from iglu repo
{
"data_sources": null,
"pipeline_id": 123,
"merge_request_id": 29,
"project_id": 465,
"user_id": null,
"scanner": {[
"type": "dependency_scanning",
"id": "gemnasium-maven",
"version": "2.18.0"
]
},
"branch_type": "feature",
"vulnerabilities": {
"diff": {
"added": 100,
"fixed": 50,
"existing": 10
},
"severities_added": {
"critical": 4,
"high": 40,
"medium": 23,
"low": 0,
"info": 0,
"unknown": 0
}
}
}
- "data_sources": null, -> will be null for now
- "user_id": null, -> will be null for now
- "scanner": { }, -> will be null for now, this needs to be converted to array
- "branch_type": "feature", -> always feature for now
- "severities_added": { } -> these numbers are belong do added vulnerabilities we want to rename this to
severities_added
Documentation
Please ask ~"group::telemetry" what documentation we may need to do
Please document your experience of deciding a plan, setup, implement, and testing this (gdocs if needed) to share with all groups trying to do similar we can ask ~"group::telemetry" and Technical Writing where this documentation belongs
Availability & Testing
-
Standard unit tests in ruby codebase, integration tests are not currently possible because they're currently blocked by Setup Tracking QA/Testing Environment using Snowplow Mini -
Manual tests using dashboard created in Secure - Metrics - SaaS - Create SCA Dashboards
What does success look like, and how can we measure that?
Data can be seen in the dashboards implemented by Secure - Metrics - SaaS - Create SCA Dashboards
Is this a cross-stage feature?
Yes, we should be trying to cover Category:SAST Category:DAST ~"Category:Dependency Scanning" Category:Container Scanning Category:Secret Detection if it is in the same code area and we can just change it easily to send a different "scan type" name/variable.
We are NOT covering ~"Category:License Compliance" here b/c it's not "findings" and "fixes"
If not we need to provide documentation and create issues for implementation of non groupcomposition analysis items - Category:SAST Category:DAST Category:Secret Detection
Extra Information
Personas are described at https://about.gitlab.com/handbook/marketing/product-marketing/roles-personas/User experience goal
Users don't notice
Product Managers have a dashboard to view this data
Permissions and Security
There should be no changes to permissions or security, this is handled by ~"group::telemetry"