Add analytics for DAST on the default branch
Problem to solve
We need to validate that DAST is providing value to people when run on the default branch of their repositories.
Intended users
Persona: Product Manager Persona: Development Team Lead
(Only Gitlab's product managers and team leads.)
Further details
This is what we want to measure:
- Number of DAST scans done on default branches in each month. Target = 1000
- Percentage of repos using AutoDevOps that have DAST on the default branch enabled. Target = 50%
- Vulnerabilities found from DAST runs on default branches.
- This will confirm that DAST on the default branch is providing valuable report results that either weren't visible or weren't noticed on the MR DAST results.
Proposal
We will gather the necessary data through Snowplow custom events. We will then create a Periscope dashboard to display the data.
Note: the engineer who works on this issue will need to have edit access for Periscope in order to create the dashboard.
Here is an implementation plan for each item we want to measure:
- Number of DAST scans done on default branches in each month. Target = 1000
Place a custom tracking event in the service for storing security reports that tracks when a DAST report is stored. Storage only happens on the default branch, so every time we receive one of these events it means DAST was run on the default branch. In our tracking syntax, it will look like:
category: Security::StoreReportsService # the class from which the event originates
action: store_dast_report
Note: we'll only get this event if the DAST job completes successfully.
- Percentage of repos using AutoDevOps that have DAST on the default branch enabled. Target = 50%
TBD
- Vulnerabilities found in DAST runs on default branches.
- This will confirm that DAST on the default branch is providing valuable report results that either weren't visible or weren't noticed on the MR DAST results.
This will also use Snowplow's custom events. The event will be fired from the service for creating vulnerabilities from a security report, and will have the following attributes:
category: Security::StoreReportService
action: create_vulnerability_from_dast_report
Open questions
- Is it okay that we only track successful DAST runs on the default branch?
- Do we/should we have documentation for Secure team analytics that lists custom events and links to Secure team Periscope dashboards?
- How will we track the percentage of repos using AutoDevOps but with DAST on the default branch disabled?