Count how many security jobs ran by scan type

Problem to solve

This is issue was part of #211621 (closed). We separated to another issue to reduce complexity. We would like to count how many job run by per scan type. For example,

Let's say we imported JS project and enabled DS. Pipeline is running for the first time with RetireJs and Gemnasium. Query in issue description will give us following result,

Scan Type Count
2 2

User re-runs the job

Scan Type Count
2 2

User runs pipeline for master again

Scan Type Count
2 4

Usage data will look like this

{
  `dependency_scans`: 4 
}

We need to finish #211621 (closed) first to see if we need to add any type of migrations to improve the performance

Intended users

devopssecure team

Further details

Proposal

Replace the existing usage ping logic to leverage the new Security::Scan records instead of counting jobs with expected name.

Please note that this would only apply to security reports, License Compliance report are not stored in that table and still need to rely on counting license_management and license_scanning jobs.

Implementation plan

scan_type in Secure::Scans will only have 1,2,3,4,5,6 . In other words analysers will be map to values 1..6. Due to logic in StoreScanService

 -- Each successful non repeated dependency scanning job
explain SELECT security_scans.scan_type,
       Count(security_scans.scan_type)
FROM   security_scans
       INNER JOIN ci_builds
               ON ci_builds.id = security_scans.build_id
                  AND ci_builds.type = 'Ci::Build'
      WHERE  (security_scans.scan_type=2 AND ci_builds.status = 'success'  AND (ci_builds.retried= 'false' OR  ci_builds.retried IS NULL) ) AND security_scans.created_at BETWEEN '2020-06-14 00:00:00' AND '2020-06-14 23:59:59'
GROUP  BY security_scans.scan_type;

Counting each pipeline that run dependency scanning job

We should use following new keys,

New Keys
container_scanning_scans
dependency_scanning_scans
sast_scans
dast_scans
secret_detection_scans
coverage_fuzzing_scans

Permissions and Security

Documentation

Availability & Testing

This change looks to be backend/metrics based rather than functionality. Engineer shall ensure that DB entries are populated when implemented.

What does success look like, and how can we measure that?

What is the type of buyer?

Is this a cross-stage feature?

This will impact:

Links / references

Edited by Will Meek