Use Security::Scan model to track usage data of security scans

Problem to solve

With the upcoming removal of DinD &2462 (closed) for security products and the orchestration layer for SAST and Dependency Scanning, we will no longer have the sast and dependency_scanning jobs running in the pipeline, but instead will have 1 job per analyzer, with a different name.

The current usage ping is relying on these job name and thus, we'll lose track of this data. This is also true for any 3rd party analyzer who will use a custom job name.

We will keep existing keys because,

we still need to count this way for LC
we want to keep counting this way for Security jobs to know how many are still running old CI config

Intended users

devopssecure team

Further details

Proposal

Replace the existing usage ping logic to leverage the new Security::Scan records instead of counting jobs with expected name.

Please note that this would only apply to security reports, License Compliance report are not stored in that table and still need to rely on counting license_management and license_scanning jobs.

Implementation plan

scan_type in Secure::Scans will only have 1,2,3,4,5,6 . In other words analysers will be map to values 1..6. Due to logic in StoreScanService

Check out telemetry guide
Add Gitlab::UsageData#usage_activity_by_stage_secure increment count for each pipeline running security jobs. For example, following query will give you how many pipelines are using dependency scanning.

Counting each pipeline that run dependency scanning job

SELECT tmp.scan_type, count(tmp.commit_id)
FROM (
    SELECT
        ci_builds.commit_id, security_scans.scan_type
    FROM
        security_scans
    INNER JOIN
        ci_builds
            ON ci_builds.id = security_scans.build_id
            AND ci_builds.type = 'Ci::Build'
    WHERE
        (
            ci_builds.status = 'success'
            AND (
                ci_builds.retried= 'false'
                OR  ci_builds.retried IS NULL
            )AND security_scans.scan_type=2 AND  security_scans.created_at BETWEEN '2020-06-14 00:00:00' AND '2020-06-14 23:59:59' 
        )
    GROUP  BY  ci_builds.commit_id, security_scans.scan_type
) AS tmp
GROUP BY tmp.scan_type

We should use following new keys,

old key	new key
container_scanning_jobs	container_scanning_pipelines
dependency_scanning_jobs	dependency_scanning_pipelines
secrets_sast	secret_detection_pipeliness
sast_jobs	sast_pipelines
dast_jobs	dast_pipelines
	coverage_fuzzing_pipelines

Update documentation in https://docs.gitlab.com/ee/development/telemetry/usage_ping.html to mention new keys

Use Security::Scan model to track usage data of security scans

Problem to solve

Intended users

Further details

Proposal

Implementation plan

Counting each pipeline that run dependency scanning job

Permissions and Security

Documentation

Availability & Testing

What does success look like, and how can we measure that?

What is the type of buyer?

Is this a cross-stage feature?

Links / references