Detection job to control the execution of Dependency Scanning jobs

Summary

Introduce a new Dependency Scanning detection job that controls the execution of the Dependency Scanning jobs, and skip these jobs when the git repo isn't compatible or when the supported files are excluded.

Proposal

NOTE: The rails backend so that it can process Security reports created in child pipelines. #215725 (comment 636132045)

• Change the Dependency Scanning template so that it only runs the detection job by default
• Make the detection job create a child pipeline where compatible, non-excluded analyzers are triggered

The child pipeline uses something equivalent to the existing CI template: it runs the scanning job but doesn't feature the detection job.

There are different ways the detection job can alter the child pipeline:

• Set DS_EXCLUDED_ANALYZERS and export it using dotenv reports.
  • Make the child pipeline run the DS CI template w/o the detection job. Jobs are skipped based on DS_EXCLUDED_ANALYZERS. To be checked.
  • Or use DS_EXCLUDED_ANALYZERS to generate a CI config that is included in the child pipeline.
• Or directly generate a CI config that only enables the relevant scanning jobs.

The detection job could rely on the analyzers to exclude the corresponding jobs that are supported but not excluded:

• We specify a CLI subcommand (like detect) that takes a target directory and returns 0 if supported.
• We change all DS analyzers so that they support this new subcommand.
• We create a new Docker image for the detection job. That images includes the binaries of each supported analyzer.
• When running the detection job, the detection script runs the "detect" command and add the analyzer name if it exists with a non-zero exit code.

Improvements

The main benefits are:

• Save resourceful by skipping scanning jobs that are normally executed.
• Improve UX by better explaining why a scanning job is skipped.

Currently a scanning job is triggered if its rules:exists parameter matches what's in the git repo even if:

• the matching files are explicitly excluded in DS_EXCLUDED_PATHS; see #218734 (closed)
• the version of the dependency file isn't supported; the filename doesn't say the exact file format that is

The detection job could also be the place where a scanning job is skipped because the files it would scan haven't changed since the previous, and the cache can be reused. TODO: create a specific issue for this

Risks

Increased complexity due to the child pipeline?

This might be harder to test unless security reports of the child pipeline appear as reports of the parent pipeline. See #215725 (comment 636132045)

Also, we need to update image used in the detection job whenever the detection logic changes. (But the CI template become more stable, as a result.)

TODO

Involved components

• Dependency Scanning CI template
• All Dependency Scanning analyzers

Optional: Intended side effects

If there are a significant number of scanning jobs being skipped, then this should also reduce the traffic between the Runners and the Docker registry. We would fetch an extra image to run the detection job, but it would only contains self-contained binaries, possibly on top of Alpine, so it would be really small.

Optional: Missing test coverage

To be tested in integration tests that trigger a project pipeline.