Detection job to control images of the Dependency Scanning jobs

Release notes

Problem to solve

A Dependency Scanning job shouldn't fetch and decompress a big Docker image when we can predict that it's going to fail or generate an empty report. This is a waste of resources.

Further details

In some cases Dependency Scanning jobs are triggered because the git repo being scanned matches the rules:exists parameter of the scanning job BUT the dependency files aren't supported by the scanner OR they've been skipped using CI variables like DS_EXCLUDED_PATHS. This is a waste of resources especially in the case of Java projects because the gemnasium-maven image is big (1.5GB):

  • We waste bandwidth (and some time) when fetching the image.
  • We waste time/CPU when decompressing that image.

Proposal

  • Update the Dependency Scanning (DS) CI template
    • Introduce new CI variables that control the Docker image used by each DS job
    • Set default values so that the CI template is backward compatible
  • Add a new detect subcommand to the DS analyzers
  • Create a "no scan" Docker image
    • The image is based on a very small image like alpine:latest (3MB).
    • It responds to /analyzer run and shows a warning, saying that the job has been skipped.
  • Introduce a new DS detection job in the predefined .pre stage
    • Have a new Docker image for this job, and add the analyzer binaries to that image.
    • Make the detection job rely on the detect subcommand of the analyzers to know what's supported
    • Use dotenv reports and set CI variables so that a scanning job uses the "no scan" image when the corresponding detect subcommand fails

See Proof of Concept: gitlab-org/security-products/tests/ruby-bundler!1272 (closed)

NOTE: Ideally scanning jobs that aren't fully compatible or that result in an empty report should be skipped altogether, as suggested in #336977 (closed). This is more difficult to implement though, so in a first iteration we might decide to change the base image (this proposal).

#336977 (closed) contains relevant implementation details since the proposal is similar.

Follow-up issues

  • We can leverage this approach to use the Python 3.9 variant of the gemnasium-python if we're able to detect that this is a Python 3.9 project.
  • We could split the gemnasium-maven image and build images that only support one build tool (Maven, Gradle or Sbt), one version of Java (8, 11, 14, 15, etc.), etc.
  • We could implement &3923 (closed), and select the official Docker image that seems most compatible with the scanned repo, like gradle:7.1.1-jdk11 if we detect that this a Gradle 7.1.1 project using JDK 11.
Edited by Fabien Catteau