Support user-defined Docker images in Dependency Scanning for Python (gemnasium-python)

Problem to solve

User of Dependency Scanning for Python (gemnasium-python) should be able to change the Docker image that is used when scanning their project, and use one that provides the dependencies their project needs.

Intended users

User experience goal

Proposal

  1. Release the analyzer as a Debian package
  2. Build the Docker image of the analyzer using this distro package
  3. Change the definition of the scanning job so that it installs the distro package when the analyzer CLI is missing
  4. Come up with a list of supported Docker images, test them, and document them

Further details

Implementation plan

  1. Write specs
  • Looking at the FORM instruction of the Dockerfile, identify whether the base image is Debian or Alpine => Debian
  • Decide what Docker images will be officially supported; users will be able to use these image in the image:name of the scanning job
  • Among these Docker images, decide which one is used by default
  • Choose the name of the CI variable that controls the version or URL of the distro package to be fetched by the scanning job
  1. Migrate analyzer project:
  • Build a distro package
    • Extract the commands of the existing Dockerfile, and turn them into a before install script
    • Find the exact command line to be used to create the distro package using fpm
    • Update the CI config to add a CI job that creates a distro package using this command line, and export it as a CI artifact; see #229561 (closed)
  • Build the Docker image using the distro package
    • Change the Dockerfile so that it simply installs the distro package (exported as a CI artifact by a job running in a previous stage) on top of one of the supported Docker images (the "default" image); RUN and ENV commands should be removed
  • Change the job definition to support user-defined images
    • Update the CI config to release the distro package when a git tag is created; see #229561 (closed)
    • In CI template, change the script of the scanning job so that it checks if the analyzer CLI is available, and fetches the MAJOR release (like v2) of the distro package if that's not the case, and installs it; it uses assets links to get the link to the package before downloading it; see #214697 (comment 377750483)
    • Update the CI config and add QA jobs to test the analyzer against various base images; these QA jobs force the image:name of the scanning job of the test projects they trigger, and pass the URL of the temporary distro package built in the pipeline; see #214697 (comment 368098558)
  • Update the documentation
    • Document the ability to change the image:name of the scanning job, and list the images that are officially supported; see #214697 (comment 375762618)
    • Document how this new feature can be used in offline environment

Permissions and Security

N/A

Documentation

The ability to change the image:name of a Dependency Scanning job needs to be documented in Overriding Dependency Scanning jobs and/or Dependency Scanning Analyzers.

The documentation should list the Docker images that are officially supported (images for which compatibility is checked when doing QA). See #214697 (comment 375762618)

Also, the documentation to offline environments should be updated, and give instructions to set up a proxy to serve the distro packages. See #214697 (comment 378274578)

Availability & Testing

Leverage the -FREEZE branches of the existing test projects, and force the image:name of the scanning job when triggering the downstream pipeline, in the CI configuration of the analyzer project. See #214697 (comment 368098558)

What does success look like, and how can we measure that?

Users can override the image:name of the scanning job, and use a Docker image that provides the dependencies needed to build the scanned project and/or install its dependencies.

What is the type of buyer?

GitLab Ultimate

Is this a cross-stage feature?

No, this is specific to one analyzer project.

Links / references

#214697 (comment 376358077)

This page may contain information related to upcoming products, features and functionality. It is important to note that the information presented is for informational purposes only, so please do not rely on the information for purchasing or planning purposes. Just like with all projects, the items mentioned on the page are subject to change or delay, and the development, release, and timing of any products, features, or functionality remain at the sole discretion of GitLab Inc.

Edited by 🤖 GitLab Bot 🤖