Skip to content

Update the stable and latest CI templates for Dependency Scanning to use the new DS analyzer instead of Gemnasium

Olivier Gonzalezrequested to merge
501103-update_DS_latest_CI_template_to_use_new_DS_analyzer into master

What does this MR do and why?

This MR updates both the stable and latest CI templates for Dependency Scanning to transition to Dependency Scanning using SBOM and the introduction of the new DS analyzer.

The rollout strategy has changed with Adjust rollout strategy for the new DS analyzer... (#517653 - closed) and this MR content updated accordingly.

See previous implementation

It update the latest CI template Dependency-Scanning.latest.gitlab-ci.yml by following these requirements:

  • Don't try to use the CI/CD component (see details in this thread)
  • introduce a new env var ENABLE_GEMNASIUM_DEPENDENCY_SCANNING_ENGINE with default value set to false, which allow to revert the CI template to its previous behavior.
  • Keep all existing gemnasium jobs defined but update them so that:
    • a new rule prevents the job from being triggered unless ENABLE_GEMNASIUM_DEPENDENCY_SCANNING_ENGINE is set to true
    • the script exits with 1 and outputs and error message unless ENABLE_GEMNASIUM_DEPENDENCY_SCANNING_ENGINE is set to true. We do this to ensure that if a user has overriden the rules from our template, the job will still be triggered but its execution will fail.
  • introduce a new dependency-scanning job that doesn't inherit existing configuration.
    • Add a rule to prevent the job from being triggered if ENABLE_GEMNASIUM_DEPENDENCY_SCANNING_ENGINE is set to true.
    • Add rules to trigger the job for all the file supported by both the new DS analyzer and the old Gemnasium analyzers. For files that are not supported by the new DS analyzer, this is done on purpose so that the analyzer runs and fail with a warning message (see Ensure the new DS analyzer handles its executio... (#509826 - closed)). Otherwise there will be no DS job running at all and this change might go un-noticed.
    • do not react to CI_GITLAB_FIPS_MODE. There is a single image for both FIPS and non FIPS modes for the new DS analyzer.

It updates the stable CI template Dependency-Scanning.gitlab-ci.yml by applying the same changes as in the latest template above, but:

  • set the ENABLE_GEMNASIUM_DEPENDENCY_SCANNING_ENGINE default value to true (instead of false).
  • do not update the script of the old gemnasium jobs, simply add the rules to disable them unless ENABLE_GEMNASIUM_DEPENDENCY_SCANNING_ENGINE is set to true

This approach ensures that by default, the CI configuration for the stable template is the same as before this MR, so existing users won't see any change. Though, this offers them the ability to easily opt-in to the new CI configuration and proactively try out the change that will happen automatically in 18.0. Indeed, in 18.0 we'll change the default value of ENABLE_GEMNASIUM_DEPENDENCY_SCANNING_ENGINE to false so that by default the template uses the new DS analyzer.

Introduce a new CI/CD variable DS_ENFORCE_NEW_ANALYZER which is set to false by default. Use this variable to control the jobs execution as follow:

  • Run gemnasium-maven on the same conditions as today, unless DS_ENFORCE_NEW_ANALYZER is set to true.
  • Run gemnasium-python on the same conditions as today, unless DS_ENFORCE_NEW_ANALYZER is set to true.
  • Run gemnasium (main) on the same conditions as today, unless DS_ENFORCE_NEW_ANALYZER is set to true.
  • Run DS analyzer:
    • if DS_ENFORCE_NEW_ANALYZER is set to true.
    • OR if the repository contains any file that is only supported by the new DS analyzer, but ensure to not scan anything already supported by the gemnasium analyzers using DS_EXCLUDED_PATHS.

This approach ensures that all existing customers stay with the Gemnasium analyzer by default and it enables automatically the new DS analyzer for its newly supported files only. And even if the new DS analyzer is triggered in a monorepository, it won't scan any file related to projects already handled by the Gemnasium analyzers.

As existng users wish to migrate to the new DS analyzer they can set DS_ENFORCE_NEW_ANALYZER to true (at the project, group, or instance level).

Here are the types of projects that are only supported by the new DS analyzer and for which this analyzer will run automatically (when DS_ENFORCE_NEW_ANALYZER is set to false):

  • C/C++/Fortran/Go/Python/R projects using conda (conda-lock.yml).
  • Objective-C projects using cocoapods (Podfile.lock).
  • Rust projects using Cargo (Cargo.lock).
  • Swift projects using Swift (Package.resolved).

Users who want to entirely prevent the use of the new DS analyzer must set the CI/CD variable DS_EXCLUDED_ANALYZERS to "dependency-scanning".

This change can be applied in 17.9 and users who want to try out the breaking changes before 18.0 can do so by setting DS_ENFORCE_NEW_ANALYZER to true.

References

Please include cross links to any resources that are relevant to this MR. This will give reviewers and future readers helpful context to give an efficient review of the changes introduced.

MR acceptance checklist

Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Screenshots or screen recordings

Screenshots are required for UI changes, and strongly recommended for all other merge requests.

Before After

How to set up and validate locally

Numbered steps to set up and validate the change are strongly suggested.

  1. Clone this test project with several compatible files and that include the DS template from this MR. (with the most recent commit sha,at the time of writing) 
     include:
   - remote: https://gitlab.com/gitlab-org/gitlab/-/raw/f69f4b90d8cf23ca5a1174a663ca56cfd7dc28d3/lib/gitlab/ci/templates/Jobs/Dependency-Scanning.latest.gitlab-ci.yml
  2. Verify the CI/CD pipeline is running the expected jobs and they generate the expected SBOM reports.

Here are some tests made on gitlab.com:

  1. Stable template - Default configuration: https://gitlab.com/gitlab-org/secure/tests/olivier/ds-deprecation-rollout/-/pipelines/1668911276
  2. Stable template - Enforce the new DS analyzer: https://gitlab.com/gitlab-org/secure/tests/olivier/ds-deprecation-rollout/-/pipelines/1668913780
  3. Latest template - Default configuration: https://gitlab.com/gitlab-org/secure/tests/olivier/ds-deprecation-rollout/-/pipelines/1668916082
  4. Latest template - Enforce the new DS analyzer: https://gitlab.com/gitlab-org/secure/tests/olivier/ds-deprecation-rollout/-/pipelines/1668918864
Edited by Olivier Gonzalez

Merge request reports