NOTE: This epic **needs to be updated**. See https://gitlab.com/groups/gitlab-org/-/epics/8206#note_1754782231 ## Summary Right now the Dependency Scanning jobs (DS) are responsible for both listing the project dependencies, and scanning to report vulnerabilities. This has a negative impact on: - complexity, b/c analyzer projects have too many responsibilities - maintenance, b/c analyzer projects needs to be updated to catch up with any of the package managers they support - flexibility, b/c analyzers projects can only support a limited of package manager versions - performance, b/c DS repeat the detection step, and this step might consume a lot of resources - community contributions, b/c of the complexity To address this, a possible solution is to extract jobs responsible for listing the dependencies, and feed DS the output of these jobs. The "dependency listing" jobs generate SBoM documents, and pass them to DS jobs as CI artifacts. We can see them as "SBoM making" jobs. SBoM making jobs could generate CycloneDX reports that users could directly download and use. It seems sufficient to have 1 DS job, because the SBoM docs passed to the scanning jobs contain all the information needed to perform the scan. DS jobs could merge the multiple SBoM inputs into a single SBoM output, and add vulnerabilities and license information to it, respectively. SBoM making jobs must be executed prior to the `test` stage. One option is to use the `build` stage defined in Auto DevOps. See [Build.gitlab-ci.yml](lib/gitlab/ci/templates/Jobs/Build.gitlab-ci.yml). As a consequence, CI linting fails if the CI template for DS is included but the project pipeline doesn't have a `build` stage. This is a breaking change. Each SBoM making job corresponds to a small project that targets 1 specific language or 1 specific package manager. A SBoM making project can be published as multiple Docker images to cover multiple versions of the tools. For instance, the SBoM maker for Python could be built on top of `python:3.6-slim`, `python:3.9-slim`, `python:3.10-slim`, and published as with the same image tags. This becomes possible because SBoM maker projects are simpler than analyzer projects, and only target one language (or package manager even). ## Challenges - This is an important breaking change. - `build` stage becomes required. - Some existing CI jobs are removed, and users might have configured them in their CI config files. - SBoM making jobs might overlap. For instance, if there's a `pip` specific job and a `setuptools` specific job, then we don't want to trigger the latter if this is a pip project. - SBoM making projects will probably share code. For instance, they should all support `DS_EXCLUDED_PATHS`. - We end up with many projects, and many Docker images. ## Improvements - SBoM generators are very flexible. - Users can provide set up their own SBoM generator jobs. - GitLab can provide SBoM generator images to cover many languages, build tools, version, etc. - DS projects are only about scanning. - They are smaller projects. - They are released less often. - They are easier to grasp, making contributions easier. - They are easier to test. - Scanning jobs do less. - They are easier to debug. - SBoMs can be cached, making possible to rescan the project w/o generating a new SBoM. - Job params can be made more specific. - High timeout for jobs that install dependencies, low timeout for DS jobs. ## Risks - The SBoM passed to the scanning job might miss some information needed to automatically dismiss vulnerabilities as false-positives, or to better assess their severity. - The tools responsible for creating the SBoM might repeat the same code and replicate the same logic. - Having separate tools to generate SBoMs for different languages and package managers might result in discrepancies. ## Implementation Plan 1. [x] [Add sbom command to gemnasium analyzers](https://gitlab.com/gitlab-org/gitlab/-/issues/364622). - [x] https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/merge_requests/341+ 1. [x] [Conditionally run gemnasium child pipelines based on MR title](https://gitlab.com/gitlab-org/gitlab/-/issues/365313). - [x] https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/merge_requests/339+ 1. [x] [Rename CycloneDX files in gemnasium analyzers to follow specification](https://gitlab.com/gitlab-org/gitlab/-/issues/365545). - [x] https://gitlab.com/gitlab-org/security-products/analyzers/integration-test/-/merge_requests/30+ - [x] https://gitlab.com/gitlab-org/gitlab/-/merge_requests/90870+ - [x] https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/merge_requests/346+ 1. [ ] [Add SBOM generator jobs to the Dependency Scanning template](https://gitlab.com/gitlab-org/gitlab/-/issues/365440). 1. [ ] Create individual SBOM generators 1. [x] [Create language-specific SBOM-generator for Go](https://gitlab.com/gitlab-org/gitlab/-/issues/364870). 1. [ ] [Create language-specific SBOM-generator for Ruby](https://gitlab.com/gitlab-org/gitlab/-/issues/365413). 1. [ ] [Create SBOM-generators for languages that use lockfiles](https://gitlab.com/gitlab-org/gitlab/-/issues/365307). 1. [ ] [Create language-specific SBOM-generator for Java & Scala](https://gitlab.com/gitlab-org/gitlab/-/issues/365309). 1. [ ] [Create language-specific SBOM-generator for Python](https://gitlab.com/gitlab-org/gitlab/-/issues/365310). 1. [ ] [Generate CycloneDX SBOM files for vendored libraries in gemnasium](https://gitlab.com/gitlab-org/gitlab/-/issues/361604). 1. [ ] [Add dependency graph information to CycloneDX SBOMs](https://gitlab.com/gitlab-org/gitlab/-/issues/366168) 1. [ ] https://gitlab.com/gitlab-org/gitlab/-/issues/367500+ 1. [ ] [Track Dependency Types in CycloneDX SBOMs](https://gitlab.com/gitlab-org/gitlab/-/issues/367310) 1. [ ] Publish the SBOM generator tools as [Debian distro packages](hhttps://gitlab.com/groups/gitlab-org/-/epics/8320). 1. [ ] Deprecate the `gemnasium*-dependency_scanning` jobs when [Continuous vulnerability scans](https://gitlab.com/groups/gitlab-org/-/epics/7886) has been completed and the rails backend is capable of ingesting the `cyclonedx-*.json` SBOM files and using this data to populate the Vulnerability and Dependency List pages. 1. For %16.0 1. [ ] remove `gemnasium*-dependency_scanning` jobs. 1. [ ] remove scanning features from `gemnasium` codebase or create a fork that only has the ability to generate SBOM files. We need to keep the old codebase for maintenance, to push fixes to older versions of GitLab that still run Dependency Scanning as a collection of CI jobs.