Add dependency resolution job for python to DS CI template v2
What does this MR do and why?
Add a dependency Resolution job for python to the Dependency Scanning CI template v2 that attempts to automatically generate a pipcompile.lock.txt file for a project using various python requirements files: requirements.txt, requirements.pip, requires.txt, setup.py, setup.cfg, pyproject.toml
Python projects may not have a lockfile commited to their repository. Accurate dependency scanning requires one, which today users must generate themselves via a custom CI job. This creates friction and makes it impossible to enforce scanning policies at scale.
The solution relies on coordinating a main dependency resolution job container with the DS analyzer running as a service container:
- The resolution job checks out the repository and signals readiness by creating a
.gitlab_ds/ready_${CI_JOB_ID}file - The service container detects projects needing resolution and writes a shell script to
.gitlab_ds/dependency_resolution.sh - The resolution job executes the script, which runs the appropriate resolution command per project (e.g.
uv pip compilefor Python), producing graph files - Those files are exported as artifacts and picked up by the main dependency scanning job that runs in a later stage.
This MR replicates the dependency resolution mechanism implementated for Maven in Add dependency resolution job for maven to DS C... (!226990 - merged)
References
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
- checkout this branch in your GDK
- clone this project: https://gitlab.com/gitlab-org/security-products/tests/python-pip
- adjust CI configuration to just use v2 template:
include: - template: Jobs/Dependency-Scanning.v2.gitlab-ci.yml - Run a pipeline and confirms that:
- a
dependency-scanning:python-resolutionjob ran in.prestage, emitting apipcompile.lock.txtfile - the
dependency-scanningjob ran and successfully scanned this lock file.
- a
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.