Dependency resolution for python projects in CI pipelines

Problem to solve

GitLab's dependency scanning relies on lockfiles or graphfiles for accurate dependency detection and vulnerability analysis. Python projects requiring resolution represent approximately 10% of projects with SBOM data in production - the largest single technology cohort after Maven.

Proposal

Following the outcomes of two internal spikes:

The proposal is further described in the architecture design document: gitlab-com/content-sites/handbook!18223 (merged)

Manifest support

Manifest Command Output Parser Notes
requirements.txt / .in / .pip / requires.txt pip-compile pip-compile txt pipcompile No mutation side effects; all network I/O via system OpenSSL (FIPS-compatible)
setup.py pip-compile pip-compile txt pipcompile
setup.cfg pip-compile pip-compile txt pipcompile Not supported by Gemnasium — new coverage
pyproject.toml (non-Poetry) pip-compile pip-compile txt pipcompile
Pipfile — — — Deferred to backlog (0.6% market share)

Validated success rate (spike #586904 (closed)): 83% overall across 23 projects; 87.5% for Python 3.10+ projects. Failures are limited to: missing wheels for new Python versions, obsolete build metadata (use_2to3), packages with invalid metadata, and APIs removed in Python 3.10+.

Out of scope:

  • Pipfile (backlog)
  • Python < 3.10 (3.9 EOL Oct 2025; per product decision)
  • Git/VCS dependencies (git executable not found)
  • Local/editable path dependencies (-e ., file:, path references)
  • Poetry without poetry.lock when Poetry is not installable

Python version strategy

Use a single Python version (3.12) in the resolution job image. Python 3.12 is used (not 3.13) as it is the current stable RHEL9 UBI Python version in the resolution image. Testing showed no meaningful difference in resolution outcomes between 3.12 and 3.13. Users targeting a specific Python version can override the job image via python_resolution_image input.

Per product decision: Python 3.10+ minimum support only.

Known parser limitation

The DS analyzer's uv.lock parser keeps only the first package entry per name, silently discarding entries with different environment markers. This affects any path that produces uv.lock output. Paths using pip-compile → pip-compile txt (pipcompile parser) are not affected, as pip-compile resolves for a single version/platform.

This is an existing limitation in the analyzer, now extended to additional manifest types. Documented in MR Document uv.lock limitation in both DS and Gemn... (!222372 - merged) • Zamir Martins • 18.9. Not a blocker for this implementation but should be tracked as a follow-up.

Known gitlink limitation

pipcompile.lock.txt is a generated file with no git history of its own. The DS analyzer's gitlink resolver checks which source manifest (requirements.txt, pyproject.toml, setup.py, etc.) is committed in the same directory and reports that as input_file in the SBOM. In environments where git is unavailable, the resolver falls back to pipcompile.lock.txt itself as the input file. The fix is implemented and unit tested in the analyzer; it activates in production where the DS image includes git.

Implementation plan

  1. Update CI configuration
    • V2 Dependency-Scanning CI/CD template: add dependency-scanning:python-resolution job
    • Job triggers on: **/requirements.txt, **/requirements.in, **/requirements.pip, **/requires.txt, **/pyproject.toml, **/setup.py, **/setup.cfg
    • Job image: ubi9/python-312-minimal + pip-tools 7.5.3 (image MR open: gitlab-org/security-products/analyzers/dependency-resolution!1 (merged))
    • Artifacts: **/pipcompile.lock.txt
  2. Update DS analyzer - Python service mode
    • Implement manifest detection logic for Python (routing per manifest type table above)
    • Implement script generation for each supported path
    • Support DS_EXCLUDED_PATHS, DS_MAX_DEPTH, DS_INCLUDE_DEV_DEPENDENCIES, DS_PIPCOMPILE_REQUIREMENTS_FILE_NAME_PATTERN
    • Pass through PIP_INDEX_URL, PIP_EXTRA_INDEX_URL natively (pip-compile reads these as standard pip env vars)
    • PIP_CERT is a native pip env var; pip-compile reads it directly (no mapping needed)
    • Support DS_PIP_DEPENDENCY_PATH: appends --find-links <path> --no-index for offline/local wheel resolution
    • PIP_* env vars passed through natively; UV_* vars not applicable
  3. Implement integration tests
    • Test fixtures covering all manifest types and key scenarios
    • See Testing plan section
  4. Update documentation
    • Supported manifest types and commands
    • Python version strategy and override instructions
    • Known limitations (parser, missing wheels)
    • Env var mapping from Gemnasium equivalents

Resolved decisions

1. requirements.txt / .in: pip-compile vs uvx migrate-to-uv

Decision: pip-compile

pip-compile uvx migrate-to-uv
Output pip-compile txt uv.lock
Parser pipcompile (existing) uv (existing)
Side effects None Creates/modifies pyproject.toml in working dir
uv.lock parser marker issue Not affected Affected
Originally recommended by Expert evaluation, spike #586904 (closed) Spike #582607

migrate-to-uv is a project-migration tool, not a resolution primitive. pip-compile produces deterministic, platform-resolved output with no side effects.

2. pyproject.toml (non-Poetry): pip-compile vs uv lock

Decision: pip-compile

pip-compile uv lock
Output pip-compile txt uv.lock
Parser pipcompile (existing) uv (existing)
Requires valid [project] table No Yes
uv.lock parser marker issue Not affected Affected
Works on pyproject.toml without [project] Yes No (silent failure or error)

3. Python 3.12 pip-compile image

Decision: ubi9/python-312-minimal + pip-tools 7.5.3 (pip-compile)

4b. pip-compile vs uv pip compile — FIPS compliance

Decision: pip-compile (pip-tools)

uv bundles its own TLS stack (rustls + aws-lc-rs) and bypasses system OpenSSL entirely, making it incompatible with FIPS 140-2/140-3 requirements. GitLab's FIPS/FedRAMP posture (Dedicated for Government) requires all network I/O to flow through the system OpenSSL FIPS provider.

pip-compile (pip-tools) is pure Python. All network I/O flows through Python ssl → system OpenSSL → FIPS provider on UBI9/RHEL9. No bundled crypto, no Rust.

Spike results: pip-compile has identical pass/fail outcomes to uv across all tested projects (11/14 pass, 3/14 fail, same root cause). See spike analysis: #593859 (note_3214525429 reply).

4. uv.lock parser fix

Not a blocker. All current Python paths produce pip-compile output; the parser limitation does not affect this implementation. Tracked as a separate follow-up.

Resolved considerations

  1. Partial lockfile presence in monorepos: resolution is skipped per-directory when a lockfile is already present.
  2. requirements.txt lockfile detection: an existing pip-compile or uv lockfile in requirements.txt format is detected; resolution is skipped.
  3. -c constraints.txt: detected in preprocessing; passed as --constraint to pip-compile.
  4. Local/editable installs: stripped before resolution; a warning is emitted.
  5. DS_INCLUDE_DEV_DEPENDENCIES: implemented for pyproject.toml ([dependency-groups] dev); requirements.txt has no dev concept; setup.py/setup.cfg dev extras are out of scope.

Testing plan

Manifest types

Test What it verifies
requirements txt happy path A requirements.txt with loose version constraints is compiled to a fully pinned pip-compile output; all transitive dependencies appear in the SBOM.
requirements in file An unpinned .in file is compiled to a pinned output; the resulting SBOM contains resolved transitive deps not present in the source file.
setup py A setup.py with an install_requires list is resolved via pip-compile and produces a correct SBOM.
setup cfg A setup.cfg with [options] install_requires is resolved via pip-compile; this represents new coverage not present in Gemnasium.
pyproject toml PEP 621 A PEP 621 pyproject.toml without [tool.poetry] is resolved via pip-compile and produces a correct SBOM.
pyproject toml Poetry A pyproject.toml with [tool.poetry] is detected and resolved via poetry lock && poetry export; the resulting pip-compile output is scanned.
uv lock already committed A pre-committed uv.lock causes the resolution job to skip; the DS job scans the existing lockfile directly without re-resolving.
poetry lock already committed A pre-committed poetry.lock causes the resolution job to skip; this exercises a distinct skip path from the uv.lock case.
requirements txt already pinned A requirements.txt that is already a fully pinned pip-compile lockfile is detected by the analyzer and scanned directly without re-resolution.
pyproject toml no project table A pyproject.toml with no [project] and no [tool.poetry] (e.g. build-system config only) is skipped with a logged warning and not passed to pip-compile.
setup py with dynamic requires A setup.py where install_requires reads from a file at runtime cannot be resolved statically; the job either fails explicitly with a clear message or falls back to a co-located requirements.txt if one is present.

Monorepo and multi-manifest

Test What it verifies
multiple requirements files at different depths Multiple requirements.txt files in different subdirectories are each resolved independently; each produces its own SBOM.
Python and another language in the same repo A repo containing both a requirements.txt and a non-Python manifest (e.g. pom.xml) runs Python resolution and language-specific scanning independently without interference.
one manifest fails and the rest succeed One manifest in a monorepo triggers a resolution failure (e.g. a missing wheel); the job continues and the remaining manifests are resolved and scanned successfully.

Configuration

Test What it verifies
private PyPI registry with credentials PIP_INDEX_URL points to a GitLab-hosted PyPI registry authenticated via CI_JOB_TOKEN; resolution succeeds. A second step omits credentials and confirms the expected auth failure.
custom CA certificate bundle PIP_CERT is mapped to SSL_CERT_FILE and a self-signed registry is reachable; a second step omits the cert and confirms the TLS failure.
subdirectory excluded by DS_EXCLUDED_PATHS A manifest inside a path matching DS_EXCLUDED_PATHS is not resolved and does not appear in the SBOM.
manifest beyond DS_MAX_DEPTH not resolved A requirements.txt nested deeper than DS_MAX_DEPTH is skipped; a shallower sibling in the same repo is resolved normally.
dev dependencies included and excluded DS_INCLUDE_DEV_DEPENDENCIES=true causes optional dependency groups to appear in the SBOM; false excludes them. Tested on a pyproject.toml with [dependency-groups].
resolution image overridden via input Setting python_resolution_image to a non-default image causes the resolution job to run with that image instead of the default.
resolution disabled via template input Setting enable_dependency_resolution: false removes the resolution job from the pipeline entirely.
resolution disabled via CI variable Setting the equivalent CI variable causes the resolution job to be skipped at runtime.
pip environment variable passthrough PIP_INDEX_URL is set in the project's CI config; resolution succeeds using it, confirming that PIP_* env vars are passed through by the service and script generation layer.

Requirements.txt edge cases

Test What it verifies
requirements txt with a constraints file A requirements.txt that references a constraints.txt via -c is compiled with the constraint passed as --constraint to pip-compile; the resolved versions respect the upper bounds in the constraint file.
requirements txt with includes A requirements.txt that pulls in another file via -r produces a resolved output containing packages from both files.
requirements txt with an editable install entry A requirements.txt containing a -e ./local-package line has that entry skipped with a warning; all other packages in the file are resolved and appear in the SBOM.
Edited by Igor Frenkel