### Problem to solve GitLab's dependency scanning relies on lockfiles or graphfiles for accurate dependency detection and vulnerability analysis. Python projects requiring resolution represent approximately **10% of projects with SBOM data in production** - the largest single technology cohort after Maven. ### Proposal Following the outcomes of two internal spikes: - Architecture spike: https://gitlab.com/gitlab-org/gitlab/-/work_items/582607 - Python validation spike: https://gitlab.com/gitlab-org/gitlab/-/work_items/586904 The proposal is further described in the architecture design document: https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/18223 --- ### Manifest support | Manifest | Command | Output | Parser | Notes | |----------|---------|--------|--------|-------| | `requirements.txt` / `.in` / `.pip` / `requires.txt` | `pip-compile` | pip-compile txt | pipcompile | No mutation side effects; all network I/O via system OpenSSL (FIPS-compatible) | | `setup.py` | `pip-compile` | pip-compile txt | pipcompile | | | `setup.cfg` | `pip-compile` | pip-compile txt | pipcompile | Not supported by Gemnasium — new coverage | | `pyproject.toml` (non-Poetry) | `pip-compile` | pip-compile txt | pipcompile | | | `Pipfile` | — | — | — | Deferred to backlog (0.6% market share) | **Validated success rate (spike #586904):** 83% overall across 23 projects; 87.5% for Python 3.10+ projects. Failures are limited to: missing wheels for new Python versions, obsolete build metadata (`use_2to3`), packages with invalid metadata, and APIs removed in Python 3.10+. **Out of scope:** - Pipfile (backlog) - Python \< 3.10 (3.9 EOL Oct 2025; per product decision) - Git/VCS dependencies (`git executable not found`) - Local/editable path dependencies (`-e .`, `file:`, path references) - Poetry without `poetry.lock` when Poetry is not installable ### Python version strategy Use a **single Python version (3.12)** in the resolution job image. Python 3.12 is used (not 3.13) as it is the current stable RHEL9 UBI Python version in the resolution image. Testing showed no meaningful difference in resolution outcomes between 3.12 and 3.13. Users targeting a specific Python version can override the job image via `python_resolution_image` input. Per product decision: Python 3.10+ minimum support only. ### Known parser limitation The DS analyzer's `uv.lock` parser keeps only the first package entry per name, silently discarding entries with different environment markers. This affects any path that produces `uv.lock` output. Paths using `pip-compile` → pip-compile txt (pipcompile parser) are not affected, as pip-compile resolves for a single version/platform. This is an existing limitation in the analyzer, now extended to additional manifest types. Documented in MR !222372+s. Not a blocker for this implementation but should be tracked as a follow-up. ### Known gitlink limitation `pipcompile.lock.txt` is a generated file with no git history of its own. The DS analyzer's gitlink resolver checks which source manifest (`requirements.txt`, `pyproject.toml`, `setup.py`, etc.) is committed in the same directory and reports that as `input_file` in the SBOM. In environments where git is unavailable, the resolver falls back to `pipcompile.lock.txt` itself as the input file. The fix is implemented and unit tested in the analyzer; it activates in production where the DS image includes git. --- ### Implementation plan 1. **Update CI configuration** - V2 Dependency-Scanning CI/CD template: add `dependency-scanning:python-resolution` job - Job triggers on: `**/requirements.txt`, `**/requirements.in`, `**/requirements.pip`, `**/requires.txt`, `**/pyproject.toml`, `**/setup.py`, `**/setup.cfg` - Job image: `ubi9/python-312-minimal` + pip-tools 7.5.3 (image MR open: gitlab-org/security-products/analyzers/dependency-resolution!1) - Artifacts: `**/pipcompile.lock.txt` 2. **Update DS analyzer - Python service mode** - Implement manifest detection logic for Python (routing per manifest type table above) - Implement script generation for each supported path - Support `DS_EXCLUDED_PATHS`, `DS_MAX_DEPTH`, `DS_INCLUDE_DEV_DEPENDENCIES`, `DS_PIPCOMPILE_REQUIREMENTS_FILE_NAME_PATTERN` - Pass through `PIP_INDEX_URL`, `PIP_EXTRA_INDEX_URL` natively (pip-compile reads these as standard pip env vars) - `PIP_CERT` is a native pip env var; pip-compile reads it directly (no mapping needed) - Support `DS_PIP_DEPENDENCY_PATH`: appends `--find-links <path> --no-index` for offline/local wheel resolution - `PIP_*` env vars passed through natively; `UV_*` vars not applicable 3. **Implement integration tests** - Test fixtures covering all manifest types and key scenarios - See Testing plan section 4. **Update documentation** - Supported manifest types and commands - Python version strategy and override instructions - Known limitations (parser, missing wheels) - Env var mapping from Gemnasium equivalents ### Resolved decisions #### 1. `requirements.txt` / `.in`: `pip-compile` vs `uvx migrate-to-uv` **Decision: `pip-compile`** | | `pip-compile` | `uvx migrate-to-uv` | |--|------------------|---------------------| | Output | pip-compile txt | `uv.lock` | | Parser | pipcompile (existing) | uv (existing) | | Side effects | None | Creates/modifies `pyproject.toml` in working dir | | uv.lock parser marker issue | Not affected | Affected | | Originally recommended by | Expert evaluation, spike #586904 | Spike #582607 | `migrate-to-uv` is a project-migration tool, not a resolution primitive. `pip-compile` produces deterministic, platform-resolved output with no side effects. --- #### 2. `pyproject.toml` (non-Poetry): `pip-compile` vs `uv lock` **Decision: `pip-compile`** | | `pip-compile` | `uv lock` | |--|------------------|-----------| | Output | pip-compile txt | `uv.lock` | | Parser | pipcompile (existing) | uv (existing) | | Requires valid `[project]` table | No | Yes | | uv.lock parser marker issue | Not affected | Affected | | Works on pyproject.toml without `[project]` | Yes | No (silent failure or error) | #### 3. Python 3.12 pip-compile image **Decision: `ubi9/python-312-minimal` + pip-tools 7.5.3 (pip-compile)** #### 4b. `pip-compile` vs `uv pip compile` — FIPS compliance **Decision: `pip-compile` (pip-tools)** `uv` bundles its own TLS stack (rustls + aws-lc-rs) and bypasses system OpenSSL entirely, making it incompatible with FIPS 140-2/140-3 requirements. GitLab's FIPS/FedRAMP posture (Dedicated for Government) requires all network I/O to flow through the system OpenSSL FIPS provider. `pip-compile` (pip-tools) is pure Python. All network I/O flows through Python `ssl` → system OpenSSL → FIPS provider on UBI9/RHEL9. No bundled crypto, no Rust. Spike results: pip-compile has **identical pass/fail outcomes** to uv across all tested projects (11/14 pass, 3/14 fail, same root cause). See spike analysis: gitlab-org/gitlab#593859 (note_3214525429 reply). #### 4. `uv.lock` parser fix Not a blocker. All current Python paths produce pip-compile output; the parser limitation does not affect this implementation. Tracked as a separate follow-up. ### Resolved considerations 1. **Partial lockfile presence in monorepos**: resolution is skipped per-directory when a lockfile is already present. 2. **`requirements.txt` lockfile detection**: an existing pip-compile or uv lockfile in `requirements.txt` format is detected; resolution is skipped. 3. **`-c constraints.txt`**: detected in preprocessing; passed as `--constraint` to `pip-compile`. 4. **Local/editable installs**: stripped before resolution; a warning is emitted. 5. **`DS_INCLUDE_DEV_DEPENDENCIES`**: implemented for `pyproject.toml` (`[dependency-groups] dev`); `requirements.txt` has no dev concept; `setup.py`/`setup.cfg` dev extras are out of scope. ## Testing plan ### Manifest types | Test | What it verifies | |------|------------------| | requirements txt happy path | A `requirements.txt` with loose version constraints is compiled to a fully pinned pip-compile output; all transitive dependencies appear in the SBOM. | | requirements in file | An unpinned `.in` file is compiled to a pinned output; the resulting SBOM contains resolved transitive deps not present in the source file. | | setup py | A `setup.py` with an `install_requires` list is resolved via `pip-compile` and produces a correct SBOM. | | setup cfg | A `setup.cfg` with `[options] install_requires` is resolved via `pip-compile`; this represents new coverage not present in Gemnasium. | | pyproject toml PEP 621 | A PEP 621 `pyproject.toml` without `[tool.poetry]` is resolved via `pip-compile` and produces a correct SBOM. | | pyproject toml Poetry | A `pyproject.toml` with `[tool.poetry]` is detected and resolved via `poetry lock && poetry export`; the resulting pip-compile output is scanned. | | uv lock already committed | A pre-committed `uv.lock` causes the resolution job to skip; the DS job scans the existing lockfile directly without re-resolving. | | poetry lock already committed | A pre-committed `poetry.lock` causes the resolution job to skip; this exercises a distinct skip path from the `uv.lock` case. | | requirements txt already pinned | A `requirements.txt` that is already a fully pinned pip-compile lockfile is detected by the analyzer and scanned directly without re-resolution. | | pyproject toml no project table | A `pyproject.toml` with no `[project]` and no `[tool.poetry]` (e.g. build-system config only) is skipped with a logged warning and not passed to `pip-compile`. | | setup py with dynamic requires | A `setup.py` where `install_requires` reads from a file at runtime cannot be resolved statically; the job either fails explicitly with a clear message or falls back to a co-located `requirements.txt` if one is present. | ### Monorepo and multi-manifest | Test | What it verifies | |------|------------------| | multiple requirements files at different depths | Multiple `requirements.txt` files in different subdirectories are each resolved independently; each produces its own SBOM. | | Python and another language in the same repo | A repo containing both a `requirements.txt` and a non-Python manifest (e.g. `pom.xml`) runs Python resolution and language-specific scanning independently without interference. | | one manifest fails and the rest succeed | One manifest in a monorepo triggers a resolution failure (e.g. a missing wheel); the job continues and the remaining manifests are resolved and scanned successfully. | ### Configuration | Test | What it verifies | |------|------------------| | private PyPI registry with credentials | `PIP_INDEX_URL` points to a GitLab-hosted PyPI registry authenticated via `CI_JOB_TOKEN`; resolution succeeds. A second step omits credentials and confirms the expected auth failure. | | custom CA certificate bundle | `PIP_CERT` is mapped to `SSL_CERT_FILE` and a self-signed registry is reachable; a second step omits the cert and confirms the TLS failure. | | subdirectory excluded by DS_EXCLUDED_PATHS | A manifest inside a path matching `DS_EXCLUDED_PATHS` is not resolved and does not appear in the SBOM. | | manifest beyond DS_MAX_DEPTH not resolved | A `requirements.txt` nested deeper than `DS_MAX_DEPTH` is skipped; a shallower sibling in the same repo is resolved normally. | | dev dependencies included and excluded | `DS_INCLUDE_DEV_DEPENDENCIES=true` causes optional dependency groups to appear in the SBOM; `false` excludes them. Tested on a `pyproject.toml` with `[dependency-groups]`. | | resolution image overridden via input | Setting `python_resolution_image` to a non-default image causes the resolution job to run with that image instead of the default. | | resolution disabled via template input | Setting `enable_dependency_resolution: false` removes the resolution job from the pipeline entirely. | | resolution disabled via CI variable | Setting the equivalent CI variable causes the resolution job to be skipped at runtime. | | pip environment variable passthrough | `PIP_INDEX_URL` is set in the project's CI config; resolution succeeds using it, confirming that `PIP_*` env vars are passed through by the service and script generation layer. | ### Requirements.txt edge cases | Test | What it verifies | |------|------------------| | requirements txt with a constraints file | A `requirements.txt` that references a `constraints.txt` via `-c` is compiled with the constraint passed as `--constraint` to `pip-compile`; the resolved versions respect the upper bounds in the constraint file. | | requirements txt with includes | A `requirements.txt` that pulls in another file via `-r` produces a resolved output containing packages from both files. | | requirements txt with an editable install entry | A `requirements.txt` containing a `-e ./local-package` line has that entry skipped with a warning; all other packages in the file are resolved and appear in the SBOM. |