### Problem to solve The current Gemnasium Python analyzer supports a single python version (python 3.11) but many underlying build commands for various package manager and build tools, creating significant maintenance overhead. ### Proposal **Objective** Validate that a single modern Python version (e.g. 3.13) coupled with the `uv` build tool can successfully generate dependency list or graphs for various python projects which might originally target other python version and package managers. **Hypothesis** Using `uv` as a single tool can handle dependency resolution for the majority of Python projects (requirements.txt, Pipfile, setup.py, pyproject.toml) with acceptable accuracy, while significantly reducing maintenance burden, as stated in the [spike proposal](https://gitlab.com/gitlab-org/gitlab/-/work_items/582607#note_2934063311). There are different build commands available depending on the files present in the project: | Input File | `uv pip compile` | `uvx migrate-to-uv` | `uv lock` | |------------|------------------|---------------------|-----------| | `requirements.txt` | ✅ Works | ✅ Works → `uv.lock` | ❌ Requires pyproject.toml | | `requirements.in` | ✅ Works | ✅ Works → `uv.lock` | ❌ Requires pyproject.toml | | `setup.py` | ✅ Works | ❌ Not supported | ❌ Requires pyproject.toml | | `pyproject.toml` | ✅ Works | ❌ Not supported | ✅ Works → `uv.lock` | | `Pipfile` | ❌ Parse error | ✅ Works → `uv.lock` | ❌ Requires pyproject.toml | The recommanded approach from the spike is as follow: - `requirements.txt`, `Pipfile`, → `uvx migrate-to-uv` → generates a `uv.lock`file → uses the `uv.lock` parser - `setup.py`, `pyproject.toml` → `uv pip compile` → `requirements.txt` → pipcompile's `requirements.txt` lockfile parser **Experiment Design** 1. **Test Matrix** - Test `uv` against various Python project types: - `requirements.txt` (pip projects) - `Pipfile` (pipenv projects) - `setup.py` (setuptools projects) - `pyproject.toml` (modern Python projects) - Validate dependency resolution accuracy across Python versions (e.g. 3.10, 3.11, 3.12, 3.13) - Compare results with native tool outputs (pip, pipenv, setuptools) 2. **Test Projects Selection Criteria** - Include projects with: - Complex dependency trees - Environment markers (`python_version`, `sys_platform`) - Dev dependencies - Package extras - Version constraints (ranges, exclusions) 3. **Python Image Selection Criteria** - The PoC should evaluate and document the rationale for which image we select based on Distribution, image size, FIPS compliance and licenssing. **Test Commands** ```bash # For requirements.txt/Pipfile - migrate to uv.lock uvx migrate-to-uv # For setup.py/pyproject.toml - compile to pip-compile format uv pip compile setup.py -o compiled.txt # Include dev dependencies uv pip compile pyproject.toml --group dev -o compiled.txt ``` **Success Criteria** >1. [ ] Acceptable success rate (>95%?) across diverse Python projects Based on the test results: 83% achieved (19/23 projects) with the following comments: - Modern projects (Python 3.8+): ~95% excluding wheel availability issues - Failures: legacy Python versions, missing wheels, ancient dependencies However as per @joelpatterson [comments](https://gitlab.com/gitlab-org/gitlab/-/issues/582607#note_2930879800): >* _Language Version:_ > * _Minimum - support v3.10+ (v3.9 is EOL Oct. 2025)_ Based on the above the success rate for python 3.10+ would be 87.5%. > 2. [x] Documented compatibility matrix for all input file types Done. Tested 5 manifest types (requirements.txt, Pipfile, setup.py, pyproject.toml, poetry.lock) and results documented with baseline and native tool comparisons. See [this thread](https://gitlab.com/gitlab-org/gitlab/-/issues/586904#note_3025028880) for more details. > 3. [x] Validated environment marker resolution with correct Python versions Done. UV handles markers correctly, but both Gemnasium and DS analyzer parser limitation identified (only keeps first package entry, ignores conditionals). [This MR](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/222372) adds clarification to the documentation. > 4. [ ] Performance benchmarks compared to native tools Formal performance hasn't been recorded, however UV does not seems to slower than the native tools. > 5. [x] Documented list of failure patterns and edge cases Done. Python ≤3.7 unsupported, missing wheels, obsolete build tools, parser limitations. See [this thread](https://gitlab.com/gitlab-org/gitlab/-/issues/586904#note_3025028880) for more details. > 6. [x] Recommendation on Python version support strategy There doesn't seem to be a considerable difference between python 12 and 13. Keeping in mind the EOL for each version, I would suggest version 13. **Known Risks to Validate** | Risk | Mitigation | Status | |------|------------|--------| | Different resolution algorithm than native tools | Document differences, validate accuracy | To validate | | Git/VCS dependencies not fully supported | Document limitations, provide fallback | To validate | | Python version auto-download doesn't work for `uv pip compile` | Pre-install required Python versions | Confirmed | **Expected Outcomes** If successful, this PoC will enable: - **Unified tool**: Single tool (`uv`) instead of managing pip, pipenv, setuptools - **Easier maintenance**: public docker image - **Faster resolution**: 10-100x faster than native tools - **Reduced maintenance**: Single output format to handle - **Existing parser support**: We already have parsers for `uv.lock` and pip-compile formats **Considerations going foward** - Document setup.py support limitations as setup.py projects need Python 3.8+ and potential migration path for older projects.