Dependency Scanning of Pipfile.lock without installing project dependencies
Problem to solve
Dependency Scanning for Python (gemnasium-python) can scan pipenv projects and report vulnerabilities for the exact versions listed in Pipfile.lock. However, it requires to install the project dependencies, and this consumes resources (time and bandwidth), and might fail if the environment of the scanning job doesn't meet all the requirements of the installed packages (version of python and system libraries).
Proposal
Parse Pipfile.lock directly, without running pipenv graph to export the dependency graph.
As a side effect, it becomes possible to scan a Pipenv project that doesn't have a Pipfile.
Warning! Pipfile.lock only contains normalized package names, like django, and not canonical names, like Django. Parsing the lock file might result in inaccurate package names. That said, vulnerabilities are accurately reported, thanks to PEP 426 support implemented in #33341 (closed).
Further details
gemnasium-python runs pipenv graph --json to export the dependency graph as a JSON document. This JSON doc is similar to the one pipdeptree creates. pipenv graph installs the project dependencies, and this might fail if the installed packages require a specific version of Python, or specific system libraries.
What does success look like, and how can we measure that?
- Less failing Dependency Scanning jobs for Python projects, more Python projects being scanned.
- Less time spent scanning Pipenv project
Links / references
Implementation plan
-
Add Pipfile.lockparser to gemnasium #33227 (closed) -
Enable Pipfile.lockparser ingemnasium-python, and make sure thepipenvbuilder is skipped when a lock file is provided -
Add existsline to detection rule forPipfile.lockin the Dependency Scanning template -
Update user docs to mention support
Testing
This is testing using test branch https://gitlab.com/gitlab-org/security-products/tests/python-pipenv/-/commits/pipfile-lock-FREEZE and the corresponding QA job.