Editable flags can cause gemnasium-python-dependencing_scanning to hang
Summary
The gemnasium python dependency scanner can get "hung up" when executing the pip3 download
step. This occurs in scenarios where users have defined an editable flag targeting the current directory in their requirements.txt
file like so:
somedep==1.0.0
-e .
This should result in a copy of the project in the current directory being zipped and added to the folder defined in the pip3 download --dest
argument. However, this causes issues in some Docker and Kubernetes environments, wherein the zip archive gets stuck infinitely inflating.
This is not specific to the analyzer image, and can be replicated in basic python environment within a Kubernetes cluster or a local Docker container (YMMV). This also doesn't seem to be an officially supported argument for the pip3 download
command. This does not occur when pip3 install
is used.
This does not consistently occur in Docker environments, but does in Kubernetes based environments.
Steps to reproduce
This does not occur consistently on docker executor based runners, but does on Kubernetes based runners.
- For the example project below.
- Configure a Kubernetes based runner.
- Execute the pipeline.
This is specifically occurring when we attempt to run the pip3 download
command within the analyzer. You can replicate this more easily, and completely separated from our analyzer image by using this deployment manifest.
- Run:
kubectl apply -f deployment.yaml
- Exec into the pod:
kubectl exec -ti python bash
- Attempt download:
pip3 -vvv download --disable-pip-version-check --dest ./dist -r requirements.txt
Pip will continuously inflating the project archive within the dist
directory until disk space is completely filled, or the command is interrupted. You can observe the size of the testing-1.0.zip
archive increase relative to how long you allow the command to execute.
Example Project
https://gitlab.com/calebw/editable_gemnasium_test
What is the current bug behavior?
Pip attempts to archive the project indefinitely, with the archive infinitely inflating until disk space is full.
What is the expected correct behavior?
Pip properly archives the project, or we remove this line from the requirements.txt
file before initiating the pip3 download
command within the scanner.
Possible fixes
Ignoring lines within a requirements.txt
file that begin with -e
may be a valid approach. I don't think this will have a bearing on dependency scanning results, and the bug behavior is ultimately not GitLab specific.
Implementation Plan
Unfortunately ignoring the -e
in the requirements.txt
file will introduce complexity and we might end up in edge cases (see #451279 (comment 1945255573)). This whole issue seems to be a pip3
and not a Gemnasium problem. For that reason we will document this case.
-
Document this case in the troubleshooting section of the docs