Python projects have extra dependencies
Summary
In some cases, the dependency list of a Python project contains extra dependencies that do not belong to the project, but belongs to gemnasium-python itself (Dependency Scanning for Python). This happens whenever the scan involves installing the project dependencies, and listing them using pipdeptree. In that case, it's as if the project depends on pipdeptree
, even if it's not the case.
This impacts Python projects using pip or setuptools. Pipenv projects are not impacted by this.
Further details
This problem was surfaced when connecting gemnasium-python to the gemnasium-db, as described in #14630 (closed). The integration required to add some dependencies to the gemnasium-python project, and these were artifically added to the dependencies of the scanned projects. See discussion.
Steps to reproduce
- create a Python project with a
requirements.txt
or aPipfile
but no lock file - enable Dependency Scanning for this project
- inspect the dependency list
Example Project
This was surfaced when working on gemnasium-python!29, see failing pipeline.
Proposal
Use --user
install option for pip
and setuptools
when building projects. And the -u
option when running analysis via pipdeptree
.
This has been successfully tested with both pip
and setuptools
using the following projects:
- https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/master/qa/fixtures/python-setuptools/main/setup.py
- https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python/-/blob/master/qa/fixtures/python-pip/no-pillow/requirements.txt
# python setup.py install --user
# pip install pipdeptree
# pipdeptree -u
test-project==0.0.1
- Django [required: ==1.11.3, installed: 1.11.3]
- pytz [required: Any, installed: 2021.3]
- docutils [required: ==0.13.1, installed: 0.13.1]
- requests [required: ==2.5.3, installed: 2.5.3]
Please note that the project itself is listed, as this is already the case. It's not a regression.
# pip install --user -r requirements.txt
# pip install pipdeptree
# pipdeptree -u
beautifulsoup4==4.6.0
django-contrib-comments==1.8.0
- Django [required: >=1.8, installed: 1.11.4]
- pytz [required: Any, installed: 2018.3]
django-mptt==0.9.0
- django-js-asset [required: Any, installed: 1.0.0]
django-tagging==0.4.6
django-xmlrpc==0.1.8
mots-vides==2015.5.11
pyparsing==2.2.0
regex==2018.2.8
NOTE: Project requirements that match pre-installed packages won't be listed. For instance, currently pip
isn't listed if requirements.txt
contains pip==21.2.4
, because it matches the pre-installed version; nothing gets installed in the user space.
See proof
root@489051b35666:/# pip install pipdeptree
Collecting pipdeptree
Downloading pipdeptree-2.2.1-py3-none-any.whl (21 kB)
Requirement already satisfied: pip>=6.0.0 in /usr/local/lib/python3.6/site-packages (from pipdeptree) (21.2.4)
Installing collected packages: pipdeptree
Successfully installed pipdeptree-2.2.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
root@489051b35666:/# pip install pip --user
Requirement already satisfied: pip in /usr/local/lib/python3.6/site-packages (21.2.4)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
root@489051b35666:/# pip install pip==21.2.4 --user^C
root@489051b35666:/# pipdeptree -u
root@489051b35666:/# pip install pip==21.2.4 --user
Requirement already satisfied: pip==21.2.4 in /usr/local/lib/python3.6/site-packages (21.2.4)
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
root@489051b35666:/# pipdeptree -u
root@489051b35666:/# pip install pip==21.3.1 --user
Collecting pip==21.3.1
Downloading pip-21.3.1-py3-none-any.whl (1.7 MB)
|████████████████████████████████| 1.7 MB 2.5 MB/s
Installing collected packages: pip
WARNING: The scripts pip, pip3 and pip3.6 are installed in '/root/.local/bin' which is not on PATH.
Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed pip-21.3.1
WARNING: Running pip as the 'root' user can result in broken permissions and conflicting behaviour with the system package manager. It is recommended to use a virtual environment instead: https://pip.pypa.io/warnings/venv
WARNING: You are using pip version 21.2.4; however, version 21.3.1 is available.
You should consider upgrading via the '/usr/local/bin/python -m pip install --upgrade pip' command.
root@489051b35666:/# pipdeptree -u
pip==21.3.1
Implementation plan
-
update the pip
andsetuptools
package managers to only install user dependencies via the--user
command -
update pipdeptree
to only show a tree of user installed packages via-u
Testing
We can rely on the existing image tests and job tests. There's no need for new test projects.
-
Integration tests for pip pass. -
Integration tests for setuptools pass.
What is the current bug behavior?
pipdeptree
is in the dependency list
What is the expected correct behavior?
pipdeptree
is NOT in the dependency list
Possible fixes
- Install dependencies with
pip install --user
, and list them withpipdeptree --user
.- Pro: Simple. No need for virtualenv.
- Pro: No need to upgrade to the latest version of
pipdeptree
(though this is something we should do).
- Leverage virtualenv to isolate the dependencies of the scanned projects from the dependencies of gemnasium-python.
- Con: More complex. See gitlab-org/security-products/analyzers/gemnasium!200 (closed)
- Leverage
pipdeptree --exclude
to excludepipdeptree
,pip
, and other package managers. See usage. Warning!* This works with pipdeptree v2 but isn't supported by v1. See comment- Con: We might skip a package that the scanned project explicitly depends on, resulting in a false negative. See comment.
- Con: We would need a CI variable to mitigate that, and we would then have to test, document, and maintain that variable.