If DS_PIP_VERSION is set gemnasium-python silently fails to install the selected version
Summary
Steps to reproduce
- Set up a project with python dependency scanning.
- Trigger the scanning job with a
DS_PIP_VERSION
env var set. - Verify that a warning about
setuptools
is printed andpip
is not installed.
Example Project
https://gitlab.com/madou-stories/pending-deletion/test-python-ds/
What is the current bug behavior?
The installPip
function fails and the analyzer continues without any warning.
What is the expected correct behavior?
The get-pip.py
script downloads and installs the correct version of pip
specified. If it cannot install the version specified, the analyzer should fail loudly or warn loudly. The analyzer does not check the error value for the installPip
function which hides the error. I think this surfaces another problem in our golangci-lint
config - we don't enforce exhaustive error handling.
Relevant logs and/or screenshots
Click to expand
[DEBU] [gemnasium-python] [2023-10-19T14:20:53Z] [/go/src/app/builder/pip/pip.go:95] ▶ /usr/local/bin/python3 /get-pip.py --disable-pip-version-check --no-cache-dir pip==20.3.1
/usr/local/lib/python3.9/site-packages/_distutils_hack/__init__.py:18: UserWarning: Distutils was imported before Setuptools, but importing Setuptools also replaces the `distutils` module in `sys.modules`. This may lead to undesirable behaviors or errors. To avoid these issues, avoid using distutils directly, ensure that setuptools is installed in the traditional way (e.g. not an editable install), and/or make sure that setuptools is always imported before distutils.
warnings.warn(
/usr/local/lib/python3.9/site-packages/_distutils_hack/__init__.py:33: UserWarning: Setuptools is replacing distutils.
warnings.warn("Setuptools is replacing distutils.")
Traceback (most recent call last):
File "/get-pip.py", line 22317, in <module>
main()
File "/get-pip.py", line 198, in main
bootstrap(tmpdir=tmpdir)
File "/get-pip.py", line 120, in bootstrap
import setuptools # noqa
File "/usr/local/lib/python3.9/site-packages/setuptools/__init__.py", line 8, in <module>
import _distutils_hack.override # noqa: F401
File "/usr/local/lib/python3.9/site-packages/_distutils_hack/override.py", line 1, in <module>
__import__('_distutils_hack').do_override()
File "/usr/local/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 77, in do_override
ensure_local_distutils()
File "/usr/local/lib/python3.9/site-packages/_distutils_hack/__init__.py", line 64, in ensure_local_distutils
assert '_distutils' in core.__file__, core.__file__
AssertionError: /usr/local/lib/python3.9/distutils/core.py
Possible fixes
Replace the /get-pip.py
script with a before_script
that downloads the latest version of the script from https://bootstrap.pypa.io/get-pip.py
.
rm -v /get-pip.py
curl -sSL -o /get-pip.py https://bootstrap.pypa.io/get-pip.py
This will ensure that the latest script/binary is installed and works. The drawback to this is that it does not work in offline mode without some configuration.
Implementation plan
- fix error handling https://gitlab.com/gitlab-org/security-products/analyzers/gemnasium/-/blob/cc62dbe8d2ab1829cf270f79e39e68a5eaf6bb2a/builder/pip/pip.go#L67
- update gemnasium-python with latest version of
get-pip.py
- get latest version from pypa
- this will automatically rebuild the docker image with the correct version of the file
- get latest version from pypa
-
document workaround using
before_script
to allow user to get their own version ofget-pip.py
Testing
-
Show previously failing job passing, eg:
# 4.7.0 expected to fail # latest image expected to pass