Dependency scanning image is missing required libraries
Summary
I'm trying to setup a Gitlab instance with Dependency Scanning, but the code I'm trying to analyse depends on psycopg2.
The script snippet looks like this:
- docker run
--env DEP_SCAN_DISABLE_REMOTE_CHECKS="${DEP_SCAN_DISABLE_REMOTE_CHECKS:-false}"
--volume "$PWD:/code"
--volume /var/run/docker.sock:/var/run/docker.sock
"registry.gitlab.com/gitlab-org/security-products/dependency-scanning:$SP_VERSION" /code/
And I can see how Gitlab is pulling the image, then starting to install the libraries in my requirements.txt, but when it gets to the point where it should install psycopg2, it fails because it can't find pg_config.
$ export SP_VERSION=$(echo "$CI_SERVER_VERSION" | sed 's/^\([0-9]*\)\.\([0-9]*\).*/\1-\2-stable/')
$ docker run --env DEP_SCAN_DISABLE_REMOTE_CHECKS="${DEP_SCAN_DISABLE_REMOTE_CHECKS:-false}" --volume "$PWD:/code" --volume /var/run/docker.sock:/var/run/docker.sock "registry.gitlab.com/gitlab-org/security-products/dependency-scanning:$SP_VERSION" /code/
Unable to find image 'registry.gitlab.com/gitlab-org/security-products/dependency-scanning:11-7-stable' locally
11-7-stable: Pulling from gitlab-org/security-products/dependency-scanning
6dee6a11e61b: Pulling fs layer
6dee6a11e61b: Verifying Checksum
6dee6a11e61b: Download complete
6dee6a11e61b: Pull complete
Digest: sha256:bac83170c80e91c4d0132748e03248492808b463794cae51760218a80cbe2580
Status: Downloaded newer image for registry.gitlab.com/gitlab-org/security-products/dependency-scanning:11-7-stable
2019/02/11 00:49:27 Copy project directory to containers
2019/02/11 00:49:27 [bundler-audit] Detect project using plugin
2019/02/11 00:49:27 [bundler-audit] Project not compatible
2019/02/11 00:49:27 [gemnasium] Detect project using plugin
2019/02/11 00:49:27 [gemnasium] Project not compatible
2019/02/11 00:49:27 [gemnasium-maven] Detect project using plugin
2019/02/11 00:49:27 [gemnasium-maven] Project not compatible
2019/02/11 00:49:27 [gemnasium-python] Detect project using plugin
2019/02/11 00:49:27 [gemnasium-python] Project is compatible
2019/02/11 00:49:27 [gemnasium-python] Starting analyzer...
2: Pulling from gitlab-org/security-products/analyzers/gemnasium-python
cd784148e348: Pulling fs layer
a5ca736b15eb: Pulling fs layer
f320f547ff02: Pulling fs layer
2edd8ff8cb8f: Pulling fs layer
9381128744b2: Pulling fs layer
a2a81dd1e4fb: Pulling fs layer
797b4f744fac: Pulling fs layer
9381128744b2: Waiting
a2a81dd1e4fb: Waiting
797b4f744fac: Waiting
2edd8ff8cb8f: Waiting
a5ca736b15eb: Verifying Checksum
a5ca736b15eb: Download complete
cd784148e348: Verifying Checksum
cd784148e348: Download complete
cd784148e348: Pull complete
2edd8ff8cb8f: Verifying Checksum
2edd8ff8cb8f: Download complete
9381128744b2: Verifying Checksum
9381128744b2: Download complete
f320f547ff02: Verifying Checksum
f320f547ff02: Download complete
a5ca736b15eb: Pull complete
a2a81dd1e4fb: Verifying Checksum
a2a81dd1e4fb: Download complete
797b4f744fac: Verifying Checksum
797b4f744fac: Download complete
f320f547ff02: Pull complete
2edd8ff8cb8f: Pull complete
9381128744b2: Pull complete
a2a81dd1e4fb: Pull complete
797b4f744fac: Pull complete
Digest: sha256:75ce781b990a7a26ecde5ab1d1b59ed01adf2c5c35fbb622d21ef5f92b0b6001
Status: Downloaded newer image for registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium-python:2
Found project in /tmp/app
Collecting redis==2.10.6 (from -r requirements.txt (line 1))
Downloading https://files.pythonhosted.org/packages/3b/f6/7a76333cf0b9251ecf49efff635015171843d9b977e4ffcf59f9c4428052/redis-2.10.6-py2.py3-none-any.whl (64kB)
Collecting decorator==4.2.1 (from -r requirements.txt (line 2))
Downloading https://files.pythonhosted.org/packages/e1/5a/53db15bf367d2028bdc6700dbdf1bdfab46b9f208b7516952817c0808118/decorator-4.2.1-py2.py3-none-any.whl
Collecting Django==2.0.4 (from -r requirements.txt (line 3))
Downloading https://files.pythonhosted.org/packages/89/f9/94c20658f0cdecc2b6607811e2c0bb042408a51f589e5ad0cb0eac3236a1/Django-2.0.4-py3-none-any.whl (7.1MB)
Collecting djangorestframework==3.8 (from -r requirements.txt (line 4))
Downloading https://files.pythonhosted.org/packages/8b/5f/20900529b5849b446b47382b7095fba81383af6240e731ce6cd50f4e0a68/djangorestframework-3.8.0-py2.py3-none-any.whl (923kB)
Collecting django-filter==2.0.0 (from -r requirements.txt (line 5))
Downloading https://files.pythonhosted.org/packages/6a/8b/8517167a0adc45ce94d0873efb9487dd4cdeff7e10f96e837ad3d58f5837/django_filter-2.0.0-py3-none-any.whl (69kB)
Collecting markdown==2.6.11 (from -r requirements.txt (line 6))
Downloading https://files.pythonhosted.org/packages/6d/7d/488b90f470b96531a3f5788cf12a93332f543dbab13c423a5e7ce96a0493/Markdown-2.6.11-py2.py3-none-any.whl (78kB)
Collecting properties==0.3.3 (from -r requirements.txt (line 7))
Downloading https://files.pythonhosted.org/packages/81/39/3d7dac15b04d8389c64396ad2788d59df184720e88029e466e53c02ad6bd/properties-0.3.3.tar.gz
Collecting psycopg2-binary==2.7.7 (from -r requirements.txt (line 16))
Downloading https://files.pythonhosted.org/packages/dd/56/c22da10f5a725d61c58a185ec0f803aa2d384646ee8eb83d8ce88ed5edb1/psycopg2-binary-2.7.7.tar.gz (428kB)
Complete output from command python setup.py egg_info:
running egg_info
creating pip-egg-info/psycopg2_binary.egg-info
writing pip-egg-info/psycopg2_binary.egg-info/PKG-INFO
writing dependency_links to pip-egg-info/psycopg2_binary.egg-info/dependency_links.txt
writing top-level names to pip-egg-info/psycopg2_binary.egg-info/top_level.txt
writing manifest file 'pip-egg-info/psycopg2_binary.egg-info/SOURCES.txt'
Error: pg_config executable not found.
pg_config is required to build psycopg2 from source. Please add the directory
containing pg_config to the $PATH or specify the full executable path with the
option:
python setup.py build_ext --pg-config /path/to/pg_config build ...
or with the pg_config option in 'setup.cfg'.
For further information please check the 'doc/src/install.rst' file (also at
<http://initd.org/psycopg/docs/install.html>).
----------------------------------------
Command "python setup.py egg_info" failed with error code 1 in /tmp/pip-install-99sq6mw_/psycopg2-binary/
You are using pip version 18.1, however version 19.0.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
2019/02/11 00:49:32 exit status 1
2019/02/11 00:49:33 Container exited with non zero status code
The problem is obvious. It can't find a binary, which lives inside a package named libpq-dev. The solution is not that obvious thou. I don't know how to install that library in the image.
Any idea how can I make the dependency scanner to install libpq-dev?
Steps to reproduce
- Create a new Django project that depends on psycopg2
- Add DS to the
.gitlab-ci.yml - Run a pipeline
What is the current bug behavior?
The DS fails because it can't find some required libraries
What is the expected correct behavior?
The DS should be able to complete it's task
Output of checks
This bug happens on GitLab.com
Possible fixes
A possible fix would be to, somehow, allow extra libraries to be installed before running the DS.