ADDITIONAL_CA_CERT_BUNDLE can break gemnasium
Summary
Originally reported - gitlab-org/security-products/analyzers/template#2 (comment 615877815) by @spyderz
Currently my gitlab pipeline is breaking for the gemnasium job. I tried to debug and was able to determine that the environment variable ADDITIONAL_CA_CERT_BUNDLE appears to be causing analyzer to exit with a 128 status. Let me know if there is anything else I can do to debug for you.
docker run --privileged -it --rm --volume "$PWD":/tmp/app --env CI_PROJECT_DIR=/tmp/app registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium:2 sh
/analyzer run
[INFO] [Gemnasium] [2021-06-06T00:39:15Z] ▶ GitLab Gemnasium analyzer v2.29.6
[INFO] [Gemnasium] [2021-06-06T00:39:16Z] ▶ Using commit 47649dd4418405650aaf9f6f605c835da43d7532
of vulnerability database
[INFO] [Gemnasium] [2021-06-06T00:39:16Z] ▶ Cannot auto-remediate dependency file, not supported: composer.lock
export 'ADDITIONAL_CA_CERT_BUNDLE=-----BEGIN CERTIFICATE-----
<cut>
-----END CERTIFICATE-----'
/analyzer run
[INFO] [Gemnasium] [2021-06-06T00:39:48Z] ▶ GitLab Gemnasium analyzer v2.29.6
exit status 128
Steps to reproduce
Pass a valid certificate which doesn't belong to gitlab.com
in the ADDITIONAL_CA_CERT_BUNDLE
variable and run gemnasium
:
$ git clone git@gitlab.com:gitlab-org/security-products/tests/js-npm.git
<snip>
$ docker run -it --rm -v "$PWD/js-npm":/tmp/app \
-e SECURE_LOG_LEVEL=debug \
-e ADDITIONAL_CA_CERT_BUNDLE="$(true | openssl s_client -connect google.com:443 -servername google.com 2>/dev/null | openssl x509)" \
-e CI_PROJECT_DIR=/tmp/app \
registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium:2
[INFO] [Gemnasium] [2021-10-13T09:20:32Z] ▶ GitLab Gemnasium analyzer v2.29.8
...
[DEBU] [Gemnasium] [2021-10-13T09:20:32Z] ▶ /usr/bin/git -C /gemnasium-db remote set-url origin https://gitlab.com/gitlab-org/security-products/gemnasium-db.git
[DEBU] [Gemnasium] [2021-10-13T09:20:32Z] ▶ /usr/bin/git -C /gemnasium-db fetch --force --tags origin master
fatal: unable to access 'https://gitlab.com/gitlab-org/security-products/gemnasium-db.git/': SSL certificate problem: unable to get local issuer certificate
exit status 128
What is the current bug behavior?
The following error is returned when running gemnasium
after configuring ADDITIONAL_CA_CERT_BUNDLE
to any valid certificate which does not belong to gitlab.com
:
fatal: unable to access 'https://gitlab.com/gitlab-org/security-products/gemnasium-db.git/': SSL certificate problem: unable to get local issuer certificate
exit status 128
What is the expected correct behavior?
gemnasium
analyzer should run successfully even if the ADDITIONAL_CA_CERT_BUNDLE
variable has been configured to a certificate which does not belong to gitlab.com
Reason for the bug
The reason for the bug is due to the way git
operates when the ADDITIONAL_CA_CERT_BUNDLE
variable is provided.
When ADDITIONAL_CA_CERT_BUNDLE
is configured, the analyzer writes the contents of this variable to /etc/ssl/certs/ca-cert-additional-gitlab-bundle.pem
when the run
command of the analyzer is executed.
In order for git
to use this CA certificate, we have to do some extra work, which was originally implemented as part of git certificate error with ADDITIONAL_CA_CERT_BUNDLE on alpine containers.
The process we implemented sets the git http.sslCAInfo
configuration option by writing the full path to the certificate to the /etc/gitconfig
file, as shown here:
# contents of /etc/gitconfig
[http]
sslCAInfo = /etc/ssl/certs/ca-cert-additional-gitlab-bundle.pem
Once this has been done, git
will pick up the certificate in this file and use it when initiating SSL connections. This allows us to use a custom CA certificate with git
without requiring any external commands to be executed before running an analyzer, we just need to write the contents of the /etc/gitconfig
configuration file.
This process works well in an offline environment where all GitLab components are self-hosted, and a single self-signed CA certificate is used to connect securely with each service, however, it doesn't work in the situation where some services are self-hosted, and others are not.
For example, in a self-hosted GitLab instance where a pipeline connects to itself to clone the source repo, it can use the self-signed CA cert provided by the ADDITIONAL_CA_CERT_BUNDLE
variable, however, when gemnasium
runs and git
attempts to clone from the gitlab.com
repo with the same self-signed CA cert, the process will fail, because this certificate is invalid for gitlab.com
.
Implementation plan
-
Remove the code from the common package which writes to /etc/gitconfig
that was added as part of git certificate error with ADDITIONAL_CA_CERT_BUNDLE on alpine containers. See here for a draft MR. -
Update the DefaultBundlePath
variable in the common package from/etc/ssl/certs/ca-cert-additional-gitlab-bundle.pem
to/etc/ssl/certs/ca-certificates.crt
-
Make these changes: -
Update the Dockerfile
to ensure that/etc/ssl/certs/ca-certificates.crt
is group writable to maintain OpenShift compatibility. -
Update each project to use the new version of the common library updated in step 2.
and release new versions
To the following projects:
-
Dependency Scanning -
SAST
-
-
Remove the cacert.ImportOptions path override used in the following analyzers: