Skip to content

ADDITIONAL_CA_CERT_BUNDLE can break gemnasium

Summary

Originally reported - gitlab-org/security-products/analyzers/template#2 (comment 615877815) by @spyderz

Currently my gitlab pipeline is breaking for the gemnasium job. I tried to debug and was able to determine that the environment variable ADDITIONAL_CA_CERT_BUNDLE appears to be causing analyzer to exit with a 128 status. Let me know if there is anything else I can do to debug for you.

docker run --privileged -it --rm --volume "$PWD":/tmp/app --env CI_PROJECT_DIR=/tmp/app registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium:2 sh
/analyzer run
[INFO] [Gemnasium] [2021-06-06T00:39:15Z] ▶ GitLab Gemnasium analyzer v2.29.6
[INFO] [Gemnasium] [2021-06-06T00:39:16Z] ▶ Using commit 47649dd4418405650aaf9f6f605c835da43d7532
 of vulnerability database

[INFO] [Gemnasium] [2021-06-06T00:39:16Z] ▶ Cannot auto-remediate dependency file, not supported: composer.lock
export 'ADDITIONAL_CA_CERT_BUNDLE=-----BEGIN CERTIFICATE-----
<cut>
-----END CERTIFICATE-----'
/analyzer run
[INFO] [Gemnasium] [2021-06-06T00:39:48Z] ▶ GitLab Gemnasium analyzer v2.29.6
exit status 128

Steps to reproduce

Pass a valid certificate which doesn't belong to gitlab.com in the ADDITIONAL_CA_CERT_BUNDLE variable and run gemnasium:

$ git clone git@gitlab.com:gitlab-org/security-products/tests/js-npm.git

<snip>

$ docker run -it --rm -v "$PWD/js-npm":/tmp/app \
   -e SECURE_LOG_LEVEL=debug \
   -e ADDITIONAL_CA_CERT_BUNDLE="$(true | openssl s_client -connect google.com:443 -servername google.com 2>/dev/null | openssl x509)" \
   -e CI_PROJECT_DIR=/tmp/app \
   registry.gitlab.com/gitlab-org/security-products/analyzers/gemnasium:2

[INFO] [Gemnasium] [2021-10-13T09:20:32Z] ▶ GitLab Gemnasium analyzer v2.29.8
...
[DEBU] [Gemnasium] [2021-10-13T09:20:32Z] ▶ /usr/bin/git -C /gemnasium-db remote set-url origin https://gitlab.com/gitlab-org/security-products/gemnasium-db.git

[DEBU] [Gemnasium] [2021-10-13T09:20:32Z] ▶ /usr/bin/git -C /gemnasium-db fetch --force --tags origin master
fatal: unable to access 'https://gitlab.com/gitlab-org/security-products/gemnasium-db.git/': SSL certificate problem: unable to get local issuer certificate

exit status 128

What is the current bug behavior?

The following error is returned when running gemnasium after configuring ADDITIONAL_CA_CERT_BUNDLE to any valid certificate which does not belong to gitlab.com:

fatal: unable to access 'https://gitlab.com/gitlab-org/security-products/gemnasium-db.git/': SSL certificate problem: unable to get local issuer certificate

exit status 128

What is the expected correct behavior?

gemnasium analyzer should run successfully even if the ADDITIONAL_CA_CERT_BUNDLE variable has been configured to a certificate which does not belong to gitlab.com

Reason for the bug

The reason for the bug is due to the way git operates when the ADDITIONAL_CA_CERT_BUNDLE variable is provided.

When ADDITIONAL_CA_CERT_BUNDLE is configured, the analyzer writes the contents of this variable to /etc/ssl/certs/ca-cert-additional-gitlab-bundle.pem when the run command of the analyzer is executed.

In order for git to use this CA certificate, we have to do some extra work, which was originally implemented as part of git certificate error with ADDITIONAL_CA_CERT_BUNDLE on alpine containers.

The process we implemented sets the git http.sslCAInfo configuration option by writing the full path to the certificate to the /etc/gitconfig file, as shown here:

# contents of /etc/gitconfig
[http]
        sslCAInfo = /etc/ssl/certs/ca-cert-additional-gitlab-bundle.pem

Once this has been done, git will pick up the certificate in this file and use it when initiating SSL connections. This allows us to use a custom CA certificate with git without requiring any external commands to be executed before running an analyzer, we just need to write the contents of the /etc/gitconfig configuration file.

This process works well in an offline environment where all GitLab components are self-hosted, and a single self-signed CA certificate is used to connect securely with each service, however, it doesn't work in the situation where some services are self-hosted, and others are not.

For example, in a self-hosted GitLab instance where a pipeline connects to itself to clone the source repo, it can use the self-signed CA cert provided by the ADDITIONAL_CA_CERT_BUNDLE variable, however, when gemnasium runs and git attempts to clone from the gitlab.com repo with the same self-signed CA cert, the process will fail, because this certificate is invalid for gitlab.com.

Implementation plan

  1. Remove the code from the common package which writes to /etc/gitconfig that was added as part of git certificate error with ADDITIONAL_CA_CERT_BUNDLE on alpine containers. See here for a draft MR.

  2. Update the DefaultBundlePath variable in the common package from /etc/ssl/certs/ca-cert-additional-gitlab-bundle.pem to /etc/ssl/certs/ca-certificates.crt

  3. Make these changes:

    • Update the Dockerfile to ensure that /etc/ssl/certs/ca-certificates.crt is group writable to maintain OpenShift compatibility.
    • Update each project to use the new version of the common library updated in step 2. and release new versions

    To the following projects:

  4. Remove the cacert.ImportOptions path override used in the following analyzers:

Edited by Adam Cohen