Skip to content

git fails in gitlab-runner with LetsEncrypt cert - SSL certificate problem: unable to get local issuer certificate

Summary

Running gitlab CE 10.4.2 docker image, with gitlab-runner:alpine-v10.3.1 on another machine and LetsEncrypt certificate generated on an outside system and copied to /etc/gitlab/ssl (mounted into docker image)

Pipeline jobs fail cloning repository

Cloning into '/builds/D2D/Zope-in-Docker'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@myhost.com:2223/D2D/Zope-in-Docker.git/': SSL certificate problem: unable to get local issuer certificate
ERROR: Job failed: exit code 1

Note that git has no problems cloning gitlab repos when run outside of gitlab-runner and https access to gitlab also works correctly without any certificate issues.

I believe the problem is with the CI_SERVER_TLS_CA_FILE that is pushed to gitlab-runner.

I have tried changing the gitlab ssl certificate from cert.pem to fullchain.pem, that did not help. I then appended the LetsEncrypt root certificate to fullchain.pem and used that as the ssl certificate for gitlab (after gitlab-ctl reconfigure AND docker-compose down, up to be sure the previous ssl cert had not been cached)

git in gitlab-runner still fails, but https access to gitlab works fine.

Running gitlab-runner in debug mode (docker run .. --debug run) I see that the CI_SERVER_TLS_CA_FILE that's being passed to git-lab runner is ONLY the first certificate in the gitlab ssl certificate.

The gitlab ssl cert (as mentioned above) is fullchain.pem + LE root cert, so there are 3 certificates in the same file. However gitlab-runner is only getting the first certificate from the file.

Therefore git, which might otherwise accept LE certs, fails to work because it's forced to use only CI_SERVER_TLS_CA_FILE via this statement:

git' "config" "--global" "http.https://code.sfi.ca:2223.sslCAInfo" "/build/D2D/Zope-in-Docker.tmp/CI_SERVER_TLS_CA_FILE"

I suspect git would work if the entire certificate file were to sent from gitlab to gitlab-runner instead of just the first certificate in the file.

gitlab-runner debug output fragments

echo -n $\'-----BEGIN CERTIFICATE-----\\nMIIE+TCCA+GgAwIBAgISA/wGXA1z7wy+bKLPeXTPy3RfMA0GCS
<snip>
kySa+BaH8ZWPA7YKCAb/U=\\n-----END CERTIFICATE-----\\n\' > "/builds/D2D/Zope-in-Docker.tmp/CI_SERVER_TLS_CA_FILE

current gitlab server ssl certificate contains 3 certificates

-----BEGIN CERTIFICATE-----
MIIE+TCCA+GgAwIBAgISA/wGXA1z7wy+bKLPeXTPy3RfMA0GCSqGSIb3DQEBCwUA
<snip>
pzuBZylUKG4t8hoXH+kySa+BaH8ZWPA7YKCAb/U=
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
<snip>
KOqkqm57TH2H3eDJAkSnh6/DNFu0Qg==
-----END CERTIFICATE-----
-----BEGIN CERTIFICATE-----
MIIFazCCA1OgAwIBAgIRAIIQz7DSQONZRGPgu2OCiwAwDQYJKoZIhvcNAQELBQAw
<snip>
emyPxgcYxn/eR44/KJ4EBs+lVDR3veyJm+kXQ99b21/+jh5Xos1AnX5iItreGCc=
-----END CERTIFICATE-----

note that the CI_SERVER_TLS_CA_FILE file sent by gitlab to gitlab-runner only contains the first certificate.

Steps to reproduce

  • Acquire a certificate from LE but not using the gitlab-ce letsencrypt settings
  • install the certificate in gitlab
  • setup gitlab-runner on docker, using docker-in-docker (DIND) execution method
  • create a pipeline and run it
  • git clone fails

What is the current bug behavior?

git clone fails because the ssl certificate sent by gitlab to gitlab-runner isn't trusted by curl.

What is the expected correct behavior?

git clone should work

Relevant logs and/or screenshots

Job log..

Waiting for services to be up and running...
Using docker image sha256:2eff3125ba05d4e070eef45124ed0d4b8eda1f7d701ba05610f01d8d0d828e1c for predefined container...
Pulling docker image registry.myothercompany.com/sfi1/docker-with-compose:201802121541 ...
Using docker image registry.myothercompany.com/sfi1/docker-with-compose:201802121541 ID=sha256:8f89f55f14aca37ac71ae57dbab41847c02a4924cebe15ba6a5ecaa6ee2bf6ae for build container...
Running on runner-08651858-project-6-concurrent-0 via docker1802a.mycompany.com...
Cloning repository...
Cloning into '/builds/D2D/Zope-in-Docker'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@code.myothercompany.com:2223/D2D/Zope-in-Docker.git/': SSL certificate problem: unable to get local issuer certificate
ERROR: Job failed: exit code 1

gitlab-runner log .. please let me know if you really need this.

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 
gitlab-rake gitlab:env:info

System information
System:
Current User:   git
Using RVM:      no
Ruby Version:   2.3.6p384
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.3.0
Redis Version:  3.2.11
Git Version:    2.14.3
Sidekiq Version:5.0.5
Go Version:     unknown


GitLab information
Version:        10.4.2
Revision:       b1c501c
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://code.sfi.ca:2223
HTTP Clone URL: https://code.sfi.ca:2223/some-group/some-project.git
SSH Clone URL:  ssh://git@code.sfi.ca:2222/some-group/some-project.git
Using LDAP:     no
Using Omniauth: no


GitLab Shell
Version:        5.11.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
gitlab-rake gitlab:check SANITIZE=true                                                                                                                                  [33/4664]
Checking GitLab Shell ...


GitLab Shell version >= 5.11.0 ? ... OK (5.11.0)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
3/1 ... ok
2/2 ... ok
3/3 ... ok
3/4 ... ok
3/5 ... ok
3/6 ... ok
3/7 ... ok
3/9 ... ok
3/10 ... ok
7/11 ... ok
3/12 ... ok
7/13 ... ok
5/14 ... ok
2/15 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


LDAP is disabled in config/gitlab.yml


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
3/1 ... yes
2/2 ... yes
3/3 ... yes
3/4 ... yes
3/5 ... yes
3/6 ... yes
3/7 ... yes
3/9 ... yes
3/10 ... yes
7/11 ... yes
3/12 ... yes
7/13 ... yes
5/14 ... yes
2/15 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.6)
Git version >= 2.7.3 ? ... yes (2.14.3)
Git user has default SSH configuration? ... yes
Active users: ... 10


Checking GitLab ... Finished

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)