Runner fails SSL verification when cloning repo from a Gitlab host that uses a custom certificate authority
Summary
When using a certificate that is from an untrusted/unknown issuer (in our case, Let's Encrypt's staging environment CA) to provide SSL for the Gitlab web host, the runner is unable to clone a repository from that host, even when the entire certificate chain is saved to /etc/gitlab-runner/certs/<servername>.crt
and the runner is able to register with the host without any errors.
I see a few similar issues have been reported before, but all the ones I've seen are from a year ago or longer, and have since been closed without a clear resolution as fr as I can tell
Steps to reproduce
I'm not sure if this will work for everyone's configuration, but:
- Obtain a staging certificate from Let's Encrypt (or possibly any untrusted issuer, such as s self-signed cert?)
- Setup Gitlab using the omnibus installer and use the staging certificate for SSL (on a machine I'll call the gitlab host)
- Create a new project in the new Gitlab instance with a .gitlab-ci.yml that will run something (e.g. rspec in our case)
- Setup a gitlab-ci-multi-runner with docker (on a machine I'll call the runner host)
- Copy the entire Let's Encrypt certificate chain to
/etc/gitlab-runner/certs/<gitlab-hostname>.crt
on the runner host (this will allow you to register the runner despite the invalid SSL on the gitlab host) - run
gitlab-runner register
to register the runner with the gitlab host - Push an update to the project so the CI pipeline runs the configured job
Actual behavior
The pipline runs with the following output (indicating that it was not able to clone the project repo due to a missing issuer certificate):
Running with gitlab-ci-multi-runner 9.4.1 (d24b11c)
on joes-runner (1b5875db)
Using Docker executor with image satorix/rails ...
Using docker image sha256:b191deea88ca4e9809f09fe03bed7be640b3bbecde1ac7583025d0c299ea8ae9 for predefined container...
Pulling docker image satorix/rails ...
Using docker image satorix/rails ID=sha256:50bb2c0edbec7e66d821cf1e7225961c812edf705025e44f6392379b0d3e2b99 for build container...
Running on runner-1b5875db-project-2-concurrent-0 via chickenbone...
Cloning repository...
Cloning into '/builds/satorix/bamble-snork'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxxxxxxxxxxxxxxxx@gitlab.widgets-inc.satorix-staging.satorix.com/satorix/bamble-snork.git/': SSL certificate problem: unable to get issuer certificate
ERROR: Job failed: exit code 1
Expected behavior
Because the certificate chain has been copied to /etc/gitlab-runner/certs
as described in gitlab-ci-multi-runner's Custom Certification Authorities documentation, it should verify the certificate using the provided data, clone the repo and run the test.
Relevant logs and/or screenshots
I'm not sure how to get the "runner log" - if someone can tell me how to do that (and if it would be helpful), please let me know!
Update: Runner Logs!
Environment description
docker info:
docker info
Containers: 61
Running: 1
Paused: 0
Stopped: 60
Images: 25
Server Version: 17.05.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 446
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.4.0-87-generic
Operating System: Ubuntu 16.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 30.39GiB
Name: chickenbone
ID: 5SFF:C5H5:Z3MT:ESUW:AMBQ:OTPG:BDMC:OEHH:U4WL:7RI5:TM4F:AY2O
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Used GitLab Runner version
Version: 9.4.1
Git revision: d24b11c
Git branch: 9-4-stable
GO version: go1.8.3
Built: Tue, 25 Jul 2017 12:04:47 +0000
OS/Arch: linux/amd64