SSL certificate problem: unable to get issuer certificate
I'm having a similiar issues as #334 (closed).
I'm running Gitlab EE 8.3.4 on RHEL 6.x server, along with the 1.0.0 Beta GitLab Multi Runner (as was suggested upgrading in #334 (closed)).
I'm behind a corporate proxy, which I did have issues with getting Docker to connect out to download images, but resolved this by adding the following to the /etc/sysconfig/docker file for testing purposes:
export HTTP_PROXY='http://<username>:<password>@proxy.domain.com'
export HTTPS_PROXY='http://<username>:<password>@proxy.domain.com'I've since run into same issue in #334 (closed), where git doesn't want to clone properly. I've created a new issue here since in my case, we have our own internal CA which has created the certificates for our internal domains (e.g. gitlab.domain.com).
Reviewing the CI Build in teh Gitlab UI, here is what is listed:
gitlab-ci-multi-runner 1.0.0~beta.14.g5f37bc5 (5f37bc5)
Using Docker executor with image centos:6 ...
Pulling docker image mysql:latest ...
Starting service mysql:latest ...
Pulling docker image postgres:latest ...
Starting service postgres:latest ...
Pulling docker image redis:latest ...
Starting service redis:latest ...
Pulling docker image mongo:latest ...
Starting service mongo:latest ...
WARNING: Service mysql is already created. Ignoring.
Waiting for services to be up and running...
Pulling docker image gitlab/gitlab-runner:service ...
Pulling docker image gitlab/gitlab-runner:service ...
Pulling docker image gitlab/gitlab-runner:service ...
Pulling docker image gitlab/gitlab-runner:service ...
Pulling docker image gitlab/gitlab-runner:build ...
Pulling docker image centos:6 ...
Running on runner-6822dcd4-project-11-concurrent-0 via <server>...
Cloning repository...
Cloning into '/builds/<username>/test'...
fatal: unable to access 'https://gitlab-ci-token:xxxxxx@gitlab.domain.com/<username>/test.git/': SSL certificate problem: unable to get issuer certificate
ERROR: Build failed with: exit code 1Now, it took me a bit of digging around the docs, but I stopped the gitlab-multi-runner service and launched it in debug mode to see what else was happening here:
(<server>)# /sbin/service gitlab-runner stop
Stopping GitLab Runner: [ OK ]
(<server>)# gitlab-runner --debug run
INFO[0000] Starting multi-runner from /etc/gitlab-runner/config.toml ... builds=0
DEBU[0000] Feeding runners to channel builds=0
DEBU[0000] Starting worker 0 builds=0
DEBU[0000] Checking runner <server> url=https://gitlab.domain.com/ci token=<token>
DEBU[0000] Trying to load /etc/ssl/certs/ca-bundle.crt ...
DEBU[0000] Checking for builds... nothing runner=6822dcd4
DEBU[0003] Feeding runners to channel builds=0
DEBU[0003] Checking runner <server> url=https://gitlab.domain.com/ci token=<token>
DEBU[0003] Checking for builds... nothing runner=6822dcd4Then I initiated a CI rebuild for the project.
Here is the gitlab-multi-runner config:
(<server>)# cat /etc/gitlab-runner/config.toml
concurrent = 1
[[runners]]
url = "https://gitlab.domain.com/ci"
token = "<token>"
tls-skip-verify = false
tls-ca-file = "/etc/ssl/certs/ca-bundle.crt"
name = "<server>"
executor = "docker"
environment = ["MYSQL_ALLOW_EMPTY_PASSWORD=1"]
[runners.docker]
image = "centos6"
privileged = false
volumes = ["/cache"]
services = ["mysql:latest", "postgres:latest", "redis:latest", "mongo:latest"]Then I noticed the gitlab multi runner received the build request and started logging some debug statements:
DEBU[0006] Checking runner <server> url=https://gitlab.domain.com/ci token=<token>
INFO[0006] Checking for builds... received runner=6822dcd4
DEBU[0006] Received new build for 6822dcd4 build 17 builds=0
DEBU[0006] Added a new build id: 17
projectid: 11
commands: |2-
which mysql
repourl: https://gitlab-ci-token:<token>@gitlab.domain.com/<username>/test.git
sha: 86fbff91acf387564ca3affaedeb6c7813908d40
refname: master
beforesha: "0000000000000000000000000000000000000000"
allowgitfetch: true
timeout: 3600
variables:
- key: CI_BUILD_NAME
value: job1
public: true
internal: false
- key: CI_BUILD_STAGE
value: build
public: true
internal: false
options:
image: centos:6
services:
- mysql
token: <token>
name: job1
stage: build
tag: false
tlscachain: |
-----BEGIN CERTIFICATE-----
################################################################
#################### Removed Cipher Text #######################
################################################################
-----END CERTIFICATE-----
network: {}
buildstate: ""
buildstarted: {}
buildfinished: {}
buildduration: "0"
runner:
name: <server>
limit: null
disableverbose: null
outputlimit: null
runnercredentials:
url: https://gitlab.domain.com/ci
token: <token>
tlscafile: /etc/ssl/certs/ca-bundle.crt
runnersettings:
executor: docker
buildsdir: null
cachedir: null
environment:
- MYSQL_ALLOW_EMPTY_PASSWORD=1
shell: null
ssh: null
docker:
dockercredentials:
host: null
certpath: null
tlsverify: null
hostname: null
image: centos6
privileged: false
disablecache: null
volumes:
- /cache
cachedir: null
extrahosts: []
links: []
services:
- mysql:latest
- postgres:latest
- redis:latest
- mongo:latest
waitforservicestimeout: null
allowedimages: []
allowedservices: []
imagettl: null
parallels: null
globalid: 0
runnerid: 0
projectrunnerid: 0
builds=1
INFO[0006] gitlab-ci-multi-runner 1.0.0~beta.14.g5f37bc5 (5f37bc5) build=17 runner=6822dcd4
...
...
...What struck me as odd here, is that the /etc/ssl/certs/ca-bundle has been updated via update-ca-trust and includes all trusted CA certificates included by default, but also both our Root PKI cert and Intermediate PKI cert, however the tlscachain appears to have only listed one cert.
Could the reason why this fails is because the build script is missing both the Root PKI and INtermediate PKI certificates that would be required for validating the cert chain?