Solution - fatal: unable to access 'https://gitlabs.private-repo.com/my-project/app.git/': SSL certificate problem: unable to get local issuer certificate

Summary

While running the pipeline , getting the following error, All the jobs are failed with below error
fatal: unable to access 'https://gitlabs.private-repo.com/my-project/app.git/': SSL certificate problem: unable to get local issuer certificate
My gitlabs instance is running in a enterprises server. It is has SSL certificate which is not a self signed SSL.

Steps to reproduce

  • Create a repo in private gitlab repo which has unknown CA (corporate/enterprises CA) issued SSL certificate or self signed SSL certificate

  • Register Runner with that private repo with mentioning the CA cert path
    gitlab-runner register --tls-ca-path /path/to/private-repo.ca.crt

  • Run the pipline
    Below given simple pipline which using for testing
    This pipline has single job, which will echo 'hello' and exit

.gitlab-ci.yml 
stages :
    - manual_buid

manual_build :
    only :
    - web
    when: manual   
    script:
        - echo "hello"
    stage : manual_buid
    tags: ['Test-Runner']

Log of the Failed JOB

job log 
Running with gitlab-runner 13.1.0 (6214287e)
  on My Docker Runner GNWEPzV1
Preparing the "docker" executor
Using Docker executor with image myDockerHub/myImage:latest ...
Authenticating with credentials from /root/.docker/config.json
Pulling docker image myDockerHub/myImage:latest ...
Using docker image sha256:b3c3748c9844567d6584de6d957e249013effd7536fbdd848f73789bd2b09020 for myDockerHub/myImage:latest ...
Preparing environment
Running on runner-gnwepzv1-project-64-concurrent-0 via myLinuxHostname.localdomain...
Getting source from Git repository
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/my-project/app/.git/
*   Trying 192.168.1.1:443...
* TCP_NODELAY set
* Connected to gitlabs.private-repo.com (192.168.1.1) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
*   CAfile: /builds/my-project/app.tmp/CI_SERVER_TLS_CA_FILE
  CApath: none
* SSL certificate problem: unable to get local issuer certificate
* Closing connection 0
fatal: unable to access 'https://gitlabs.private-repo.com/my-project/app.git/': SSL certificate problem: unable to get local issuer certificate
ERROR: Job failed: exit code 1

Environment description

config.toml contents 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "My Runner - Docker"
  url = "https://gitlabs.private-repo.com/"
  token = "iweSKLAER93249isdfiSIA"
  tls-ca-file = "/path/to/gitlabs.private-repo.com.crt"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "myDockerHub/myImage:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Used GitLab Runner version

Version: 13.1.0
Git revision: 6214287e
Git branch: 13-1-stable
GO version: go1.13.8
Built: 2020-06-19T21:12:22+0000
OS/Arch: linux/amd64

Possible fixes

Upgrading the alpine version 3.10 to 3.12 in gitlab-runner-helper

Solved by add this below lines to [[runners]] in /etc/gitlab-runner/config.toml

pre_clone_script = "sed -i 's/3.10/3.12/g' /etc/apk/repositories && apk update && apk upgrade"

How this was verified

  • Started the container with alpine:latest and added git by apk add git

  • Added the SSL CA certificate path to the GIT Config, through below command
    git config --global http.https://gitlabs.private-repo.com/.sslcainfo /path/to/private-repo.ca.crt

  • Tried to clone the same private gitlab repo
    GIT_CURL_VERBOSE=1 git clone https://gitlabs.private-repo.com/my-project/app.git
    Successfully clone gitlab private repo in alpine 3.12

Actions based on Morale form the above story

  • So, Added the below line to [[runner]] section of config.toml in gitlab-runner machine
    pre_clone_script = "sed -i 's/3.10/3.12/g' /etc/apk/repositories && apk update && apk upgrade"
modified config.toml contents 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "My Runner - Docker"
  url = "https://gitlabs.private-repo.com/"
  token = "iweSKLAER93249isdfiSIA"
  tls-ca-file = "/etc/gitlab-runner/certs/gitlabs.private-repo.com.crt"
  environment = ["GIT_CURL_VERBOSE=1"]
  pre_clone_script = "sed -i 's/3.10/3.12/g' /etc/apk/repositories && apk update && apk upgrade"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "myDockerHub/myImage:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

  • Restarted the gitlab-runner service
    systemctl restart gitlab-runner

  • Ran again the pipeline.
    Pipeline passed

Suspecting

May be, due to build configuration of libcURL or ssl_client or libTLS in alpine 3.10 of gitlab-runner-helper image

Edited by 🤖 GitLab Bot 🤖