12.4.0: All jobs fail with SSL certificate problem: unable to get issuer certificate

Summary

Since tonight brought the update to GitLab Runner 12.4.0, all jobs fail with: SSL certificate problem: unable to get issuer certificate

Steps to reproduce

Install gitlab-runner 12.4.0 on Debian 10.

Relevant logs and/or screenshots

job log

Running with gitlab-runner 12.4.0 (1564076b)
  on gitlab-rlp-runner-01 0c7c02d6
Using Docker executor with image docker:stable ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:eaed9efc02d231a28b0f7725551a71459a7763e83bcda5464db66ec4684a1639 for docker:dind ...
Waiting for services to be up and running...
Pulling docker image docker:stable ...
Using docker image sha256:cf85f29ec76f50421c5ba0ff999b8856e044bf08525a8cf2fe2d3fd3d4a8cd86 for docker:stable ...
Running on runner-0c7c02d6-project-2629-concurrent-0 via gitlab-rlp-runner-01...
Fetching changes...
Reinitialized existing Git repository in /builds/zdvsysunix/docker-php/.git/
fatal: unable to access 'https://gitlab-ci-token:[MASKED]@gitlab.rlp.net/zdvsysunix/docker-php.git/': SSL certificate problem: unable to get issuer certificate
Uploading artifacts...
WARNING: php54-fpm/goss-report.xml: no matching files 
ERROR: No files to upload                          
ERROR: Job failed: exit code 1

Environment description

This is GitLab 12.3.5-ee with GitLab Runners 12.4.0 on Debian 10

docker info

Client:
 Debug Mode: false

Server:
 Containers: 9
  Running: 2
  Paused: 0
  Stopped: 7
 Images: 59
 Server Version: 19.03.4
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339
 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.0-6-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 2
 Total Memory: 3.811GiB
 Name: gitlab-rlp-runner-01
 ID: JXBQ:RO6A:HK32:6S2Y:4KOO:MRGO:DCRR:SZRH:WUL3:3I7V:SLNZ:GFEA
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

config.toml contents

concurrent = 1
check_interval = 0

[session_server]
  listen_address = "0.0.0.0:8093"
  advertise_address = "gitlab-rlp-runner-01.zdv.uni-mainz.de:8093"
  session_timeout = 1800

[[runners]]
  name = "gitlab-rlp-runner-01"
  url = "https://gitlab.rlp.net/"
  token = "[]"
  executor = "docker"
  output_limit = 8192
  [runners.docker]
    tls_verify = false
    image = "alpine:latest"
    memory = "3072m"
    memory_swap = "3072m"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    shm_size = 0
    environment = ["DOCKER_DRIVER=overlay2"]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Used GitLab Runner version

GitLab Runner 12.4.0 (1564076b)
gitlab/gitlab-runner-helper x86_64-1564076b 89c664931531 3 hours ago 52.7MB

Workaround/Fix

Rollback

Rollback the GitLab Runner version to v12.3.0

Use patched version

We have a working fix in !1643 (merged) which you can download in https://gitlab-runner-downloads.s3.amazonaws.com/fix-tls-chain-building/index.html until we release a patch release.

If you are using the docker executor and facing issues like Pulling docker image gitlab/gitlab-runner-helper:x86_64-2d8b7be4 ... ERROR: Job failed: Error response from daemon: manifest for gitlab/gitlab-runner-helper:x86_64-2d8b7be4 not found: manifest unknown: manifest unknown Update config.toml to point to a specific helper image until the issue is merged into master and a patch release it released. To do this check https://docs.gitlab.com/runner/configuration/advanced-configuration.html#overriding-the-helper-image

It should look something like:

config.toml

[[runners]]
  name = "docker"
  url = "http://xxxx"
  token = "x-xxxx"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    helper_image = "gitlab/gitlab-runner-helper:x86_64-latest"