## Summary Since tonight brought the update to GitLab Runner 12.4.0, all jobs fail with: `SSL certificate problem: unable to get issuer certificate` ## Steps to reproduce Install gitlab-runner 12.4.0 on Debian 10. ## Relevant logs and/or screenshots <details> <summary> job log </summary> ```sh Running with gitlab-runner 12.4.0 (1564076b) on gitlab-rlp-runner-01 0c7c02d6 Using Docker executor with image docker:stable ... Starting service docker:dind ... Pulling docker image docker:dind ... Using docker image sha256:eaed9efc02d231a28b0f7725551a71459a7763e83bcda5464db66ec4684a1639 for docker:dind ... Waiting for services to be up and running... Pulling docker image docker:stable ... Using docker image sha256:cf85f29ec76f50421c5ba0ff999b8856e044bf08525a8cf2fe2d3fd3d4a8cd86 for docker:stable ... Running on runner-0c7c02d6-project-2629-concurrent-0 via gitlab-rlp-runner-01... Fetching changes... Reinitialized existing Git repository in /builds/zdvsysunix/docker-php/.git/ fatal: unable to access 'https://gitlab-ci-token:[MASKED]@gitlab.rlp.net/zdvsysunix/docker-php.git/': SSL certificate problem: unable to get issuer certificate Uploading artifacts... WARNING: php54-fpm/goss-report.xml: no matching files ERROR: No files to upload ERROR: Job failed: exit code 1 ``` </details> ## Environment description This is GitLab 12.3.5-ee with GitLab Runners 12.4.0 on Debian 10 <details> <summary> docker info </summary> ``` Client: Debug Mode: false Server: Containers: 9 Running: 2 Paused: 0 Stopped: 7 Images: 59 Server Version: 19.03.4 Storage Driver: overlay2 Backing Filesystem: extfs Supports d_type: true Native Overlay Diff: true Logging Driver: json-file Cgroup Driver: cgroupfs Plugins: Volume: local Network: bridge host ipvlan macvlan null overlay Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog Swarm: inactive Runtimes: runc Default Runtime: runc Init Binary: docker-init containerd version: b34a5c8af56e510852c35414db4c1f4fa6172339 runc version: 3e425f80a8c931f88e6d94a8c831b9d5aa481657 init version: fec3683 Security Options: apparmor seccomp Profile: default Kernel Version: 4.19.0-6-amd64 Operating System: Debian GNU/Linux 10 (buster) OSType: linux Architecture: x86_64 CPUs: 2 Total Memory: 3.811GiB Name: gitlab-rlp-runner-01 ID: JXBQ:RO6A:HK32:6S2Y:4KOO:MRGO:DCRR:SZRH:WUL3:3I7V:SLNZ:GFEA Docker Root Dir: /var/lib/docker Debug Mode: false Registry: https://index.docker.io/v1/ Labels: Experimental: false Insecure Registries: 127.0.0.0/8 Live Restore Enabled: false ``` </details> <details> <summary> config.toml contents </summary> ```toml concurrent = 1 check_interval = 0 [session_server] listen_address = "0.0.0.0:8093" advertise_address = "gitlab-rlp-runner-01.zdv.uni-mainz.de:8093" session_timeout = 1800 [[runners]] name = "gitlab-rlp-runner-01" url = "https://gitlab.rlp.net/" token = "[]" executor = "docker" output_limit = 8192 [runners.docker] tls_verify = false image = "alpine:latest" memory = "3072m" memory_swap = "3072m" privileged = true disable_entrypoint_overwrite = false oom_kill_disable = false disable_cache = false volumes = ["/certs/client", "/cache"] shm_size = 0 environment = ["DOCKER_DRIVER=overlay2"] [runners.cache] [runners.cache.s3] [runners.cache.gcs] ``` </details> ### Used GitLab Runner version ``` GitLab Runner 12.4.0 (1564076b) gitlab/gitlab-runner-helper x86_64-1564076b 89c664931531 3 hours ago 52.7MB ``` ## Workaround/Fix ### Rollback Rollback the GitLab Runner version to v12.3.0 ### Use patched version We have a working fix in https://gitlab.com/gitlab-org/gitlab-runner/merge_requests/1643 which you can download in https://gitlab-runner-downloads.s3.amazonaws.com/fix-tls-chain-building/index.html until we release a patch release. If you are using the docker executor and facing issues like `Pulling docker image gitlab/gitlab-runner-helper:x86_64-2d8b7be4 ... ERROR: Job failed: Error response from daemon: manifest for gitlab/gitlab-runner-helper:x86_64-2d8b7be4 not found: manifest unknown: manifest unknown` Update `config.toml` to point to a specific helper image until the issue is merged into master and a patch release it released. To do this check https://docs.gitlab.com/runner/configuration/advanced-configuration.html#overriding-the-helper-image It should look something like: <details> <summary> config.toml </summary> ``` [[runners]] name = "docker" url = "http://xxxx" token = "x-xxxx" executor = "docker" [runners.custom_build_dir] [runners.docker] helper_image = "gitlab/gitlab-runner-helper:x86_64-latest" ``` </details>