Gitlab-ci: building docker gives “certificate signed by unknown authority”

Building docker gives “certificate signed by unknown authority”

An apology upfront to create this issue here. But posting on the forum [1] did not trigger any answer. And if it is not a bug, then the available documentation is not correct, or sufficient. I've used [2] and [3] to configure the runner and the gitlab-ci script.

The project is a simple one, trying to build a docker image. The server is a self-hosted GitLab 13.0.1 and the runner is a "docker" executor. The build fails with:

 $ docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
error during connect: Post https://gitlab.m2m4all.com:5050/v1.40/auth: x509: certificate signed by unknown authority

Steps to reproduce

The project has a Dockerfile. Just basic stuff. The contents is not relevant, because the build fails before it gets to it.

Create a runner

$ sudo gitlab-runner register -n \
  --url https://gitlab.example.com/ \
  --registration-token m-secret \
  --executor docker \
  --description "My Privileged Docker Runner" \
  --docker-image "docker:19.03.11" \
  --docker-privileged \
  --docker-volumes "/certs/client"

Give this runner a tag docker-in-docker so that gitlab-ci uses the correct runner.

.gitlab-ci.yml 
variables:
  DOCKER_HOST: tcp://gitlab.example.com:5050

docker-build-master:
  image: docker:19.03.11
  stage: build
  tags:
    - docker-in-docker
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE" .
    - docker push "$CI_REGISTRY_IMAGE"
  only:
    - master

docker-build:
  image: docker:19.03.11
  stage: build
  tags:
    - docker-in-docker
  services:
    - docker:dind
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG" .
    - docker push "$CI_REGISTRY_IMAGE:$CI_COMMIT_REF_SLUG"
  except:
    - master

Actual behavior

Build fails with "certificate signed by unknown authority"

Expected behavior

Build succeeds

Relevant logs and/or screenshots

Hosts vps04.example.com and gitlab.example.com are the same machine. One is a CNAME of the other.

job log 
 Running with gitlab-runner 13.0.1 (21cb397c)
  on My Privileged Docker Runner wYCaiXxj
Preparing the "docker" executor
00:14
Using Docker executor with image docker:19.03.11 ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:d5d139be840a6ffa04348fc87740e8c095cade6e9cb977785fdba51e5fd7ffec for docker:dind ...
Waiting for services to be up and running...
Pulling docker image docker:19.03.11 ...
Using docker image sha256:0bfe00e7bcd536451161fd64b2fe05f4ff6373ce9dd241e9762b68c7f56ce438 for docker:19.03.11 ...
Preparing environment
00:02
Running on runner-wycaixxj-project-609-concurrent-0 via vps04.example.com...
Getting source from Git repository
00:02
Fetching changes with git depth set to 50...
Reinitialized existing Git repository in /builds/arduino/dockers/arduino-base/.git/
From https://gitlab.example.com/arduino/dockers/arduino-base
 * [new ref]         refs/pipelines/3575 -> refs/pipelines/3575
   6695f28..4c8fa08  staging/kees/add-gitlab-ci -> origin/staging/kees/add-gitlab-ci
Checking out 4c8fa088 as staging/kees/add-gitlab-ci...
Skipping Git submodules setup
Restoring cache
00:01
Downloading artifacts
00:02
Running before_script and script
00:02
$ docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
error during connect: Post https://gitlab.example.com:5050/v1.40/auth: x509: certificate signed by unknown authority
Running after_script
00:01
Uploading artifacts for failed job
00:02
ERROR: Job failed: exit code 1

Environment description

Ubuntu 16.04.6 LTS with GitLab 13.0.1

GitLab is configured to use Let's Encrypt certificates. The certificate is correct, and this certificate is also served at port 5050. (Checked with openssl s_client -showcerts)

And gitlab-runner (see below)

config.toml contents 
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "Runner at vps04.example.com"
  url = "https://gitlab.example.com/"
  token = "FjjE7WYe..."
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "ubuntu:18.04"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

[[runners]]
  name = "My Privileged Docker Runner"
  url = "https://gitlab.example.com/"
  token = "wYCaiXxj..."
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]
  [runners.docker]
    tls_verify = false
    image = "docker:19.03.11"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/certs/client", "/cache"]
    shm_size = 0

Used GitLab Runner version

# gitlab-runner --version
Version:      13.0.1
Git revision: 21cb397c
Git branch:   13-0-stable
GO version:   go1.13.8
Built:        2020-06-01T08:24:47+0000
OS/Arch:      linux/amd64

[1] https://forum.gitlab.com/t/gitlab-ci-build-docker-gives-certificate-signed-by-unknown-authority/39029

[2] https://docs.gitlab.com/ee/ci/docker/using_docker_build.html#tls-enabled

[3] https://gitlab.com/help/user/packages/container_registry/index

Edited by Kees Bakker