CI: cannot run dind (Docker in Docker) with public/shared gitlag-org private runners

Summary

I've been trying to setup a job to build an image and push to our bundled registry, but I can't get the DinD service to run correctly.

I've followed this documentations to build the CI job:

Steps to reproduce

With a project in the user namespace (running against GitLab public shared runners), try to build an image with the following CI job:

container:build:
  image: docker:23-cli
  stage: build
  needs: []
  services:
    - name: docker:23-dind
      alias: docker
  variables:
    RELEASE_IMAGE: "$CI_REGISTRY_IMAGE/image-name:$CI_MERGE_REQUEST_IID"
    DOCKER_DRIVER: overlay2
    DOCKER_TLS_CERTDIR: "/certs"
  before_script:
    - docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
  script:
    - docker build --pull -t "$RELEASE_IMAGE" .
    - docker push "$RELEASE_IMAGE"

What is the current bug behavior?

It seems the service doesn't start up as docker fails to connect:

error during connect: Post "https://docker:2376/v1.24/auth": dial tcp: lookup docker on 127.0.0.11:53: no such host

What is the expected correct behavior?

It should work and build correctly

Relevant logs and/or screenshots

Using Docker executor with image docker:23.0-cli ...
Starting service docker:23.0-dind ...
Pulling docker image docker:23.0-dind ...
Using docker image sha256:10e3f7d9491aa6e09fb0c65ba6670c930cad6addfd8867ac27cbab774c90d606 for docker:23.0-dind with digest docker@sha256:eb9f1d80fbe98f6343fd432d8f89db19d5de324096498e67886c4a984f6ac670 ...
WARNING: Service docker:23.0-dind is already created. Ignoring.
Waiting for services to be up and running (timeout 30 seconds)...
*** WARNING: Service runner-mf8bef5g-project-32950782-concurrent-0-2d990f37f240e301-docker-0 probably didn't start properly.
Health check error:
service "runner-mf8bef5g-project-32950782-concurrent-0-2d990f37f240e301-docker-0-wait-for-service" timeout
Health check container logs:
2023-08-01T16:04:07.982718862Z waiting for TCP connection to 7dab3be5e20e on [2375 2376]...
2023-08-01T16:04:07.982962572Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:07.983192381Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:08.986112468Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:08.986151778Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:09.987526927Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:09.987582577Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:10.988482096Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:10.988549196Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:12.047014763Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:12.047403562Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:13.059056733Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:13.059125364Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:14.077693275Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:14.077751734Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:15.100392121Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:15.100559920Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:16.111270662Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:16.111317582Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:17.122111070Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:17.122190680Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:18.132673049Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:18.132752598Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:19.144864953Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:19.144963913Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:20.158220064Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:20.158356154Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:21.168497574Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:21.168716454Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:22.179839137Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:22.181015276Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:23.191139517Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:23.191342767Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:24.203047098Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:24.203122928Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:25.213601230Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:25.213721100Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:26.224442007Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:26.224507436Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:27.234399813Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:27.234613123Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:28.246693365Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:28.246940445Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:29.256803377Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:29.256964907Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:30.267785222Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:30.267897792Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:31.278701522Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:31.278973102Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:32.294804102Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:32.294868702Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:33.308671409Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:33.309037729Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:34.318808294Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:34.318963514Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:35.329759829Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:35.330010649Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:36.340053955Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:36.340513655Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:37.351291104Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:37.351364184Z dialing 7dab3be5e20e:2375...
2023-08-01T16:04:38.364450997Z dialing 7dab3be5e20e:2376...
2023-08-01T16:04:38.364645727Z dialing 7dab3be5e20e:2375...
Service container logs:
2023-08-01T16:04:09.541703572Z Certificate request self-signature ok
2023-08-01T16:04:09.541770852Z subject=CN = docker:dind server
2023-08-01T16:04:09.556183222Z /certs/server/cert.pem: OK
2023-08-01T16:04:10.641128729Z Certificate request self-signature ok
2023-08-01T16:04:10.641173149Z subject=CN = docker:dind client
2023-08-01T16:04:10.656089908Z /certs/client/cert.pem: OK
2023-08-01T16:04:10.659571093Z ip: can't find device 'ip_tables'
2023-08-01T16:04:10.660872131Z modprobe: can't change directory to '/lib/modules': No such file or directory
2023-08-01T16:04:10.664620455Z mount: permission denied (are you root?)
2023-08-01T16:04:10.664692175Z Could not mount /sys/kernel/security.
2023-08-01T16:04:10.664703615Z AppArmor detection and --privileged mode might break.
2023-08-01T16:04:10.665952353Z mount: permission denied (are you root?)
*********
Pulling docker image docker:23.0-cli ...
Using docker image sha256:95eded59f32e7afc8ecaedb82316f296b68cf9b5b782541d95d011c6ad2dfbbc for docker:23.0-cli with digest docker@sha256:5f85260f1c78cfd796463b51fd000ff9f033d88b7047739f79c6df9d0fb531d4 ...

---

Executing "step_script" stage of the job script 00:01
Using docker image sha256:95eded59f32e7afc8ecaedb82316f296b68cf9b5b782541d95d011c6ad2dfbbc for docker:23.0-cli with digest docker@sha256:5f85260f1c78cfd796463b51fd000ff9f033d88b7047739f79c6df9d0fb531d4 ...
$ docker login -u "$CI_REGISTRY_USER" -p "$CI_REGISTRY_PASSWORD" $CI_REGISTRY
WARNING! Using --password via the CLI is insecure. Use --password-stdin.
error during connect: Post "https://docker:2376/v1.24/auth": dial tcp: lookup docker on 127.0.0.11:53: no such host

Output of checks

Possible fixes

I've reported this on slack and @hfyngvason thinks this may be a configuration issue on ourside:

Yes, might need an issue. Could be a misconfiguration in recent shared runners. Based on gitlab-runner#1544 (comment 1336346549), it sounds like there might be a missing services_privileged = true

Edited by Gabriel Mazetto