Skip to content

Docker in Docker Service (DIND) broken on 11.11 runner

Summary

CI job uses docker:dind image as a service for a specific build. Works fine on 11.10 runner, service fails to start on 11.11

Steps to reproduce

.gitlab-ci.yml
build_portable_disposable:
  stage: build
  services:
    - name: docker:dind
      alias: dind
  variables:
    DOCKER_HOST: tcp://dind:2375
  script:
    - docker/build-portable disposable
          --docker-host dind
    # NOTE: because our image exists in an isolated dind instance, we need to
    # push it if we're going to use it anywhere else
    - docker push "$PORTABLE_DISPOSABLE"

Actual behavior

docker:dind service fails to start

Running with gitlab-runner 11.11.0 (6c154264)
  on ci-runner-4 9o5VTDrE
Using Docker executor with image asia.gcr.io/XXXX/ci-docker-image/18-04:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:bed64de70fa1f4d0b5a498791647c45d954cb0306ec2852dbcfb956f4ff3b0d6 for docker:dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-9o5VTDrE-project-110-concurrent-0-docker-0 probably didn't start properly.

Health check error:
service "runner-9o5VTDrE-project-110-concurrent-0-docker-0-wait-for-service" timeout

Health check container logs:


Service container logs:
2019-05-22T21:35:45.319176683Z time="2019-05-22T21:35:45.318940693Z" level=warning msg="could not change group /var/run/docker.sock to docker: group docker not found"
2019-05-22T21:35:45.319658994Z Failed to load listeners: can't create unix socket /var/run/docker.sock: device or resource busy

*********

Expected behavior

When run on 11.10 CI runner, docker:dind service starts correctly.

Running with gitlab-runner 11.10.1 (1f513601)
  on ci-runner-4 9o5VTDrE
Using Docker executor with image asia.gcr.io/XXXXX/ci-docker-image/18-04:latest ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:bed64de70fa1f4d0b5a498791647c45d954cb0306ec2852dbcfb956f4ff3b0d6 for docker:dind ...
Waiting for services to be up and running...
Pulling docker image asia.gcr.io/XXXXX/ci-docker-image/18-04:latest ...
Using docker image sha256:aaffda7b01e42d225a14e8eceb51d2214dec46613e3f3f93c147991fab497d99 for asia.gcr.io/XXXXX/ci-docker-image/18-04:latest ...
Running on runner-9o5VTDrE-project-110-concurrent-0 via ci-runner-4...

Environment description

Self hosted CE with runners in GCP

$ sudo docker info
Containers: 1
 Running: 0
 Paused: 0
 Stopped: 1
Images: 4
Server Version: 18.09.6
Storage Driver: overlay2
 Backing Filesystem: extfs
 Supports d_type: true
 Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
 Volume: local
 Network: bridge host macvlan null overlay
 Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: bb71b10fd8f58240ca47fbb579b9d1028eea7c84
runc version: 2b18fe1d885ee5083ef9f0838fee39b62d653e30
init version: fec3683
Security Options:
 apparmor
 seccomp
  Profile: default
Kernel Version: 4.15.0-1032-gcp
Operating System: Ubuntu 18.04.2 LTS
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 14.68GiB
Name: ci-runner-4
ID: P27V:3VWB:PC3J:O5WP:R2S7:BRZ6:ZUHI:WXFX:ZAOW:RLA5:PT5A:S2MX
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
 127.0.0.0/8
Live Restore Enabled: false
Product License: Community Engine

WARNING: No swap limit support
config.toml contents
concurrent = 6
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "ci-runner-4"
  request_concurrency = 6
  url = "https://version-control.hostname/"
  token = "XXXXXX"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "docker:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    wait_for_services_timeout = 120
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Used GitLab Runner version

$ gitlab-runner --version
Version:      11.10.1
Git revision: 1f513601
Git branch:   11-10-stable
GO version:   go1.8.7
Built:        2019-04-24T09:29:18+0000
OS/Arch:      linux/amd64
$ gitlab-runner --version
Version:      11.11.0
Git revision: 6c154264
Git branch:   11-11-stable
GO version:   go1.8.7
Built:        2019-05-20T20:19:37+0000
OS/Arch:      linux/amd64