Skip to content

docker:dind service broken in 11.11: Cannot link to a non running container

Summary

docker:dind does not work any longer with gitlab-runner 11.11.2. An error message about "Cannot link to a non running container" is produced (more details below)

Steps to reproduce

Enable GitLab CI runs with a CI config like below:

.gitlab-ci.yml
stages:
  - build

# Global variables have lower precedence than per-job variables.
variables:
  GRADLE_OPTS: "-Dorg.gradle.daemon=false"
  GRADLE_USER_HOME: "/cache/gradle_home"

  # Needed for Testcontainers
  DOCKER_HOST: "tcp://docker:2375"
  DOCKER_DRIVER: overlay2

# DinD service is required for Testcontainers
services:
  - docker:dind

build:openjdk8:
  stage: build
  image: hiboxsystems/openjdk:8
  variables:
    CI_CACHE_KEY: "openjdk8"
  script:
    - java -version
    - ./gradlew build

Actual behavior

The following error is produced in the CI runs, and the Docker-in-Docker service is not working correctly.

Running with gitlab-runner 11.11.2 (ac2a293c)
  on gitlab-ci-1 a1159bbc
Using Docker executor with image hiboxsystems/openjdk-with-docker:8 ...
Starting service docker:dind ...
Using locally found image version due to if-not-present pull policy
Using docker image sha256:bed64de70fa1f4d0b5a498791647c45d954cb0306ec2852dbcfb956f4ff3b0d6 for docker:dind ...
Waiting for services to be up and running...

*** WARNING: Service runner-a1159bbc-project-36-concurrent-0-docker-0 probably didn't start properly.

Health check error:
ContainerStart: Error response from daemon: Cannot link to a non running container: /runner-a1159bbc-project-36-concurrent-0-docker-0 AS /runner-a1159bbc-project-36-concurrent-0-docker-0-wait-for-service/service (executor_docker.go:1248:0s)

Service container logs:
2019-06-13T13:15:17.882947094Z time="2019-06-13T13:15:17.882755300Z" level=warning msg="could not clean up old root propagation unmount file" error="remove /var/run/docker/unmount-on-shutdown: not a directory" file=/var/run/docker/unmount-on-shutdown
2019-06-13T13:15:17.883015506Z mkdir /var/run/docker: not a directory

*********

Using locally found image version due to if-not-present pull policy

Expected behavior

With version 11.9, this is what I get instead, and Docker-in-Docker works correctly.

Running with gitlab-runner 11.9.0 (692ae235)
  on gitlab-ci-1 a1159bbc
Using Docker executor with image hiboxsystems/openjdk:8 ...
Starting service docker:dind ...
Using locally found image version due to if-not-present pull policy
Using docker image sha256:bed64de70fa1f4d0b5a498791647c45d954cb0306ec2852dbcfb956f4ff3b0d6 for docker:dind ...
Waiting for services to be up and running...
Using locally found image version due to if-not-present pull policy
Using docker image sha256:4044b014b7068731eddef448f5937c7215d7d749ea07e2ca34deea3e014e1a5f for hiboxsystems/openjdk:8 ...

Environment description

Locally hosted installation.

Docker version: I've tested both 18.09.6 and 18.06.3, same problem with both.

I even wiped /var/lib/docker and did a full reinstall of Docker, but the problem still persisted.

config.toml contents
concurrent = 1
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "gitlab-ci-1"
  url = "https://git.some-company.bar/"
  token = "<removed>"
  executor = "docker"

  [runners.docker]
    tls_verify = false
    image = "acme-inc/openjdk:8"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker"]
    pull_policy = "if-not-present"
    shm_size = 0

Used GitLab Runner version

Version:      11.11.2
Git revision: ac2a293c
Git branch:   
GO version:   go1.8.7
Built:        2019-06-03T10:57:49+0000
OS/Arch:      linux/amd64

Possible fixes

Downgrading to 11.9.0 (I had the .deb laying around on another machine) helped resolve the issue.