docker:dind service broken in 11.11: Cannot link to a non running container
Summary
docker:dind
does not work any longer with gitlab-runner
11.11.2. An error message about "Cannot link to a non running container" is produced (more details below)
Steps to reproduce
Enable GitLab CI runs with a CI config like below:
.gitlab-ci.yml
stages:
- build
# Global variables have lower precedence than per-job variables.
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
GRADLE_USER_HOME: "/cache/gradle_home"
# Needed for Testcontainers
DOCKER_HOST: "tcp://docker:2375"
DOCKER_DRIVER: overlay2
# DinD service is required for Testcontainers
services:
- docker:dind
build:openjdk8:
stage: build
image: hiboxsystems/openjdk:8
variables:
CI_CACHE_KEY: "openjdk8"
script:
- java -version
- ./gradlew build
Actual behavior
The following error is produced in the CI runs, and the Docker-in-Docker service is not working correctly.
Running with gitlab-runner 11.11.2 (ac2a293c)
on gitlab-ci-1 a1159bbc
Using Docker executor with image hiboxsystems/openjdk-with-docker:8 ...
Starting service docker:dind ...
Using locally found image version due to if-not-present pull policy
Using docker image sha256:bed64de70fa1f4d0b5a498791647c45d954cb0306ec2852dbcfb956f4ff3b0d6 for docker:dind ...
Waiting for services to be up and running...
*** WARNING: Service runner-a1159bbc-project-36-concurrent-0-docker-0 probably didn't start properly.
Health check error:
ContainerStart: Error response from daemon: Cannot link to a non running container: /runner-a1159bbc-project-36-concurrent-0-docker-0 AS /runner-a1159bbc-project-36-concurrent-0-docker-0-wait-for-service/service (executor_docker.go:1248:0s)
Service container logs:
2019-06-13T13:15:17.882947094Z time="2019-06-13T13:15:17.882755300Z" level=warning msg="could not clean up old root propagation unmount file" error="remove /var/run/docker/unmount-on-shutdown: not a directory" file=/var/run/docker/unmount-on-shutdown
2019-06-13T13:15:17.883015506Z mkdir /var/run/docker: not a directory
*********
Using locally found image version due to if-not-present pull policy
Expected behavior
With version 11.9, this is what I get instead, and Docker-in-Docker works correctly.
Running with gitlab-runner 11.9.0 (692ae235)
on gitlab-ci-1 a1159bbc
Using Docker executor with image hiboxsystems/openjdk:8 ...
Starting service docker:dind ...
Using locally found image version due to if-not-present pull policy
Using docker image sha256:bed64de70fa1f4d0b5a498791647c45d954cb0306ec2852dbcfb956f4ff3b0d6 for docker:dind ...
Waiting for services to be up and running...
Using locally found image version due to if-not-present pull policy
Using docker image sha256:4044b014b7068731eddef448f5937c7215d7d749ea07e2ca34deea3e014e1a5f for hiboxsystems/openjdk:8 ...
Environment description
Locally hosted installation.
Docker version: I've tested both 18.09.6
and 18.06.3
, same problem with both.
I even wiped /var/lib/docker
and did a full reinstall of Docker, but the problem still persisted.
config.toml contents
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "gitlab-ci-1"
url = "https://git.some-company.bar/"
token = "<removed>"
executor = "docker"
[runners.docker]
tls_verify = false
image = "acme-inc/openjdk:8"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/cache", "/var/run/docker.sock:/var/run/docker"]
pull_policy = "if-not-present"
shm_size = 0
Used GitLab Runner version
Version: 11.11.2
Git revision: ac2a293c
Git branch:
GO version: go1.8.7
Built: 2019-06-03T10:57:49+0000
OS/Arch: linux/amd64
Possible fixes
Downgrading to 11.9.0 (I had the .deb
laying around on another machine) helped resolve the issue.