Job fails when runner is configured with `privileged = true`
Summary
Job fails with ERROR: Job failed (system failure): prepare environment: Error response from daemon: Cannot link to a non running container when runner is configured with privileged = true.
Steps to reproduce
Everything was working fine until I wanted to use testcontainers. For this to work, I followed their gitlab-ci related instructions telling me to use the DinD service. This failed with something like "mount: permission denied (are you root?)". Googling this, I found out that DinD requires the "privileged" flag to be set in order to work, so I went and set it in the gitlab-runner config (see below). After this, I started getting the error (see job log below).
.gitlab-ci.yml
# DinD service is required for Testcontainers
services:
- docker:dind
variables:
GRADLE_OPTS: "-Dorg.gradle.daemon=false"
GRADLE_USER_HOME: $CI_PROJECT_DIR/.gradleHome
# Instruct Testcontainers to use the daemon of DinD.
DOCKER_HOST: "tcp://docker:2375"
# Improve performance with overlayfs.
DOCKER_DRIVER: overlay2
cache:
paths:
- .gradleHome/wrapper/
- .gradleHome/caches/
build_job:
stage: build
image: adoptopenjdk:11-hotspot
tags: [ docker ]
script:
- ./gradlew assemble
artifacts:
expire_in: 10 minutes
paths:
- "*/build"
- ".gradle"
test_job:
stage: test
image: adoptopenjdk:11-hotspot
tags: [ docker ]
script:
- ./gradlew test
artifacts:
expire_in: 10 minutes
when: on_failure
paths:
- "*/build/reports/"
Actual behavior
The job doesn't even start and fails in the "Preparing Environment" phase with the error below.
Expected behavior
Job should work, including DinD and Testcontainers should be able to use the DinD docker daemon via TCP.
Relevant logs and/or screenshots
job log
Running with gitlab-runner 13.1.1 (6fbc7474)
on [host] fa8694f5
Preparing the "docker" executor
Using Docker executor with image adoptopenjdk:11-hotspot ...
Starting service docker:dind ...
Pulling docker image docker:dind ...
Using docker image sha256:d5d139be840a6ffa04348fc87740e8c095cade6e9cb977785fdba51e5fd7ffec for docker:dind ...
Waiting for services to be up and running...
Pulling docker image adoptopenjdk:11-hotspot ...
Using docker image sha256:81bace7bcde3b2c650154bfb2716bab74d18fe8697f35ba8bce757803bee8507 for adoptopenjdk:11-hotspot ...
Preparing environment
ERROR: Job failed (system failure): prepare environment: Error response from daemon: Cannot link to a non running container: /runner-fa8694f5-project-78-concurrent-0-3c79b234512e2ff1-docker-0 AS /runner-fa8694f5-project-78-concurrent-0-3c79b234512e2ff1-predefined-0/docker (docker.go:740:0s). Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information
Environment description
This is a custom GitLab installation on Debian Jessie using the GitLab omnibus installer.
config.toml contents
concurrent = 3
check_interval = 0
[[runners]]
name = "[name]"
url = "[url]"
token = "[token]"
executor = "docker"
[runners.docker]
tls_verify = false
image = "buildpack-deps:jessie"
privileged = true
disable_cache = false
volumes = ["/cache"]
[runners.cache]
docker info
Containers: 8
Running: 6
Paused: 0
Stopped: 2
Images: 132
Server Version: 17.05.0-ce
Storage Driver: aufs
Root Dir: /var/lib/docker/aufs
Backing Filesystem: extfs
Dirs: 422
Dirperm1 Supported: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host macvlan null overlay
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 9048e5e50717ea4497b757314bad98ea3763c145
runc version: 9c2d8d184e5da67c95d601382adf14862e4f2228
init version: 949e6fa
Kernel Version: 3.16.0-6-amd64
Operating System: Debian GNU/Linux 8 (jessie)
OSType: linux
Architecture: x86_64
CPUs: 8
Total Memory: 31.32GiB
Name: [hostname]
ID: [ID]
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No memory limit support
WARNING: No swap limit support
WARNING: No kernel memory limit support
WARNING: No oom kill disable support
WARNING: No cpu cfs quota support
WARNING: No cpu cfs period support
Used GitLab Runner version
Version: 13.1.1
Git revision: 6fbc7474
Git branch: 13-1-stable
GO version: go1.13.8
Built: 2020-07-01T06:49:55+0000
OS/Arch: linux/amd64
Possible fixes
If I set privileged back to false, the build starts but DinD fails with the error message in the summary (which is expected, since DinD requires privileged mode).