GitLab runner ignores the "runtime" configuration for service containers
Summary
The GitLab runner's Docker Executor accepts a "runtime" configuration to select the Docker runtime (e.g., runc) to be used when deploying containers for the job.
This runtime configuration is applied to the job container, but is being ignored for "service" containers. This appears to be a bug since other Docker Executor configurations (e.g., "privileged", "userns-mode", etc.) are honored for both the job container and all service containers.
Steps to reproduce
To reproduce, simply configure the GitLab Runner's Docker Executor with a "runtime" other than the default runc, and then launch a job that has a service container.
For example, I configured the GitLab Runner's Docker Executor to use the Sysbox runtime which supports Docker-in-Docker without using privileged containers (and thus more securely).
The GitLab runner's config file looks like this (notice the runtime
and privileged
configs):
[[runners]]
url = "https://gitlab.com/"
token = REGISTRATION_TOKEN
executor = "docker"
[runners.docker]
tls_verify = true
image = "docker:19.03.12"
privileged = false
disable_cache = false
volumes = ["/certs/client", "/cache"]
runtime = "sysbox-runc"
I then created a GitLab Docker-in-Docker job (.gitlab-ci.yml
) as follows:
image: docker:19.03.12
services:
- docker:19.03.12-dind
build:
stage: build
script:
- docker build -t my-docker-image .
Actual behavior
The actual behavior was that the GitLab runner deployed the job container (docker:19.03.12
) using Sysbox (as expected), but deployed the service container (docker:19.03.12-dind
) using the default runc runtime.
This caused the job to fail since the default runc does not support Docker-in-Docker unless privileged containers are used.
Expected behavior
The GitLab runner should have deployed both the job container and the service container using the configured runtime (Sysbox). It is not correct for the GitLab runner to honor the runtime config for the job container but ignore it for the service container.
As a point of comparison, the GitLab runner did honor the "privileged" config for both the job and service containers (as expected). Thus the behavior on the "runtime" flag is inconsistent.
Environment description
This was tested using a custom GitLab runner installation on a Linux machine (Ubuntu).
Since I am using the Docker Executor, here is the docker info
:
$ docker info
Client:
Debug Mode: false
Server:
Containers: 1
Running: 1
Paused: 0
Stopped: 0
Images: 7
Server Version: 19.03.12
Storage Driver: overlay2
Backing Filesystem: extfs
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc sysbox-runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 5.3.0-64-generic
Operating System: Ubuntu 19.10
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 7.777GiB
Docker Root Dir: /var/lib/docker
Debug Mode: false
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
WARNING: No swap limit support
Used GitLab Runner version
$ gitlab-runner --version
Version: 13.5.0
Git revision: ece86343
Git branch: 13-5-stable
GO version: go1.13.8
Built: 2020-10-20T12:05:22+0000
OS/Arch: linux/amd64
Possible fixes
I believe the bug is in the GitLab Runner's Docker Executor code, here:
https://gitlab.com/gitlab-org/gitlab-runner/-/blob/master/executors/docker/docker.go#L469
Looks like function createHostConfigForService()
is returning a container.HostConfig
struct which ignores the "runtime" configuration for the GitLab Runner.