Skip to content

GitLab runner ignores the "runtime" configuration for service containers

Summary

The GitLab runner's Docker Executor accepts a "runtime" configuration to select the Docker runtime (e.g., runc) to be used when deploying containers for the job.

This runtime configuration is applied to the job container, but is being ignored for "service" containers. This appears to be a bug since other Docker Executor configurations (e.g., "privileged", "userns-mode", etc.) are honored for both the job container and all service containers.

Steps to reproduce

To reproduce, simply configure the GitLab Runner's Docker Executor with a "runtime" other than the default runc, and then launch a job that has a service container.

For example, I configured the GitLab Runner's Docker Executor to use the Sysbox runtime which supports Docker-in-Docker without using privileged containers (and thus more securely).

The GitLab runner's config file looks like this (notice the runtime and privileged configs):

[[runners]]
    url = "https://gitlab.com/"
    token = REGISTRATION_TOKEN
    executor = "docker"
    [runners.docker]
        tls_verify = true
        image = "docker:19.03.12"
        privileged = false
        disable_cache = false
        volumes = ["/certs/client", "/cache"]
        runtime = "sysbox-runc"

I then created a GitLab Docker-in-Docker job (.gitlab-ci.yml) as follows:

image: docker:19.03.12

services:
  - docker:19.03.12-dind

build:
    stage: build
    script:
      - docker build -t my-docker-image .

Actual behavior

The actual behavior was that the GitLab runner deployed the job container (docker:19.03.12) using Sysbox (as expected), but deployed the service container (docker:19.03.12-dind) using the default runc runtime.

This caused the job to fail since the default runc does not support Docker-in-Docker unless privileged containers are used.

Expected behavior

The GitLab runner should have deployed both the job container and the service container using the configured runtime (Sysbox). It is not correct for the GitLab runner to honor the runtime config for the job container but ignore it for the service container.

As a point of comparison, the GitLab runner did honor the "privileged" config for both the job and service containers (as expected). Thus the behavior on the "runtime" flag is inconsistent.

Environment description

This was tested using a custom GitLab runner installation on a Linux machine (Ubuntu).

Since I am using the Docker Executor, here is the docker info:

$ docker info
Client:
 Debug Mode: false

Server:
 Containers: 1
  Running: 1
  Paused: 0
  Stopped: 0
 Images: 7
 Server Version: 19.03.12
 Storage Driver: overlay2
  Backing Filesystem: extfs
  Supports d_type: true
  Native Overlay Diff: true
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc sysbox-runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 5.3.0-64-generic
 Operating System: Ubuntu 19.10
 OSType: linux
 Architecture: x86_64
 CPUs: 4
 Total Memory: 7.777GiB
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false

WARNING: No swap limit support

Used GitLab Runner version

$ gitlab-runner --version
Version:      13.5.0
Git revision: ece86343
Git branch:   13-5-stable
GO version:   go1.13.8
Built:        2020-10-20T12:05:22+0000
OS/Arch:      linux/amd64

Possible fixes

I believe the bug is in the GitLab Runner's Docker Executor code, here:

https://gitlab.com/gitlab-org/gitlab-runner/-/blob/master/executors/docker/docker.go#L469

Looks like function createHostConfigForService() is returning a container.HostConfig struct which ignores the "runtime" configuration for the GitLab Runner.

Edited by Cesar Talledo