Skip to content

Gitlab-runner breaks when `volume_driver = "btrfs"` is used in the gitlab-runner toml file

Workaround/fix

As explained in #25454 (comment 336617711) btrfs and overlay2 shouldn't be values for volume_driver since these are StorageDrivers. Remove the volume_driver setting and everything will work as expected.

Summary

When using a docker container on btrfs, gitlab-runner now fails to create jobs. Rolling back to v12.9.0 works fine, so this was introduced with v12.10.0

The issue seems to relate when using VOLUMES in Dockerfiles, which I personally have not seen work with btrfs, it complains about missing plugins.

Steps to reproduce

Install docker on a btrfs /var/lib/docker and run any pipeline

Example Project

See here for an example: https://gitlab.com/esbs/updater/-/jobs/525365274

What is the current bug behavior?

We get an error as seen above, ERROR: Job failed (system failure): Error response from daemon: create runner-uzyuuwsf-project-10743483-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70: error looking up volume plugin btrfs: plugin "btrfs" not found (docker.go:788:15s)

What is the expected correct behavior?

The pipeline runs normally

Output of checks

This bug happens on GitLab.com, using 'gitlab-runner' inside of docker

Environment description

Client:
 Debug Mode: false

Server:
 Containers: 40
  Running: 3
  Paused: 0
  Stopped: 37
 Images: 50
 Server Version: 19.03.8
 Storage Driver: btrfs
  Build Version: Btrfs v4.20.1 
  Library Version: 102
 Logging Driver: json-file
 Cgroup Driver: cgroupfs
 Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
 Swarm: inactive
 Runtimes: runc
 Default Runtime: runc
 Init Binary: docker-init
 containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
 runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
 init version: fec3683
 Security Options:
  apparmor
  seccomp
   Profile: default
 Kernel Version: 4.19.0-8-amd64
 Operating System: Debian GNU/Linux 10 (buster)
 OSType: linux
 Architecture: x86_64
 CPUs: 12
 Total Memory: 15.47GiB
 Name: race-01
 ID: ZLNG:ZCS3:QCRZ:I527:HCUE:QBQD:5MXB:SMBF:T2MN:YNQF:IVF6:3D3S
 Docker Root Dir: /var/lib/docker
 Debug Mode: false
 Registry: https://index.docker.io/v1/
 Labels:
 Experimental: false
 Insecure Registries:
  127.0.0.0/8
 Live Restore Enabled: false
config.toml contents 
cat /var/lib/docker/data/gitlab-runner/config.toml
concurrent = 10
check_interval = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "esbs-runner-01@race-01"
  url = "https://gitlab.com"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "registry.hub.docker.com/library/alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/run/docker.sock:/run/docker.sock", "/cache"]
    volume_driver = "btrfs"
    pull_policy = "if-not-present"
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

[[runners]]
  name = "evbox-runner-01@race-01"
  url = "https://gitlab.com"
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "registry.hub.docker.com/library/alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/run/docker.sock:/run/docker.sock", "/cache"]
    volume_driver = "btrfs"
    pull_policy = "if-not-present"
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

[[runners]]
  name = "gitlab-runner-mplab-01@race-01"
  url = "https://gitlab.com"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "registry.hub.docker.com/library/alpine:latest"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/run/docker.sock:/run/docker.sock", "/cache"]
    volume_driver = "btrfs"
    network_mode = "host"
    pull_policy = "if-not-present"
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

[[runners]]
  name = "signing-runner-01@race-01"
  url = "https://gitlab.com"
  executor = "docker"
  [runners.custom_build_dir]
  [runners.docker]
    tls_verify = false
    image = "registry.hub.docker.com/library/alpine:latest"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/run/docker.sock:/run/docker.sock", "/cache", "/srv/keystore:/srv/keystore"]
    volume_driver = "btrfs"
    pull_policy = "if-not-present"
    shm_size = 0
  [runners.cache]
    [runners.cache.s3]
    [runners.cache.gcs]

Used GitLab Runner version

Running with gitlab-runner 12.10.1 (ce065b93)

Removing the option volume_driver = "btrfs" seems to fix the issue, and so it seems that previously, this setting was ignored or previous this setting did not matter for 'normal' operation. I do know that defining 'VOLUME" in dockerfiles breaks on btrfs, if the volume was not pre-created. I haven't spend much time on it; just noticed in the past.

Edited by Steve Xuereb