Gitlab-runner breaks when `volume_driver = "btrfs"` is used in the gitlab-runner toml file
Workaround/fix
As explained in #25454 (comment 336617711) btrfs
and overlay2
shouldn't be values for volume_driver
since these are StorageDrivers
. Remove the volume_driver
setting and everything will work as expected.
Summary
When using a docker container on btrfs, gitlab-runner now fails to create jobs. Rolling back to v12.9.0 works fine, so this was introduced with v12.10.0
The issue seems to relate when using VOLUMES in Dockerfiles, which I personally have not seen work with btrfs, it complains about missing plugins.
Steps to reproduce
Install docker on a btrfs /var/lib/docker and run any pipeline
Example Project
See here for an example: https://gitlab.com/esbs/updater/-/jobs/525365274
What is the current bug behavior?
We get an error as seen above, ERROR: Job failed (system failure): Error response from daemon: create runner-uzyuuwsf-project-10743483-concurrent-0-cache-3c3f060a0374fc8bc39395164f415a70: error looking up volume plugin btrfs: plugin "btrfs" not found (docker.go:788:15s)
What is the expected correct behavior?
The pipeline runs normally
Output of checks
This bug happens on GitLab.com, using 'gitlab-runner' inside of docker
Environment description
Client:
Debug Mode: false
Server:
Containers: 40
Running: 3
Paused: 0
Stopped: 37
Images: 50
Server Version: 19.03.8
Storage Driver: btrfs
Build Version: Btrfs v4.20.1
Library Version: 102
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
apparmor
seccomp
Profile: default
Kernel Version: 4.19.0-8-amd64
Operating System: Debian GNU/Linux 10 (buster)
OSType: linux
Architecture: x86_64
CPUs: 12
Total Memory: 15.47GiB
Name: race-01
ID: ZLNG:ZCS3:QCRZ:I527:HCUE:QBQD:5MXB:SMBF:T2MN:YNQF:IVF6:3D3S
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
config.toml contents
cat /var/lib/docker/data/gitlab-runner/config.toml
concurrent = 10
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "esbs-runner-01@race-01"
url = "https://gitlab.com"
executor = "docker"
[runners.docker]
tls_verify = false
image = "registry.hub.docker.com/library/alpine:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/run/docker.sock:/run/docker.sock", "/cache"]
volume_driver = "btrfs"
pull_policy = "if-not-present"
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[[runners]]
name = "evbox-runner-01@race-01"
url = "https://gitlab.com"
executor = "docker"
[runners.docker]
tls_verify = false
image = "registry.hub.docker.com/library/alpine:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/run/docker.sock:/run/docker.sock", "/cache"]
volume_driver = "btrfs"
pull_policy = "if-not-present"
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[[runners]]
name = "gitlab-runner-mplab-01@race-01"
url = "https://gitlab.com"
executor = "docker"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "registry.hub.docker.com/library/alpine:latest"
privileged = false
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/run/docker.sock:/run/docker.sock", "/cache"]
volume_driver = "btrfs"
network_mode = "host"
pull_policy = "if-not-present"
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
[[runners]]
name = "signing-runner-01@race-01"
url = "https://gitlab.com"
executor = "docker"
[runners.custom_build_dir]
[runners.docker]
tls_verify = false
image = "registry.hub.docker.com/library/alpine:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/run/docker.sock:/run/docker.sock", "/cache", "/srv/keystore:/srv/keystore"]
volume_driver = "btrfs"
pull_policy = "if-not-present"
shm_size = 0
[runners.cache]
[runners.cache.s3]
[runners.cache.gcs]
Used GitLab Runner version
Running with gitlab-runner 12.10.1 (ce065b93)
Removing the option volume_driver = "btrfs"
seems to fix the issue, and so it seems that previously, this setting was ignored or previous this setting did not matter for 'normal' operation. I do know that defining 'VOLUME" in dockerfiles breaks on btrfs, if the volume was not pre-created. I haven't spend much time on it; just noticed in the past.