Update from gitlab-runner to v12.10.1 causes 'plugin "overlay2" not found' error
Since upgrading from v12.9.0 to v12.10.0/v12.10.1, all jobs are failing with the following 'plugin "overlay2" not found' errors. Downgrading from v12.10.0/v12.10.1 to v12.9.0 makes it completely working again, thus I suspect this is a gitlab-runner related issue rather a Docker one (however the gitlab-runner changelog doesn't mention a overlay2 related change thus it feels like a regression due to other changes).
Apr 29 17:32:18 git-ci-2 dockerd: time="2020-04-29T17:32:18.851369617+02:00" level=warning msg="Unable to locate plugin: overlay2, retrying in 1s"
Apr 29 17:32:19 git-ci-2 dockerd: time="2020-04-29T17:32:19.851735245+02:00" level=warning msg="Unable to locate plugin: overlay2, retrying in 2s"
Apr 29 17:32:21 git-ci-2 dockerd: time="2020-04-29T17:32:21.852527189+02:00" level=warning msg="Unable to locate plugin: overlay2, retrying in 4s"
Apr 29 17:32:25 git-ci-2 dockerd: time="2020-04-29T17:32:25.853076064+02:00" level=warning msg="Unable to locate plugin: overlay2, retrying in 8s"
Apr 29 17:32:33 git-ci-2 dockerd: time="2020-04-29T17:32:33.902268712+02:00" level=warning msg="Unable to locate plugin: overlay2, retrying in 1s"
Apr 29 17:32:34 git-ci-2 dockerd: time="2020-04-29T17:32:34.903176200+02:00" level=warning msg="Unable to locate plugin: overlay2, retrying in 2s"
Apr 29 17:32:36 git-ci-2 dockerd: time="2020-04-29T17:32:36.903520945+02:00" level=warning msg="Unable to locate plugin: overlay2, retrying in 4s"
Apr 29 17:32:40 git-ci-2 dockerd: time="2020-04-29T17:32:40.904000575+02:00" level=warning msg="Unable to locate plugin: overlay2, retrying in 8s"
Apr 29 17:32:48 git-ci-2 gitlab-runner[2290]: #033[31;1mERROR: Job failed (system failure): Error response from daemon: create runner-aed6ccdb-project-1-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e: error looking up volume plugin overlay2: plugin "overlay2" not found (docker.go:788:15s)#033[0;m #033[31;1mduration#033[0;m=40.635055319s #033[31;1mjob#033[0;m=53182 #033[31;1mproject#033[0;m=1 #033[31;1mrunner#033[0;m=aed6ccdb
Apr 29 17:32:48 git-ci-2 gitlab-runner: #033[31;1mERROR: Job failed (system failure): Error response from daemon: create runner-aed6ccdb-project-1-concurrent-0-cache-904f6ed42e0fa2b14c1d7a2ed6f1875e: error looking up volume plugin overlay2: plugin "overlay2" not found (docker.go:788:15s)#033[0;m #033[31;1mduration#033[0;m=40.635055319s #033[31;1mjob#033[0;m=53182 #033[31;1mproject#033[0;m=1 #033[31;1mrunner#033[0;m=aed6ccdb
Apr 29 17:32:49 git-ci-2 containerd: time="2020-04-29T17:32:49.189855131+02:00" level=info msg="shim reaped" id=bb8fb8e18ed902f5b9d5484979ca8fa81d6523fdcc4659cf94bc7be73380815c
Apr 29 17:32:49 git-ci-2 containerd: time="2020-04-29T17:32:49.196893423+02:00" level=info msg="shim reaped" id=fd883a3e9f7bf8929840a75fcc7e23ac653d161a9b823b827c7b0a66fd4cbb4e
Apr 29 17:32:49 git-ci-2 containerd: time="2020-04-29T17:32:49.197424083+02:00" level=info msg="shim reaped" id=1ab1bf3c7dd821ef6b78215761427c72d139ca63f5827846b3171c3453d12bdb
Apr 29 17:32:49 git-ci-2 dockerd: time="2020-04-29T17:32:49.200144179+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Apr 29 17:32:49 git-ci-2 dockerd: time="2020-04-29T17:32:49.207001617+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
Apr 29 17:32:49 git-ci-2 dockerd: time="2020-04-29T17:32:49.207108475+02:00" level=info msg="ignoring event" module=libcontainerd namespace=moby topic=/tasks/delete type="*events.TaskDelete"
System is Red Hat Enterprise Linux 7.8 (3.10.0-1127.el7.x86_64) with docker-ce-19.03.8-3.el7.x86_64 and gitlab-runner-12.10.0-1.x86_64 (before gitlab-runner-12.9.0-1.x86_64).
[root@git-ci-2 ~]# docker info
Client:
Debug Mode: false
Server:
Containers: 120
Running: 0
Paused: 0
Stopped: 120
Images: 31
Server Version: 19.03.8
Storage Driver: overlay2
Backing Filesystem: <unknown>
Supports d_type: true
Native Overlay Diff: true
Logging Driver: json-file
Cgroup Driver: cgroupfs
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 7ad184331fa3e55e52b890ea95e65ba581ae3429
runc version: dc9208a3303feef5b3839f4323d9beb36df0a9dd
init version: fec3683
Security Options:
seccomp
Profile: default
Kernel Version: 3.10.0-1127.el7.x86_64
Operating System: Red Hat Enterprise Linux
OSType: linux
Architecture: x86_64
CPUs: 4
Total Memory: 9.598GiB
Name: git-ci-2.example.net
ID: XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX:XXXX
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
[root@git-ci-2 ~]#
[root@git-ci-2 ~]# cat /etc/gitlab-runner/config.toml
concurrent = 1
check_interval = 0
[session_server]
session_timeout = 1800
[[runners]]
name = "xxxxxxxxxxx"
url = "https://git.example.net/"
token = "xxxxxxxxxxxxxxxxxxxxxxxxxxxxxx"
executor = "docker"
[runners.docker]
tls_verify = false
image = "alpine:latest"
privileged = true
disable_entrypoint_overwrite = false
oom_kill_disable = false
disable_cache = false
volumes = ["/certs/client", "/cache"]
volume_driver = "overlay2"
shm_size = 0
[runners.cache]
[root@git-ci-2 ~]#
Edited by Robert Scheck