GitLab Runner doesn't automatically clean up containers after it completes using them and exits
Summary
When using the GitLab CI Runner as a Docker executor, it does not appear to be reaping the containers it creates after a task is done. Glancing at the code, it appears that it's supposed to, but doesn't.
We're currently using a script to regularly clean up the containers because all the leftover containers in Docker wind up occupying all the available network or device mapper resources, causing the machine to lock up and die. However, because we're doing the cleanup on a schedule, and the invocation of CI jobs is erratic and unpredictable (because developers manually invoke some tasks), we are sometimes colliding with the runner and causing jobs to fail.
This boils down to that the GitLab CI Runner needs to remove the exited instance it creates at the end of each job run, rather than leaving them behind. Effectively docker --rm run
instead of docker run
.
Steps to reproduce
- Set up GitLab CI shared docker runner
- Set up cleanup script that runs the following command every 5 minutes (
docker rm $(docker ps -qa --no-trunc --filter "status=exited") || true
) - Run CI jobs on a GitLab instance
Actual behavior
GitLab CI Runner runs, leaves behind stuff, and then the script eventually reaps them. Sometimes, this will happen at unexpected times, causing weird job failures.
Expected behavior
GitLab CI Runner runs jobs and removes the container instances when it's done.
Relevant logs and/or screenshots
This shows up at the end of jobs when they fail:
ERROR: Job failed (system failure): persistent connection closed
When the system runs out of resources, this error shows:
ERROR: Preparation failed: Cannot connect to the Docker daemon at unix:///var/run/docker.sock. Is the docker daemon running?
Environment description
[~]$ sudo docker info
Containers: 6
Running: 1
Paused: 0
Stopped: 5
Images: 84
Server Version: 1.12.6
Storage Driver: btrfs
Build Version: Btrfs v4.4.1
Library Version: 101
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: null bridge host overlay
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-514.21.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 8
Total Memory: 7.638 GiB
Name: gitlab-c7-docker-runner
ID: IUL5:JUGX:V7XK:W575:QRJK:ITVY:2RM3:DROO:ECZ2:Y3MS:JN5E:OMX4
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8
Registries: docker.io (secure)
[~]$ sudo docker info
Containers: 24
Running: 0
Paused: 0
Stopped: 24
Images: 7
Server Version: 1.12.6
Storage Driver: devicemapper
Pool Name: docker-253:0-67737945-pool
Pool Blocksize: 65.54 kB
Base Device Size: 10.74 GB
Backing Filesystem: xfs
Data file: /dev/loop0
Metadata file: /dev/loop1
Data Space Used: 1.421 GB
Data Space Total: 107.4 GB
Data Space Available: 44.63 GB
Metadata Space Used: 3.428 MB
Metadata Space Total: 2.147 GB
Metadata Space Available: 2.144 GB
Thin Pool Minimum Free Space: 10.74 GB
Udev Sync Supported: true
Deferred Removal Enabled: false
Deferred Deletion Enabled: false
Deferred Deleted Device Count: 0
Data loop file: /var/lib/docker/devicemapper/devicemapper/data
WARNING: Usage of loopback devices is strongly discouraged for production use. Use `--storage-opt dm.thinpooldev` to specify a custom block storage device.
Metadata loop file: /var/lib/docker/devicemapper/devicemapper/metadata
Library Version: 1.02.135-RHEL7 (2016-11-16)
Logging Driver: journald
Cgroup Driver: systemd
Plugins:
Volume: local
Network: host bridge overlay null
Swarm: inactive
Runtimes: docker-runc runc
Default Runtime: docker-runc
Security Options: seccomp selinux
Kernel Version: 3.10.0-514.26.2.el7.x86_64
Operating System: CentOS Linux 7 (Core)
OSType: linux
Architecture: x86_64
Number of Docker Hooks: 2
CPUs: 8
Total Memory: 7.638 GiB
Name: gitlab-c7-docker-runner-2
ID: X3QX:R2KS:PZT2:62NF:LLYX:HOPU:CROY:Q2NX:X4VP:6SNA:YJX4:MNZB
Docker Root Dir: /var/lib/docker
Debug Mode (client): false
Debug Mode (server): false
Registry: https://index.docker.io/v1/
Insecure Registries:
127.0.0.0/8
Registries: docker.io (secure)
Used GitLab Runner version
[~]$ gitlab-runner --version
Version: 9.4.1
Git revision: d24b11c
Git branch: 9-4-stable
GO version: go1.8.3
Built: Tue, 25 Jul 2017 12:04:47 +0000
OS/Arch: linux/amd64
[~]$ gitlab-runner --version
Version: 9.4.2
Git revision: 6d06f2e
Git branch: 9-4-stable
GO version: go1.8.3
Built: Wed, 02 Aug 2017 12:46:17 +0000
OS/Arch: linux/amd64