Prevent Gitlab-runner from leaving dead containers/volumes
We have a docker executor running to allow different projects using different technologies to CI/CD. We have had to write a set of small clean up routines to make sure that the system doesn't run out of resources because gitlab-runner does not remove the containers that it uses and if your environment is busy this can add up very quickly.
GitLab is by design not automatically clearing this cache because we use it to speed up pipeline execution. That said, this cache can grow large, so there is actually already a script bundled with the runner called
/usr/share/gitlab-runner/clear-docker-cache which will handle this for you.
We should make it clear in the Runner documentation that this is the case, and a cron job (or custom scheduled job associated with disk space usage limits) can be created to run this script.
Note that this issue was originally reported as a bug, please see below for case where the right approach was unclear/not easy to discover.
Steps to reproduce
Run the shared runner with the following configuration, register the shared runner with gitlab as shared, run a job that uses the runner. A minute after the job is finished you'll see the container in "Exited" state and it remains there.
executor = "docker" [runners.docker] tls_verify = false image = "nodejs:8" privileged = false disable_cache = false volumes = ["/cache"] shm_size = 0
This is after a week of leaving the projects to do their own thing with the shared resource:
$ docker ps --filter "status=exited" | wc -l 183 $ docker system prune Deleted Containers: 680dde3849fb1f1e291eb495f6f98c77be18fc6b313f4513ae705a17a495a5ba 7db6c62568203505237219bb3a230392725b964e17755eb6df2984e783a66650 ee61a51efd4aadb8d5c9405705f34d87e07eeb9e44560eb0e028147211bd02cb ... Deleted Images: untagged: node@sha256:998b099a9790327db5c2808b162862accc609f4ef96de53fbf77b468d2a9ffdb deleted: sha256:727b047a1f4e9ff0eb663f058577e1c8c1afa357802eda02430a85bb1ea56397 deleted: sha256:cea095fbfd1787818d166a44e978821caf4ad02cab1febf4f4404006333999a7 deleted: sha256:904cf0412e9f4aa456066d58b083a17109aaab5aacdf7074f39dcd4ca1f025f7 ... Total reclaimed space: 912.6MB
the containers should be removed after the run ends. It would be nice if the container is re-used but it's not so it should be removed.
CentOS 7.4.108 Docker version 17.09.1-ce, build 19e2cf6
Used GitLab Runner version
$ /usr/bin/gitlab-runner -v Version: 10.2.0 Git revision: 0a75cdd1 Git branch: 10-2-stable GO version: go1.8.3 Built: Wed, 22 Nov 2017 09:19:04 +0000 OS/Arch: linux/amd64 config.toml: name ... url ... token ... executor = "docker" [runners.docker] tls_verify = false image = "nodejs:8" privileged = false disable_cache = false volumes = ["/cache"] shm_size = 0