Gitlab-runner leaves dead containers/volumes
We have a docker executor running to allow different projects using different technologies to CI/CD. We have had to write a set of small clean up routines to make sure that the system doesn't run out of resources because gitlab-runner does not remove the containers that it uses and if your environment is busy this can add up very quickly.
Steps to reproduce
Run the shared runner with the following configuration, register the shared runner with gitlab as shared, run a job that uses the runner. A minute after the job is finished you'll see the container in "Exited" state and it remains there.
executor = "docker" [runners.docker] tls_verify = false image = "nodejs:8" privileged = false disable_cache = false volumes = ["/cache"] shm_size = 0
This is after a week of leaving the projects to do their own thing with the shared resource:
$ docker ps --filter "status=exited" | wc -l 183 $ docker system prune Deleted Containers: 680dde3849fb1f1e291eb495f6f98c77be18fc6b313f4513ae705a17a495a5ba 7db6c62568203505237219bb3a230392725b964e17755eb6df2984e783a66650 ee61a51efd4aadb8d5c9405705f34d87e07eeb9e44560eb0e028147211bd02cb ... Deleted Images: untagged: node@sha256:998b099a9790327db5c2808b162862accc609f4ef96de53fbf77b468d2a9ffdb deleted: sha256:727b047a1f4e9ff0eb663f058577e1c8c1afa357802eda02430a85bb1ea56397 deleted: sha256:cea095fbfd1787818d166a44e978821caf4ad02cab1febf4f4404006333999a7 deleted: sha256:904cf0412e9f4aa456066d58b083a17109aaab5aacdf7074f39dcd4ca1f025f7 ... Total reclaimed space: 912.6MB
the containers should be removed after the run ends. It would be nice if the container is re-used but it's not so it should be removed.
CentOS 7.4.108 Docker version 17.09.1-ce, build 19e2cf6
Used GitLab Runner version
$ /usr/bin/gitlab-runner -v Version: 10.2.0 Git revision: 0a75cdd1 Git branch: 10-2-stable GO version: go1.8.3 Built: Wed, 22 Nov 2017 09:19:04 +0000 OS/Arch: linux/amd64 config.toml: name ... url ... token ... executor = "docker" [runners.docker] tls_verify = false image = "nodejs:8" privileged = false disable_cache = false volumes = ["/cache"] shm_size = 0