Prevent Gitlab-runner from leaving dead containers/volumes

Summary

We have a docker executor running to allow different projects using different technologies to CI/CD. We have had to write a set of small clean up routines to make sure that the system doesn't run out of resources because gitlab-runner does not remove the containers that it uses and if your environment is busy this can add up very quickly.

Proposal

GitLab is by design not automatically clearing this cache because we use it to speed up pipeline execution. That said, this cache can grow large, so there is actually already a script bundled with the runner called /usr/share/gitlab-runner/clear-docker-cache which will handle this for you.

We should make it clear in the Runner documentation that this is the case, and a cron job (or custom scheduled job associated with disk space usage limits) can be created to run this script.

Note that this issue was originally reported as a ~bug, please see below for case where the right approach was unclear/not easy to discover.

Steps to reproduce

Run the shared runner with the following configuration, register the shared runner with gitlab as shared, run a job that uses the runner. A minute after the job is finished you'll see the container in "Exited" state and it remains there.

Configuration:

  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "nodejs:8"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Actual behavior

This is after a week of leaving the projects to do their own thing with the shared resource:

$ docker ps --filter "status=exited" | wc -l 
183

$ docker system prune 
Deleted Containers:
680dde3849fb1f1e291eb495f6f98c77be18fc6b313f4513ae705a17a495a5ba
7db6c62568203505237219bb3a230392725b964e17755eb6df2984e783a66650
ee61a51efd4aadb8d5c9405705f34d87e07eeb9e44560eb0e028147211bd02cb
...
Deleted Images:
untagged: node@sha256:998b099a9790327db5c2808b162862accc609f4ef96de53fbf77b468d2a9ffdb
deleted: sha256:727b047a1f4e9ff0eb663f058577e1c8c1afa357802eda02430a85bb1ea56397
deleted: sha256:cea095fbfd1787818d166a44e978821caf4ad02cab1febf4f4404006333999a7
deleted: sha256:904cf0412e9f4aa456066d58b083a17109aaab5aacdf7074f39dcd4ca1f025f7
...

Total reclaimed space: 912.6MB

Expected behavior

the containers should be removed after the run ends. It would be nice if the container is re-used but it's not so it should be removed.

Environment description

CentOS 7.4.108 Docker version 17.09.1-ce, build 19e2cf6

Used GitLab Runner version

$ /usr/bin/gitlab-runner -v
Version:      10.2.0
Git revision: 0a75cdd1
Git branch:   10-2-stable
GO version:   go1.8.3
Built:        Wed, 22 Nov 2017 09:19:04 +0000
OS/Arch:      linux/amd64

config.toml:
  name ... 
  url ... 
  token ...
  executor = "docker"
  [runners.docker]
    tls_verify = false
    image = "nodejs:8"
    privileged = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0