Smart cache cleanup for Docker images & volumes
Status update 2023-10-09
The clear-docker-cache script was added to the runner Docker executor in Runner 13.9. Refer to this MR for additional details.
If you are running into the problems described in this issue, please confirm if the clear-docker-cache is not addressing the problem in your scenario and also include why its not working.
Problem
It has been pointed out in several issues and tickets that users and customers are running out of disk space on their Runner hosts because the caching for the GitLab Runner seems to continue to consume disk space until an admin manually intervenes.
Currently we have a small documentation section that suggests running the clear-docker-cache script included with the runner on a weekly cron.
But this script only cleans up images, not volumes. And it clears up images indiscriminately. Which means that builds will be slower for a while until caches for most-used resources are rebuilt. Ideally we would only evict the least recently used resources.
I found another tool here: GitLab.org / gitlab-runner-docker-cleanup. But it isn't being maintained.
Proposal
A smart cache cleanup mechanism included in the Runner.
Links to related issues and merge requests / references
GitLab Runner doesn't automatically clean up containers after it completes using them and exits