Prevent Gitlab-runner from leaving dead containers/volumes

changed the description

This seems to be a relevant issue, since I have to run a script to clean every 5 minutes on my shared runner.

Possibly related and closed, but still with recent "Me too" comments: #151 (closed), #2658 (closed)

This happens to me too. Facing a very similar situation as the mkdir example in the comments of #2658 (closed)

Only noticed this happening after switching from using gitlabs shared runners to a self-hosted runner (which, in this context, makes sense)

Yes, this is a serious problem that filled up all the disk space on our private runner. I had to add the infamous cronjob to clean up, but that does not feel right :-(

would you mind sharing the shell script for your cron job?

A simple docker system prune -a -f on a weekly cronjob is what I'm using now (see docs). I've got a single gitlab-runner server hosting all our dockerized gitlab runners and was running into the same issue. This works in a pinch assuming you don't leave unused images/containers/networks/volumes lying around that you intend to re-use. Easy enough when you have a dedicated build server. That said though, I'd very much like a better option as well, precisely how I found this issue.

Any updates from GitLab on this???

We have an 1,000 seat prospect that is affected by this. Are there any updates?

link: https://gitlab.my.salesforce.com/00161000008qZHo?srPos=0&srKp=001

cc: @philcamillo

marked this issue as related to #151 (closed)

marked this issue as related to #2658 (closed)

@ayufan and @erushton this seems to be incorrect behavior - one of the links indicates there may be code to clean this up that is not working. Can you please take a look and determine LOE to resolve this?

@jlenny @kristmcg By design, we preserve some number of container/volumes for cache purposes, they are named runner-ID-cache-HASH. We use that to speed-up subsequent runs on the same machine. The specific of GitLab Runner can be disabled with disable_cache=true in config.toml.

Without knowing what containers are left I assume that these are caches.

Should it be possible for the container/volume cache to fill the entire disk?

@jlenny , docker has limited support for storage quota https://docs.docker.com/engine/reference/commandline/run/#set-storage-driver-options-per-container , but AFAIK gitlab runner doesn't make use of it

@jlenny Yes, it is possible. Runner does not remove cache volumes. In the past I had the https://gitlab.com/gitlab-org/gitlab-runner-docker-cleanup to help with that as a side tool, but we are not actively maintaining that.

@redbaron1 It helps in the scope of single build, but it does not help if you process many projects as it can still run out of the memory.

added ~2278654 devopsverify labels

@ayufan Can you explain more about "single build" , I'm not quite understanding? How does disable_cache = true have an impact or not in a scenario where there are many Gitlab Runners across several VM's ? And where there might be multiple projects/pipelines but no re-use between each project that gets built.

Experiencing this as well. The machine where my build runner runs on has a very limited storage. It builds the qt lib from scratch which occupies about 3GB of space per build. That means, I can only run my ci pipeline once before I have do manually clean the cached volumes.

Ultimate customer reporting this issue: ZD - https://gitlab.zendesk.com/agent/tickets/104009 (Internal Only)

added customer label

@ayufan @jlenny In the meantime, any tips for our users and customers who are plagued by this issue?

added direction label

@tmaczukin Can you help here?

We faced with this issue and trying to remove dead container volumes manually.

First let's describe how containers are handled by Runner, since this will put a light on why some containers are left.

Runner, when docker executor is used, creates several containers to handle one job:

the main container, where user's script is executed,
containers for all specified services,
a number of containers with predefined in name, where git clone/fetch, artifacts and cache operations are done,
a container with cache in name, where build's directory is stored for use in future job. Thanks to this container one may use a fetch strategy in future jobs, without it, docker executor would support only clone strategy and would not preserve files created during previous jobs (which may be useful for the user; this tires to replicate the way how Runner with shell executor may be used). There may be also other containers with -cache- in name, if the user will connect a volumes via volumes = [] setting, but not specify a host directory - then a cache container will be created to store the content of such volume.

From all these containers, all - except the *-cache-* containers are removed. When knowing what cache containers are for, it's obvious that they may not be removed. Otherwise they would not give the value that we expect from them (to preserve data between jobs).

It's important to know, that these cache containers have nothing to do with cache defined in .gitlab-ci.yml nor the remote cache configuration (S3, GCS) from config.toml. Both features can be configured and used independently. These are just cache mechanisms for two separated layers of the system.

In some cases (probably like the one here) user may decide that he don't want to use the docker cache feature (and I think this is the case that creates the problem of this issue). It may be disabled with disable_cache = true in [runners.docker] section for a specific Runner. The setting and the difference of 2 caching layers available for docker executor are described at https://docs.gitlab.com/runner/configuration/advanced-configuration.html#the-runners-docker-section.

And for example - this is how a list of exited containers looks on my local machine, after I started a Runner and executed four jobs:

f1e5b79deef6  Exited (0) 2 minutes ago runner-c8d11a2a-project-8472090-concurrent-2-cache-15a0ecfd391da7edde863f3b2ea17397
8a746c95a67b  Exited (0) 3 minutes ago runner-c8d11a2a-project-8472090-concurrent-1-cache-15a0ecfd391da7edde863f3b2ea17397
8f70cc6eaacd  Exited (0) 3 minutes ago runner-c8d11a2a-project-8472090-concurrent-0-cache-15a0ecfd391da7edde863f3b2ea17397

Only cache containers are present. Rest of containers used for the job was removed.

At #2980 (comment 91513853) @ayufan wrote:

Without knowing what containers are left I assume that these are caches.

I don't see any clear statement that would confirm or deny that this is the case, so I'm assuming that it is. Normally no containers - except *-cache-* ones - should be left and this is how Runner works on my machine and on other machines I'm managing. And for that the disable_cache setting is the way to go (again, this doesn't affect caches defined in .gitlab-ci.yml).

Apart of this there is also the tool that @ayufan mentioned: https://gitlab.com/gitlab-org/gitlab-runner-docker-cleanup.

I think that what's mostly expected by all people here would be to make this tool a part of Runner. After refactoring this tool and putting it inside of Runner there are at least two possibilities:

We may start this as a separated goroutine at Runner's startup, but only when Docker or Docker Machine executor is used. In that case the tool behaves just like it behaves now, but one doesn't need to install and configure another tool - all is placed in one place.
We may make it a one-shot mechanism that would be started on each job start. As part of prepare process we would just execute it to try to proclaim some free space (if configured requirements of used space are met).

The first one makes the job start faster, while the second one saves a little CPU power since it's started only when a free space may be needed.

But in any of these two cases, it should ensure that:

the maintenance of cleanup mechanism gets the same level of attention as the Runner's maintenance (since it's now a part of Runner),
in case of changes in Runner it would be easier to remember about updating the cleanup mechanism.

Implementing this inside of Runner may be a little tricky for Docker Machine, since Docker is not available locally there, but it should be not hard to make it working also for this executor.

So, as a summary I propose to:

Extend our documentation to clearly describe a difference between different cache mechanisms that Runner is using, so the related configuration options will not confuse users.
Look on how to make https://gitlab.com/gitlab-org/gitlab-runner-docker-cleanup an internal part of Runner.

added to epic &453 (closed)

added 1 deleted label and removed 1 deleted label

added Category:Continuous Integration label

@tmaczukin Thanks for the long writeup. I also came here, like others, because I was running out of disk space on my GitLab Runner server. We are a paying GitLab EE customer.

This is what really helped:

$ docker volume prune 
WARNING! This will remove all local volumes not used by at least one container.
Are you sure you want to continue? [y/N] y
Deleted Volumes:
4578154e8b7ee9add613c07f94b84c1275b82c7beceb99491925482045d660bc
2e9717177c0a7a0ce802c702da51df5e2ed8b63b89813624bbeef0d649b39685
a4d57233b5074720a3d29b080f8de656b5586e5d44512f918b3a935b71f831cf
...

Total reclaimed space: 96.78GB

You speak about containers used for caching in your text - but how about these volumes? Are they also involved with the caching, or what is their purpose? Or is this just a bug, that containers are removed but their volumes are left dangling? (since docker volume prune only removes unused volumes, I get the feeling that the latter is the case.)

For reference, here is a part of my config.toml - as can be seen, I have a cache volume enabled.

[[runners]]
  name = "gitlab-ci-1"
  url = "https://some-gitlab-url/"
  token = "some-gitlab-token"
  executor = "docker"

  [runners.docker]
    tls_verify = false
    image = "openjdk:11-jdk-slim-sid"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0
    limit = 1
    pull_policy = "if-not-present"

added [Deprecated] Category:Runner label

FWIW, when updating gitlab-runner today it removed a bunch of stale Docker containers

Setting up gitlab-runner (11.5.1) ...
GitLab Runner: detected user gitlab-runner
Runtime platform                                    arch=amd64 os=linux pid=1184 revision=7f00c780 version=11.5.1
gitlab-runner: Service is not running.
Runtime platform                                    arch=amd64 os=linux pid=1195 revision=7f00c780 version=11.5.1
gitlab-ci-multi-runner: Service is not running.
Runtime platform                                    arch=amd64 os=linux pid=1232 revision=7f00c780 version=11.5.1
Runtime platform                                    arch=amd64 os=linux pid=1300 revision=7f00c780 version=11.5.1
Clearing docker cache...
722d3dd9348e
1384d335aa3c
cd9d02fac7a6
ee24643a1589
b26a4ab66683
...

My disk space on /var went down from about 87% to 3% in the process.

Maybe there was a bug here that got fixed in the recent gitlab-runner update? If anyone knows more about the issue, please enlighten us.

@grzesiek did we expect the change above?

ZD https://gitlab.zendesk.com/agent/tickets/110943 (internal)

mentioned in issue #2658 (closed)

@jlenny I digged a bit further into this: the reason why my /var usage shrank drastically is because it executed /usr/share/gitlab-runner/clear-docker-cache.

I ran this command manually now; went down from about 85% to 9% usage on my /var volume.

Maybe the solution/workaround in this case would be to purge the Docker cache at a configurable interval?

@ayufan do you have thoughts on why the above fix worked? Could that be part of a solution to this issue?

@jlenny

This is done by design: we preserve cache/build volumes in form of containers/volumes as @tmaczukin described.

The script can be used to clean-up these containers. This is not ~bug, but rather a specification of the system and Runner should not manage these containers. This should be done externally. What we could do is to simply propose to run periodically (simple cronjob on weekend?) this script bundled with Runner to clean up the containers periodically and get rid of stale caches.

marked #2658 (closed) as a duplicate of this issue

@ayufan Thanks for clearing things up. Maybe then this can be solved by adding some notes about this in the documentation. I set it up in /etc/cron.weekly/gitlab-clear-docker-cache like this, very trivial:

#!/bin/sh

/usr/share/gitlab-runner/clear-docker-cache

I think for most installations, clearing the cache once a week should be a fine balance between "maintain cache" and "don't fill up the disk".

changed the description

All right, then what I'm going to do is convert this into a ~Documentation task so that the process for setting this up is clear and discoverable.

I have updated the description above with a proposal indicating that this is a documentation update, and assigned to %11.8 for resolution (cc @axil)

If anyone on this thread feels like this is not a solution to this issue, please give a shout and lets discuss.

changed milestone to %11.8

added documentation label

I think for most installations, clearing the cache once a week should be a fine balance between "maintain cache" and "don't fill up the disk".

This is I think very good approach. Sometimes you might not want to do it, as on each Monday the rebuild will take longer :)

Cool! I agree that this needs to be documented along with what Tomasz wrote in #2980 (comment 106845694).

For reference what the script does: https://gitlab.com/gitlab-org/gitlab-runner/blob/master/packaging/root/usr/share/gitlab-runner/clear-docker-cache.

I'll cc @eread who's on top of ~Verify.

I'm also one of those people with 0 3 * * * /usr/bin/docker system prune -af in my crontab on several runners, which worked for a while. But since the volume of jobs has gone up, it's something I'd have to do every hour or so, so I went looking for better solutions.

This is a dedicated gitlab-runner machine with 20 GBs of space. Currently completely full.

/usr/share/gitlab-runner/clear-docker-cache doesn't do anything, presumably because gitlab-runner-docker-cleanup is also running.
docker volume prune cleaned up 4.7gb worth of volumes, there's still 15 gigs of something.

When I go into /var/lib/docker, it's the overlay2 directory that's massive, even with the cleanups above. The only thing that cleans that up is a docker system prune -af. So after searching for a more elegant solution, I've decided to 1: resize my machines and 2: change that cron job to run every 30 minutes :).

Happy to learn how to do it better, but this will probably work for a while.

changed epic to &573 (closed)

changed milestone to %11.9

removed 1 deleted label

mentioned in issue #4191 (closed)

In case this helps anyone else ... I run the following every hour ... so if the disk in question is more than 70% full then the clean up runs. Feel free to adjust as needed, but you can run it as often as you like (and your system can handle).

Adjust: DRIVE should be the drive that either is mounted as /var/lib/docker or contains it WATERLINE is the highest level you want the drive to be filled up before you run the cleaner.

This is my /etc/cron.hours/cleanUpAfterGitLabRunner:

#!/bin/sh

DRIVE=${1:-sdb1}
WATERLINE=${2:-70}
# echo $DRIVE $WATERLINE
[ `df -h | awk -v pat="$DRIVE" '$0~pat{print $5 }' | cut -d\% -f1` -gt ${WATERLINE} ] && docker system prune -af > /dev/null 2>&1

Here is my version of the same:

#!/bin/sh

LIMIT=70
DRIVE=/dev/vdb

df -H | grep -vE '^Filesystem|$DRIVE' | awk '{ print $5 " " $1 }' | while read output;
do
  #echo $output
  size=$(echo $output | awk '{ print $1}' | cut -d'%' -f1  )
  partition=$(echo $output | awk '{ print $2 }' )
  if [[ $size -ge $LIMIT ]] && [[ $partition == $DRIVE ]]; then
    echo "$(date): $(hostname) reached the space limit \"$partition ($size%)\" pruning now.." \
      && docker system prune -af > /dev/null 2>&1
  fi
done

So should we update https://docs.gitlab.com/runner/executors/docker.html to mention clear-docker-cache might need to be run regularly? Is the scope of this limited to just the Docker executor? Would an update there catch everyone

And what about folks that need to run docker system prune? Is that a problem with the script.

CC @steveazz

So should we update https://docs.gitlab.com/runner/executors/docker.html to mention clear-docker-cache might need to be run regularly?

@eread Ideally yes, it can be used for kubernetes I imagine but that gets complex since the runner is deployed in the pod so it doesn't have privelaged access to the docker daemon so it really depends on the user set up. I think leaving it under the Docker executor is good.

And what about folks that need to run docker system prune? Is that a problem with the script.

It is not a problem with the script because the script does not clean build/cache volumes as specified in #2980 (comment 131281466) & #2980 (comment 106845694) so we can suggest docker system prune for a more aggressive approach but explain to the user of the downsides in doing so.

changed milestone to %12.0

assigned to @eread

mentioned in merge request !1390 (merged)

Please take a look at !1390 (merged), which is an MVC attempt to address the concerns of the issue.

closed via merge request !1390 (merged)

mentioned in commit 3e07892e

mentioned in merge request gitlab-com/www-gitlab-com!23143 (merged)

changed title from Gitlab-runner leaves dead containers/volumes to Prevent Gitlab-runner from leaving dead containers/volumes

removed Category:Continuous Integration label

Instead of arbitrarily removing cache containers, it'd have been nice to:

expire caches after X time
remove least frequently used caches when approaching a threshold

mentioned in issue #4765 (closed)

Below is a script I've just created, from a starting point of @awm's script above. There are a few differences:

Supports thresholds for both free space percentage and free inode percentage (defaults to 70% for each)
Dynamically determines the filesystem containing /var/lib/docker
Prunes only containers, images, and volumes (some may prefer to use a single docker system prune)
Can attempt to initially prune only Docker elements older than a given threshold in days - if that doesn't return enough disk then we resort to pruning everything

This could be enhanced to only prune volumes whose names match the 'runner-...' naming used by recent versions of GitLab runner.

#!/usr/bin/env sh

FREE_SPACE_PERCENT_THRESHOLD=${1:-70} # Disk space % usage threshold
FREE_INODE_PERCENT_THRESHOLD=${2:-70} # Inode % usage threshold
PRUNE_FILTER_UNTIL_DAYS=${3:-30}      # Initially prune only this many days
USE_PRUNE_FILTER_UNTIL=Y              # Requires Docker CLI 1.14.0+ and API v1.26+
DOCKER_STORAGE_DIR=/var/lib/docker

get_docker_storage_usage() {
    METRIC=$1
    echo $(df $DOCKER_STORAGE_DIR --output=$METRIC | tail -n -1 | grep -P -o "\d+")
}

prune_is_required() {
    [ $(get_docker_storage_usage pcent) -gt $FREE_SPACE_PERCENT_THRESHOLD ] || [ $(get_docker_storage_usage ipcent) -gt $FREE_INODE_PERCENT_THRESHOLD ] && return 0
}

do_docker_prune() {
    DOCKER_PRUNE_FILTER="$([ "$1" != "" ] && echo "--filter \"until=$(($1*24))h\"")"
    docker container prune -f "$DOCKER_PRUNE_FILTER" &> /dev/null
    docker image prune -af "$DOCKER_PRUNE_FILTER" &> /dev/null
    docker volume prune -f "$DOCKER_PRUNE_FILTER" &> /dev/null
}

if prune_is_required; then
    echo "The filesystem containing $DOCKER_STORAGE_DIR exceeded a disk usage threshold; pruning docker elements"
    if [ "$USE_PRUNE_FILTER_UNTIL" = "Y" ]; then
        # Initially prune only items older than the filter in days
        do_docker_prune $PRUNE_FILTER_UNTIL_DAYS
	# If that still hasn't freed enough space, omit the filter
        prune_is_required && do_docker_prune
    else
        # Prune everything
        do_docker_prune
    fi
fi

Based on the script by @ashleyghooper (thanks a lot!), I made the following modified version. There were a few problems with the previous implementation:

The $DOCKER_PRUNE_FILTER string should not be quoted inside a string. If it is, the Docker process will receive --filter "until as a single parameter, instead of separate parameters.
docker volume prune doesn't support the same kind of filters that the container prune and image prune commands, so the filtering was removed there altogether.
#!/usr/bin/env sh is incorrect for this script, which uses bashisms like &>.

(The first of these can easily be seen by disabling the &> /dev/null redirection. This means that in essence, the filtering would never work; it would always fall back to the full pruning.)

#!/usr/bin/env bash

FREE_SPACE_PERCENT_THRESHOLD=${1:-70} # Disk space % usage threshold
FREE_INODE_PERCENT_THRESHOLD=${2:-70} # Inode % usage threshold
PRUNE_FILTER_UNTIL_DAYS=${3:-30}      # Initially prune only this many days
USE_PRUNE_FILTER_UNTIL=Y              # Requires Docker CLI 1.14.0+ and API v1.26+
DOCKER_STORAGE_DIR=/var/lib/docker

get_docker_storage_usage() {
    METRIC=$1
    echo $(df $DOCKER_STORAGE_DIR --output=$METRIC | tail -n -1 | grep -P -o "\d+")
}

prune_is_required() {
    [ $(get_docker_storage_usage pcent) -gt $FREE_SPACE_PERCENT_THRESHOLD ] || [ $(get_docker_storage_usage ipcent) -gt $FREE_INODE_PERCENT_THRESHOLD ] && return 0
}

do_docker_prune() {
    DOCKER_PRUNE_FILTER="$([ "$1" != "" ] && echo "--filter until=$(($1*24))h")"
    docker container prune -f $DOCKER_PRUNE_FILTER &> /dev/null
    docker image prune -af $DOCKER_PRUNE_FILTER &> /dev/null
    docker volume prune -f &> /dev/null
}

if prune_is_required; then
    echo "The filesystem containing $DOCKER_STORAGE_DIR exceeded a disk usage threshold; pruning docker elements"
    if [ "$USE_PRUNE_FILTER_UNTIL" = "Y" ]; then
        # Initially prune only items older than the filter in days
        do_docker_prune $PRUNE_FILTER_UNTIL_DAYS
        # If that still hasn't freed enough space, omit the filter
        prune_is_required && do_docker_prune

        echo "Disk usage after cleanup: $(get_docker_storage_usage pcent)%"
    else
        # Prune everything
        do_docker_prune
    fi
fi

Thanks @perlun. Good catch on the quoting - I don't exactly remember specifics of my testing, but I think I'd wanted to avoid pruning a lot of hard-earned cache :), so stopped short of actually allowing the docker ... prune invocations, probably prefixing them with echo instead.

I also hadn't realised &> was a bashism; I normally try to avoid making scripts dependent on bash, but for some things it's better just to use it. I guess it'd be possible to revert to using > /dev/null 2>&1, but I think I'll just use your version.

@perlun, for me, in the context of gitlab-runner caches, being able to selectively remove volumes was the most important thing, so how about something like the below function? It could also be used in place of docker prune for other types if desired.

docker_remove_by_age() {
    TYPE=${1:-volume}
    for ITEM_ID in $(docker $TYPE ls -q); do
        # 'docker volume inspect' uses "CreatedAt", whereas container and image use "Created"
        CREATED_AT=$(docker $TYPE inspect $ITEM_ID | sed -n -e 's/"Created\(At\)\?": "\(.\+\)",/\2/p' )
        ITEM_EPOCH=$(date --date="$CREATED_AT" +%s)
        ITEM_AGE_DAYS=$(( ( $(date +%s) - $ITEM_EPOCH ) / 86400 ))
        if [[ $ITEM_AGE_DAYS -gt $PRUNE_FILTER_UNTIL_DAYS ]]; then
            docker $TYPE rm -f $ITEM_ID
        fi
    done
}

(Unfortunately for us, the version of Docker on our gitlab-runner servers (1.13.1) doesn't seem to include CreatedAt in the docker volume inspect output, so I can't actually use this myself yet).

Of course it'd be nice if docker would just add the missing filters for docker volume prune.

The ITEM_EPOCH= line could of course be replaced by another subshell within the ITEM_AGE_DAYS= calculation.

NB: For other Docker types like containers and images, which return much larger JSON objects, there is a small risk of the sed expression matching something that's not the creation date of the item itself, but rather another, nested Created or CreatedAt key. This could be avoided by using jq or similar, but trying to keep it simple here. docker volume inspect returns a flat JSON object.

Here's a version for anybody stuck on a really old Docker version, like us:

#!/usr/bin/env bash

FREE_SPACE_PERCENT_THRESHOLD=${1:-70} # Disk space % usage threshold
FREE_INODE_PERCENT_THRESHOLD=${2:-70} # Inode % usage threshold
PRUNE_FILTER_UNTIL_DAYS=${3:-30}      # Initially prune only this many days
USE_PRUNE_FILTER_UNTIL=Y              # Requires Docker CLI 1.14.0+ and API v1.26+
DOCKER_STORAGE_DIR=/var/lib/docker

get_docker_storage_usage() {
    METRIC=$1
    echo $(df $DOCKER_STORAGE_DIR --output=$METRIC | tail -n -1 | grep -P -o "\d+")
}

prune_is_required() {
    [ $(get_docker_storage_usage pcent) -gt $FREE_SPACE_PERCENT_THRESHOLD ] || [ $(get_docker_storage_usage ipcent) -gt $FREE_INODE_PERCENT_THRESHOLD ] && return 0
}

purge_runner_docker_volumes() {
    find $DOCKER_STORAGE_DIR/volumes -maxdepth 1 -type d -name 'runner-*' -mtime +$1 -printf ' %f' | xargs --no-run-if-empty docker volume rm -f
}

do_docker_prune() {
    DOCKER_PRUNE_FILTER="$([ "$1" != "" ] && echo "--filter until=$(($1*24))h")"
    docker container prune -f $DOCKER_PRUNE_FILTER &> /dev/null
    docker image prune -af $DOCKER_PRUNE_FILTER &> /dev/null
    purge_runner_docker_volumes $1
}

if prune_is_required; then
    echo "The filesystem containing $DOCKER_STORAGE_DIR exceeded a disk usage threshold; pruning docker elements"
    if [ "$USE_PRUNE_FILTER_UNTIL" = "Y" ]; then
        # Initially prune only items older than the filter in days
        do_docker_prune $PRUNE_FILTER_UNTIL_DAYS
        # If that still hasn't freed enough space, omit the filter
        prune_is_required && do_docker_prune

        echo "Disk usage after cleanup: $(get_docker_storage_usage pcent)%"
    else
        # Prune everything
        do_docker_prune
    fi
fi

@perlun, for me, in the context of gitlab-runner caches, being able to selectively remove volumes was the most important thing, so how about something like the below function? It could also be used in place of docker prune for other types if desired.

Yeah, something like that could work. In my case, the volumes are not "independent" (they are connected to the containers) so once the container is removed, they should be able to be safely removed for us. And before the container is removed, the docker volume prune won't touch them anyway.

(Unfortunately for us, the version of Docker on our gitlab-runner servers (1.13.1) doesn't seem to include CreatedAt in the docker volume inspect output, so I can't actually use this myself yet).

Sorry to hear that (that's a very old version, isn't it?). We have our CI runners mostly set up as "cattle", managing them with Puppet and Ansible, so we can easily run a fairly recent version of Docker on them.

Besides cleaning up volumes and something the above scripts don't take into account is many projects use old images and docker itself only keeps track of the created date for an image.. which is often months or years old thus flushing all the utilities and layers we required for fast jobs and hammering our docker repository every time we flushed the runners. What we found to be the best solution for image management is an LRU eviction cache that watches docker events, tracks the timestamp everytime an image is used and only does anything once a disk threshold has passed. This lets you get the most out of your docker image cache based on how much space you are willing to allocate on your runners.

https://github.com/stepchowfun/docuum Docuum is used by Airbnb on its fleet of 1.5k+ CI workers.

https://github.com/stepchowfun/docuum

Thanks for that pointer Andrew!

For GitLab: it would be great that LRU and automatic management of the disk space would be at least an option or even better the default for docker runners? can this be prioritized? we shouldn't have to manually clean/manage disk space or install third party packages to do that?

You bet! Hope it helps..

It would probably be best to open a new issue though if you want Gitlab to possibly action anything since this one has been met with the 'Closed' hammer...

Having such a feature integrated into the runner process itself does not seem too far fetched to me. The other project is written in Rust tho and Gitlab Runner is Golang.. so it's a start from scratch effort.. but the implementation seems pretty simple overall.

The difficulty is convincing the runner team it's needed or something they should maintain.. and even then I wouldn't hold my breath on getting it for a year or two...

A lot of people are facing this issue and it doesn't seem to the team possible to implement it? Strange to force users to invent the same band-aids, again and again, thousands of times.

mentioned in issue #27332

I created an issue proposing smart runner cache management here: Smart cache cleanup for Docker images & volumes.

If this is important to you or your organization then please upvote that issue and feel free to add to the discussion if you have anything to share.

We ended up for now with the docuum and a cron script.

mentioned in issue gitlab#297223

It's been 4 years but still the Runner's Team doesn't care about solving this issue. What's the point in publishing new features if old features are riddled with BUGS

I would not say that this is Gitlab Runner's bug. Docker is a separate component that gitlab-runner can use to run your CI jobs. Docker system fills up with used and unused images and overlays, this can be caused by gitlab-runner or by any other user or software that uses docker. To clean that up from gitlab-runner software, it does sound like a feature or improvement.

Add below cron job to your server where your gitlab-runner is setup, use https://crontab.guru/ to help you define the frequency you need.

1 23 * * * /usr/bin/docker system prune -a -f

There has been a feature request that perhaps you should follow Smart cache cleanup for Docker images & volumes (#27332)

mentioned in issue gitlab-org/database-team/gitlab-com-database-testing#97 (closed)

If you are using the kubernetes executor to run docker build, I've written an article on how to fix + a temporary workaround.

In my case, the problem was buildkit/buildx and the GC was not configured.

I think the same approach at 99% can be used in any VM because by default the Garbage Collector is not enabled..

details and solution here: Article

Prevent Gitlab-runner from leaving dead containers/volumes

Summary

Proposal

Steps to reproduce

Actual behavior

Expected behavior

Environment description

Used GitLab Runner version

Designs

Child items ...

Activity

Prevent Gitlab-runner from leaving dead containers/volumes

Summary

Proposal

Steps to reproduce

Actual behavior

Expected behavior

Environment description

Used GitLab Runner version

Relates to

Activity