Automatically disable/enable Gitaly pack-objects cache when cache disk is full
In #2208, it came up that the current implementation of the Gitaly pack-objects cache does not have a good way to limit the size of the cache. Because we store the cache on filesystems with ample amounts of free space this is never a problem. But if we wanted to move the cache to a dedicated, smaller filesystem, we need to be able to stay within a size limit.
As a solution, I propose we add two configuration options:
-
watchdog_script
: a shell script provided by the administrator. Will be executed by Gitaly with/bin/sh -c
. If the exit status is 0, enable the cache. If 1, disable the cache. Default value: empty string. -
watchdog_interval
: Time to wait in between runs of the watchdog script. Default 1 minute.
The actual watchdog script depends on the infrastructure. Do we want a percentage of free space? Percentage of used space? An absolute amount of free space? By using a script we achieve the necessary flexibility without complicating Gitaly. Example script:
df -m /var/gitaly-cache | awk '{ if ($4 < 10000) exit(1) }'
The cache already has time based expiry. Once the watchdog disables the cache, time based expiry will eventually reclaim space and then the watchdog should re-enable the cache.
Suggested implementation
We can make a wrapper (middleware) around the real cache that calls the real cache or the null cache based on a boolean flag.