gitaly/config: Introduce runtime directory configuration
In Gitaly, we're creating different kinds of files at runtime which are
required to operate correctly. These files are by default created in the
operating system's temporary directory, which is typically /tmp
. While
it is clear that this directory can often be tmpfs and thus volatile,
this is perfectly fine: we regenerate the runtime data on every start
anyway.
Modern systems based on systemd use systemd-tmpfiles(8) though, which
also supports regular pruning of temporary files. So if the files we
create in /tmp
aren't accessed for a specific grace period then the
daemon will clean those up. This problem becomes a lot worse though if
/tmp
is mounted with the noatime
mount option: even if files are
constantly used, systemd will eventually remove them anyway. Of course,
this completely breaks all parts of Gitaly which rely on these files:
hooks, the Git execution environment, and internal sockets.
The root cause for this problem is that Gitaly doesn't have a go-to solution to host all such files, but instead it has ad-hoc solutions for every new kind of file we need to exist at runtime. If we had that, and if its location was configurable such that administrators can decide themselves where to put them so that they don't get pruned, then this problem wouldn't exist or at least be the responsibility of the admin.
This commit thus introduces a new runtime directory configuration into Gitaly that is supposed to unify all current locations where we create runtime files into a single well-defined location. This reduces the problem we need to solve into a single one instead of creating the problem anew for every new kind of runtime data.
By default, we're still kind of forced to create the runtime directory
in /tmp
: except for the storage locations, it is the only location
known to be writeable by us. While we could try and abuse storage
locations, e.g. by just using the first storage as the location for the
runtime directory, this would put additional restrictions on the storage
paths which don't currently exist because we need to ensure short path
names so that sockets continue to work alright. But on systems where it
is known that /tmp
will get regularly cleaned up, an administrator can
just point the new runtime_dir
config to an arbitrary existing path,
which will then cover all runtime files.
Changelog: added
Closes #4113 (closed)