Skip to content

gitaly/config: Introduce runtime directory configuration

Patrick Steinhardt requested to merge pks-gitaly-runtime-dir into master

In Gitaly, we're creating different kinds of files at runtime which are required to operate correctly. These files are by default created in the operating system's temporary directory, which is typically /tmp. While it is clear that this directory can often be tmpfs and thus volatile, this is perfectly fine: we regenerate the runtime data on every start anyway.

Modern systems based on systemd use systemd-tmpfiles(8) though, which also supports regular pruning of temporary files. So if the files we create in /tmp aren't accessed for a specific grace period then the daemon will clean those up. This problem becomes a lot worse though if /tmp is mounted with the noatime mount option: even if files are constantly used, systemd will eventually remove them anyway. Of course, this completely breaks all parts of Gitaly which rely on these files: hooks, the Git execution environment, and internal sockets.

The root cause for this problem is that Gitaly doesn't have a go-to solution to host all such files, but instead it has ad-hoc solutions for every new kind of file we need to exist at runtime. If we had that, and if its location was configurable such that administrators can decide themselves where to put them so that they don't get pruned, then this problem wouldn't exist or at least be the responsibility of the admin.

This commit thus introduces a new runtime directory configuration into Gitaly that is supposed to unify all current locations where we create runtime files into a single well-defined location. This reduces the problem we need to solve into a single one instead of creating the problem anew for every new kind of runtime data.

By default, we're still kind of forced to create the runtime directory in /tmp: except for the storage locations, it is the only location known to be writeable by us. While we could try and abuse storage locations, e.g. by just using the first storage as the location for the runtime directory, this would put additional restrictions on the storage paths which don't currently exist because we need to ensure short path names so that sockets continue to work alright. But on systems where it is known that /tmp will get regularly cleaned up, an administrator can just point the new runtime_dir config to an arbitrary existing path, which will then cover all runtime files.

Changelog: added

Closes #4113 (closed)

Merge request reports