gitaly/config: Support generating configuration via external command
In the context of FIPS we need a mechanism that allows us to not store any credentials in plaintext format on-disk. Ideally, we would hook into a solution like AWS Secrets or Kubernetes Secrets that lets us retrieve the secrets at startup time so that they don't have to exist on disk in any form at all. But there exists a bunch of different solutions for secrets management, where many make sense in one context but don't in any other contexts.
So implementing support for e.g. AWS Secrets directly would put us into a position where we now need to argue which solutions we want to support and which we don't. This shows that directly supporting any of them directly is not the right way to go, but that we should instead support a general schema that allows administrators to easily plug in whatever they are using without too much of a hassle.
A simple and boring solution to this is to implement support for running
an external command on startup that generates the secrets for us. Like
this, an administrator can simply provide a script that runs e.g. aws get-secret-value and Gitaly would know to put the secrets into the
correct spot in our configuration. But that again has another issue:
there are several different fields that may contain secrets, and making
sure they all can be adjusted might be tedious.
To fix that, we simply generalize the idea: why only care about secrets, when we can instead provide a mechanism that allows the external command to generate the complete configuration which we then merge back into the initial configuration read from disk?
The implementation of this is straight-forward: we read the initial configuration from disk, and when a specific key is set we use its value as the executable that is to be spawned. We expect the executable to generate JSON, which we then deserialize and thus merge into the config structure we have already parsed. Like this, the command is free to only generate configuration at runtime that it wants to actually change, and all the other configuration would stay as-is.
A simple external executable generating secrets via the AWS command line client could thus look like this:
#!/usr/bin/env ruby
require 'json'
JSON.generate({"gitlab": {"http_settings": {"password": `aws get-secret-value --secret-id ...`}}})But theoretically-speaking, you can generate the complete configuration of Gitaly.