Don't generate new secrets by default
In !34249 (merged) / gitlab-com/gl-infra/production#2284 (closed), we added a new secret key in the Rails codebase. We did not add it to Omnibus or our Helm charts.
This will fail in interesting ways:
- Kubernetes is the best. Because
secrets.yml
is not writable, it will just fail to start. (This is what happened in gitlab-com/gl-infra/production#2284 (closed).)- The comment at the top of https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/01_secret_token.rb also doesn't mention Kubernetes.
- Omnibus and source installs will generate and write a new secret on application start. This is bad because:
- In multi-node setups, each node will generate its own secret. That means that we'll be encrypting with different keys depending on which node performs the write.
- We don't tell Omnibus users to back up
secrets.yml
, they should usegitlab-secrets.json
(which writessecrets.yml
): https://docs.gitlab.com/ee/raketasks/backup_restore.html#storing-configuration-files
In this Slack thread, @skarbek, @ayufan, @stanhu, and I discussed that we should probably:
- Never generate secrets automatically. This is only useful for source installs, and for those we could provide either instructions or a dedicated Rake task.
- Make an automatic check for a new secret being added to the Rails app but not to Omnibus and the Helm charts.
- Make
secrets.yml
(or even the whole codebase) not writable by thegit
user (or other configured user) in Omnibus installs, so that the Rails process cannot rewrite its own configuration.
Point 3 is more speculative on my part as it might break some valid workflows, and there might be other cases where we intend to rewrite configuration files from the app itself. But we can definitely pursue points 1 and 2. Point 2 would also be nice to generalise, so we can help groupdistribution out for general config items