Don't generate new secrets by default

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Label this issue
• Close this issue

In !34249 (merged) / gitlab-com/gl-infra/production#2284 (closed), we added a new secret key in the Rails codebase. We did not add it to Omnibus or our Helm charts.

This will fail in interesting ways:

1. Kubernetes is the best. Because secrets.yml is not writable, it will just fail to start. (This is what happened in gitlab-com/gl-infra/production#2284 (closed).)
   1. The comment at the top of https://gitlab.com/gitlab-org/gitlab/-/blob/master/config/initializers/01_secret_token.rb also doesn't mention Kubernetes.
2. Omnibus and source installs will generate and write a new secret on application start. This is bad because:
   1. In multi-node setups, each node will generate its own secret. That means that we'll be encrypting with different keys depending on which node performs the write.
   2. We don't tell Omnibus users to back up secrets.yml, they should use gitlab-secrets.json (which writes secrets.yml): https://docs.gitlab.com/ee/raketasks/backup_restore.html#storing-configuration-files

In this Slack thread, @skarbek, @ayufan, @stanhu, and I discussed that we should probably:

1. Never generate secrets automatically. This is only useful for source installs, and for those we could provide either instructions or a dedicated Rake task.
2. Make an automatic check for a new secret being added to the Rails app but not to Omnibus and the Helm charts.
3. Make secrets.yml (or even the whole codebase) not writable by the git user (or other configured user) in Omnibus installs, so that the Rails process cannot rewrite its own configuration.

Point 3 is more speculative on my part as it might break some valid workflows, and there might be other cases where we intend to rewrite configuration files from the app itself. But we can definitely pursue points 1 and 2. Point 2 would also be nice to generalise, so we can help groupdistribution out for general config items 🙂