Fail or warn Omnibus backup restores when secrets change
Summary
We have many customers and users who restore GitLab and forget to restore the secrets file first, and sometimes it takes weeks or months for the problem to be fixed. I would like to see it be much harder to make this mistake.
Proposal
We could generate a hash of the secrets file, or each of its secret values, store those in the backup itself, and then compare that vs the secrets file on the target server.
I'm thinking we should fail by default and add an option that allows you to ignore secrets failures.
References
If you search Zendesk for "get_token"
you should see tons of tickets where we've ran into problems decrypting runner tokens. If you look for that or lib/gitlab/crypto_helper.rb
in our issue trackers, then you'll see many examples there as well.
I'm not complaining. Just trying to be clear that this is a common problem.