[Cells 1.0] (Size: S) Migrate `otp_key_base` to an encrypted `ApplicationSetting` attribute

GitLab stores the secret data required for two-factor authentication (2FA) in an encrypted database column. The encryption key for this data is known as otp_key_base, and is stored in config/secrets.yml.

Decide what to do with this secret with respect to Cells

Options:

• Per cell
• Ensure it is the same for all cells
• Something else

Proposal (2024-09-12)

Since we need a consistent otp_key_base for the whole Cells cluster. A solution is to migrate the current secret to an ApplicationSetting attribute. We'll look into ApplicationSetting sync between Cells later with &13165.

That would also be more consistent with other similar settings (e.g. ci_jwt_signing_key introduced by !43950 (diffs)). One benefit of this is that it removes the need for Secrets management in omnibus-gitlab and charts/gitlab as it's handled by the application directly.

The migration shouldn't be too hard (a similar to !43950 (diffs) but we'd populate the setting from the secret if it exists).

Estimate

• 1 day of backend
• 1 day of frontend