Helm chart secrets rotating automatically on deploys

Summary

This bug issue was generated using GitLab Duo.

Encryption keys are rotating on each Helm chart deployment despite having shared-secrets enabled, causing ActiveRecord decryption errors and breaking functionality like OIDC/OAuth integrations.

Steps to reproduce

  1. Install GitLab using Helm chart (version 17.10.1 or later)
  2. Ensure shared-secrets is enabled in the Helm configuration
  3. Configure OIDC/OAuth functionality (e.g., AWS integration using https://docs.gitlab.com/ci/cloud_services/aws/)
  4. Deploy/redeploy the Helm chart
  5. Attempt to use OIDC/OAuth functionality

What is the current bug behavior?

  • Encryption keys (active_record_encryption_primary_key, active_record_encryption_deterministic_key, active_record_encryption_key_derivation_salt) and secrets.yml are rotating/changing on each Helm chart deployment
  • This causes ActiveRecord::Encryption::Errors::Decryption errors when trying to decrypt previously encrypted data
  • OIDC/OAuth functionality fails with decryption errors
  • The issue persists even after running the recommended mitigation step CloudConnector::Keys.delete_all

What is the expected correct behavior?

  • When shared-secrets is enabled in Helm charts, encryption keys should remain consistent across deployments
  • Encryption keys should not rotate automatically on each deployment
  • Previously encrypted data should remain accessible after deployments
  • OIDC/OAuth functionality should continue working after deployments

Relevant logs and/or screenshots

"exception.class": "ActiveRecord::Encryption::Errors::Decryption",
"exception.message": "ActiveRecord::Encryption::Errors::Decryption",
"exception.backtrace": [
  "activerecord (7.0.8.7) lib/active_record/encryption/encryptor.rb:58:in `rescue in decrypt'",
  "activerecord (7.0.8.7) lib/active_record/encryption/encryptor.rb:52:in `decrypt'",
  "activerecord (7.0.8.7) lib/active_record/encryption/encrypted_attribute_type.rb:73:in `block in decrypt'",
  ...
  "ee/app/models/cloud_connector/keys.rb:19:in `all_as_pem'",
  "ee/app/controllers/ee/jwks_controller.rb:11:in `load_keys'",
  ...
]

Environment

  • GitLab version: 17.10.1 (and later versions)
  • Installation method: Helm chart (Kubernetes)
  • Shared-secrets: Enabled
  • Affected functionality: OIDC/OAuth, Cloud Connector, any encrypted data

Possible fixes

The issue appears to be related to how the Helm chart handles encryption secrets generation/management even when shared-secrets is enabled. Investigation needed into:

  1. Why encryption keys are being regenerated on each deployment despite shared-secrets being enabled
  2. Whether there's a bug in the shared-secrets job logic for Helm installations
  3. Proper secret persistence mechanism in Helm chart deployments

Additional context

  • This is a regression that started appearing after GitLab 17.10 upgrades
  • Multiple customers have been affected (see support tickets #625785, #634915)
  • Current workaround involves deleting CloudConnector keys, but this breaks Duo functionality
  • The issue is specific to Helm chart installations - Omnibus installations work correctly with the documented unification steps
  • Related to the broader encryption secrets unification work in GitLab 17.x series
  • Even after applying the workaround (CloudConnector::Keys.delete_all), customers report that keys continue to rotate on subsequent deployments

Related Issues/MRs

Edited by 🤖 GitLab Bot 🤖