Helm shared-secrets hook doesn't update existing secrets during chart upgrades, leaving encryption keys missing

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Summary

The gitlab-shared-secrets Helm hook fails to update existing secrets when upgrading from older chart versions. This leaves the secret stuck at an older chart version with missing encryption keys required for GitLab 17.8+, blocking upgrades to v18.x.

Steps to Reproduce

  1. Deploy GitLab with Helm chart 8.3.3 (GitLab 17.x or earlier)
  2. Upgrade to chart 8.11.8 (GitLab 17.11.7)
  3. Verify the gitlab-rails-secret label: 
    kubectl get secret gitlab-rails-secret -o jsonpath='{.metadata.labels.chart}'
# Returns: gitlab-8.3.3 (should be gitlab-8.11.8)
  4. Verify encryption keys are missing: 
    kubectl get secret gitlab-rails-secret -o jsonpath='{.data.secrets\.yml}' | base64 -d | grep active_record_encryption
# Returns: (empty - keys are missing)
  5. Attempt to upgrade to v18.2.8
  6. Observe upgrade failure with Errno::EXDEV when GitLab tries to initialize encryption keys

Current Behavior

  • The gitlab-rails-secret secret retains its old chart label (chart: gitlab-8.3.3)
  • The secret is missing three encryption keys:
    • active_record_encryption_primary_key
    • active_record_encryption_deterministic_key
    • active_record_encryption_key_derivation_salt
  • The gitlab-shared-secrets ConfigMap and Job have correct labels (chart: gitlab-8.11.8)
  • The ConfigMap contains the correct commands to create the missing keys, but the hook doesn't execute
  • The upgrade appears successful, but critical configuration is silently missing

Expected Behavior

  • The gitlab-shared-secrets hook should update existing secrets during upgrades
  • The secret should be updated with new encryption keys when upgrading to versions that require them
  • All Helm-managed resources should have consistent chart labels after an upgrade
  • Users should not encounter missing encryption keys when upgrading to v18.x

Environment

  • GitLab Version: 17.11.7 (chart 8.11.8)
  • Deployment Type: Kubernetes Helm (AWS EKS)
  • Shared-secrets: Enabled
  • Affected Secret: gitlab-rails-secret
  • Affected Resources:
    • gitlab-shared-secrets ConfigMap (correct label: chart: gitlab-8.11.8)
    • gitlab-shared-secrets Job (correct label: chart: gitlab-8.11.8)
    • gitlab-rails-secret Secret (incorrect label: chart: gitlab-8.3.3)

Root Cause Analysis

The gitlab-shared-secrets hook likely has logic that skips updating the secret if it already exists, without checking if the secret needs to be updated with new keys. This is a silent failure—the upgrade appears successful but critical configuration is missing.

Evidence:

  • Test environment deployment of v17.11.7 successfully created encryption keys in gitlab-rails-secret
  • Production gitlab-rails-secret has incorrect label showing it's stuck at chart version 8.3.3
  • gitlab-shared-secrets ConfigMap and Job have correct labels (8.11.8)
  • ConfigMap contains correct commands to create missing keys, but hook didn't execute properly

Impact

  • Users cannot upgrade to GitLab v18.x without manually adding encryption keys
  • The upgrade process appears successful but leaves the system in a broken state
  • Affects all Helm deployments that have been upgraded from older versions

Possible Fixes

  1. Update hook logic to check if existing secrets need to be updated with new keys
  2. Add version detection to the hook to determine if secret needs updating
  3. Improve error messaging to alert users when secrets are not properly updated
  4. Add validation to verify all required encryption keys exist before considering upgrade successful

Related Issues

  • Similar issue with migrations subchart and cross-device filesystem errors during upgrades
  • Affects users upgrading from GitLab 17.x to 18.x with Helm deployments
Edited by 🤖 GitLab Bot 🤖