GitLab Secrets Manager migration tool for domain changes

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

Problem to solve

When the GitLab external URL changes, OpenBao's JWT authentication configuration becomes invalid because URLs are hardcoded in multiple places:

  • Root-level JWT auth mount: oidc_discovery_url, bound_issuer, and bound_audiences on the gitlab_rails_jwt auth config
  • Per-project/group JWT mounts: Each namespace has pipeline_jwt and user_jwt auth mounts with their own oidc_discovery_url and bound_issuer
  • CEL role programs: The expected_aud value is hardcoded as a string literal in each CEL program at provisioning time

This primarily impacts Geo failover to a secondary site with a secondary domain, but also applies to:

  • Any external_url change on a GitLab instance
  • Migration between domains
  • Environment promotions (e.g., staging to production URL)

Current state

As validated in #595236 (closed), the failover scenario requires manually re-provisioning JWT authentication for every project and group where GitLab Secrets Manager is enabled. For deployments with thousands of projects, this process is slow and error-prone.

Without tooling, administrators must:

  1. Generate a new root token using the recovery key
  2. Manually update the root gitlab_rails_jwt auth mount via REST API
  3. Iterate over all active ProjectSecretsManager records and call configure_jwt + update_jwt_cel_role for each

Proposal

Create a migration tool (Rake task and/or Sidekiq worker) that automates the re-provisioning of OpenBao JWT authentication when the GitLab domain changes.

Update Geo docs, GSM admin docs, and OpenBao chart docs to reference this tool as needed. See #595236 (comment 3218740200)

Additionally, document how to reconfigure JWT auth at the root level (for privileged JWT) and create tools or rake tasks to support this as needed.

Technical approach

Instructions for reconfiguring JWT authentication at the root level: #595236 (comment 3212434034)

Technical approach to migrate groups and projects

Based on the investigation in #595236 (closed), the implementation would:

  1. Add a reprovision_auth method to ProvisionService:

    # in ProvisionService - make public + add reprovision path
def reprovision_auth
  configure_jwt(secrets_manager.ci_auth_mount)   # always, not guarded
  configure_pipeline_auth_cel

  configure_jwt(secrets_manager.user_auth_mount)
  configure_user_auth_cel
end

  2. Create a Sidekiq worker for batch processing:

    # app/workers/secrets_management/reprovisioning_auth_worker.rb
module SecretsManagement
  class ReprovisioningAuthWorker
    include ApplicationWorker

    def perform(secrets_manager_id)
      sm = ProjectSecretsManager.find(secrets_manager_id)
      ProvisionService.new(sm, Users::Internal.admin_bot).reprovision_auth
    end
  end
end

  3. Create a Rake task to trigger the migration for all active secrets managers

What the migration updates

For each project/group with Secrets Manager enabled:

Component Field Updated to
pipeline_jwt mount config oidc_discovery_url New GitLab URL
pipeline_jwt mount config bound_issuer New GitLab URL
user_jwt mount config oidc_discovery_url New GitLab URL
user_jwt mount config bound_issuer New GitLab URL
CEL role (pipeline_auth_cel) bound_audiences New OpenBao URL
CEL role (pipeline_auth_cel) expected_aud in CEL program New OpenBao URL
CEL role (user_auth_cel) bound_audiences New OpenBao URL
CEL role (user_auth_cel) expected_aud in CEL program New OpenBao URL

Considerations

  • Idempotency: The worker should be idempotent and safe to retry
  • Progress tracking: For large deployments, provide visibility into migration progress
  • Partial failure handling: Allow resuming from where migration left off
  • Root auth mount: The root gitlab_rails_jwt mount must be updated separately (requires root token)

Documentation requirements

Edited by 🤖 GitLab Bot 🤖