Secrets Manager: Backfill api_jwt mount and role into existing SM namespaces

Problem

Non-CI/CD secret access (#594090 (closed)) added a new api_jwt OpenBao mount and all_api CEL role. The provision service creates these for new SM enrollments. Projects and groups enrolled before that shipped do not have the mount and role, so the access-token endpoint does not work for them until they are backfilled.

Status

Delivered in 19.3 by !245543 (merged) (merged 2026-08-11).

The version boundary matters, and it is narrower than this issue originally assumed:

  • New enrollments have been provisioned with the api_jwt mount and all_api role since 19.2 (f3c2135a70e8, contained in v19.2.0-ee). This is not gated by the secrets_manager_api_access feature flag, which only gates the endpoint, not provisioning.
  • So a secrets manager provisioned in 19.2 or later needs nothing. The backfill is a no-op for it.
  • Only secrets managers provisioned in 19.1 or earlier are missing the mount and need action.

What shipped

  • A Rake task, gitlab:secrets_management:backfill_api_auth, that adds the api_jwt mount and all_api role to existing active project and group secrets managers. It is idempotent and safe to re-run.
  • Optional root_namespace_id argument to scope a run to a single customer instead of the whole fleet, for a targeted SaaS change request. Invalid input aborts rather than silently truncating to a different namespace id.
  • Per-secrets-manager error isolation: one failure does not stop the batch, and the run ends with a succeeded/failed summary listing failed ids.
  • ApiAuthConfigurator extracted and shared by the provision flow and the backfill, so a backfilled SM ends up identically configured to a newly-provisioned one.
  • Docs: Enable secrets access from external requests.

Why a Rake task, and not a background migration

This issue originally proposed anchoring on the maintenance-task / cron pattern. That was built first, then replaced with a Batched Background Migration (BBM) after review, and finally replaced with the Rake task. The reason for the last pivot:

  • A BBM over OpenBao data cannot be tested today. The migration-testing pipeline needs a live OpenBao cluster separate from production but connected to a clone of the OpenBao database, and that tooling does not exist. See the discussion on !245543 (merged).
  • Secrets Manager is still in beta, and disabling then re-enabling Secrets Manager is already a supported way to re-provision. It deletes existing secrets, which is exactly the tradeoff the Rake task avoids for customers who cannot lose them.

So we do not need the automatic, guaranteed-convergence rollout a BBM provides. An on-demand task that self-managed admins run directly, and SaaS SREs run through a change request, covers the same customer need without blocking on tooling that does not exist yet.

Plan and follow-ups

  • SaaS: run the task per customer request through a change request. Group customers into batches rather than raising one CR per customer.
  • Self-managed: covered by the documented Rake task. No action needed unless a customer asks.
  • Tooling gap (not owned by this issue): we still need proper support for migrating OpenBao data orchestrated by Rails. The Rake task is a stopgap. A future behaviour-changing migration such as #605100 (short token TTL on existing user_jwt roles) would want a real completion guarantee, which needs that tooling first.
  • Watch: the UX change in #605033 may remove the ability to re-provision. If it lands, the Rake task becomes the only path for pre-19.2 enrollments.

Notes

  • Additive only. Existing pipeline_jwt and user_jwt mounts are untouched.

Related

  • #594090 (closed) (endpoint + new-enrollment provisioning, this issue's parent)
  • !245543 (merged) (implementation)
  • #605100 (future migration that would need real OpenBao migration tooling)
  • #605033 (UX change that may remove re-provisioning)
Edited by Erick Bajao