Secrets Manager: backfill api_jwt auth onto existing secrets managers

What does this MR do and why?

Non-CI/CD API access (#594090 (closed)) added a new api_jwt OpenBao mount and all_api CEL role to secrets manager provisioning, shipped in GitLab 19.2. Secrets managers enrolled before that do not have this mount, so direct (non-CI) API secret access does not work for them until backfilled.

This MR:

  • Extracts the existing provisioning logic into a shared ApiAuthConfigurator service, so the provision flow (new enrollments) and the new backfill (existing enrollments) never drift.
  • Adds a gitlab:secrets_management:backfill_api_auth Rake task that an administrator (self-managed) or a GitLab SaaS SRE (via a change request) can run to backfill existing group and project secrets managers. It reuses ApiAuthConfigurator, so a backfilled SM ends up identically configured to a newly-provisioned one.
  • Documents when and how to run it, in Secrets Manager maintenance.

Why a Rake task, not a Batched Background Migration

An earlier version of this MR used a bespoke queue table and worker, then a Batched Background Migration (BBM). The BBM ran into a real blocker: the migration-testing pipeline needs a live OpenBao cluster, and we do not yet have a dedicated OpenBao test cluster wired into that pipeline (see the discussion starting at note_3626004181). Building that tooling is its own project.

Since Secrets Manager is still in beta, and disabling/re-enabling Secrets Manager is already an existing, supported way to re-provision an SM (it just deletes existing secrets, which is the tradeoff this task avoids), we do not need the automatic, guaranteed-eventual-consistency rollout a BBM gives us. A Rake task that admins or SREs can run on demand covers the same customer need without depending on OpenBao migration-testing tooling that does not exist yet. If and when that tooling exists, a future OpenBao-data migration (see #605100) can use a BBM properly.

What the task does

  • Iterates active project and group secrets managers in batches of 500, reusing ApiAuthConfigurator for each.
  • Idempotent: safe to re-run, including for SMs that failed a previous attempt.
  • Optionally scoped to a single customer with root_namespace_id, for a targeted SaaS change request instead of walking the whole fleet.
  • One SM failing does not stop the batch: failures are logged to stderr with the SM id and error, and the run ends with a succeeded/failed summary. The task itself never raises.

References

  • Parent, added api_jwt to provisioning: #594090 (closed)
  • Backfill tracking issue: #602549
  • Future migration that would revisit BBM for OpenBao data: #605100

Databases

No schema changes and no background migration. Nothing to review on main, ci, or sec.

Testing

All specs run against real OpenBao. For both project and group secrets managers:

  • Recreates the api_jwt mount and all_api role on an SM whose mount was removed (simulating a pre-#594090 enrollment).
  • Idempotent: reprocessing an already-configured SM reports success again, with no failures.
  • One SM errors: it is logged and reported, the other SMs in the batch are still backfilled, and the task does not raise.
  • Multiple SMs error: all failing ids and the correct succeeded count are reported.
  • Scoped to a root_namespace_id: only secrets managers under that root namespace are backfilled.
  • A previously-failed SM is backfilled on a later run once the cause is fixed.
  • Skips provisioning/deprovisioning SMs.
  • Empty fleet: completes cleanly, reporting zero counts.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Erick Bajao

Merge request reports

Loading
Loading