Make Secrets Manager activation read-your-writes safe on replicas

What does this MR do and why?

Makes the Secrets Manager activation read-your-writes safe for requests that did not perform the write.

Cause. The provisioning worker flips the group or project manager row to active from Sidekiq. No web request is stuck to the primary for that write, so while a replica has not replayed it a request still reads provisioning, and every reader of secrets_manager.active? returns "Secrets manager is not active" for an active manager. On GitLab.com this showed as a red alert on the Secrets page right after a customer started the trial: the status poll read ACTIVE from a current replica, the secrets list query 200 ms later read a replica that was still about 10 s behind (Kibana evidence in the issue).

Fix.

• BaseSecretsManager#stick_activation!: called by both provision services right after activate!, records the primary's WAL position for the owner with the load balancer's sticking API (:group_secrets_manager / :project_secrets_manager, keyed by the owner id).
• BaseSecretsManager#refreshed_after_activation: a no-op without replicas and unless the row still reads provisioning (the only state a lagging replica can show for an activated manager); then it asks the load balancer for a caught-up replica (or the primary) and reloads the row from it. Returns nil when the row was already deleted there (activated and deprovisioned within the lag window), which callers treat as inactive and the status resolvers resolve through the pending deprovision task.
• Readers call it before checking active?: GroupSecrets/ProjectSecrets ListService and ReadMetadataService (through a secrets_manager_active? helper on the base services), both secrets count resolvers, both status resolvers (so the poll flips to ACTIVE as soon as the primary has it), BaseSecretsPermission validation, ApiAccess::IssueTokenService and its REST endpoint guard in API::SecretsManagement::AccessTokens, and UserPermissions::EffectiveCapabilitiesService.

Deprovisioning already handles the reverse direction with the phantom DEPROVISIONING record; this MR gives provisioning the equivalent.

Resolves https://gitlab.com/gitlab-org/gitlab/-/work_items/633014

Cost and caveats

• Only while the row reads provisioning (the provisioning window), per reader: one Redis GET for the recorded position; when a position exists, a WAL catch-up check against each configured replica (LoadBalancer#select_up_to_date_host) and a Redis unstick script once all have caught up; then one indexed single-row SELECT. Nothing changes for active managers.
• The stick record expires after 30 s (the sticking default). After that a provisioning manager behaves as before.
• Single-database installs (GDK default, the Linux package without load balancing) skip all of it: the method returns early when the load balancer is primary-only, and the writer records nothing there either.

How to set up and validate locally

Needs a replica and load balancing in GDK:

gdk config set postgresql.replica.enabled true
gdk config set load_balancing.enabled true
gdk reconfigure && gdk restart
echo "recovery_min_apply_delay = '10s'" >> postgresql-replica/data/postgresql.conf
gdk restart postgresql-replica

GDK lists the primary socket and the replica as the two read hosts, so reads alternate between a current and a 10 s stale host, the production shape. Restart rails-web and rails-background-jobs after switching branches: with load balancing on, a development code reload leaves the load-balancer constants unresolvable in both processes.

The production state is: row active on the primary with its position recorded, still provisioning on a lagging replica. To create it by hand on a provisioned group:

1. update group_secrets_managers set status = 0 where group_id = <id> on the primary; wait until the replica shows 0.
2. update group_secrets_managers set status = 1 where group_id = <id>; select pg_current_wal_insert_lsn() on the primary, then SET database-load-balancing/write-location/main/group_secrets_manager/<id> <lsn> EX 30 in the load-balancing Redis (what stick_activation! records).
3. Within 10 s, query { groupSecrets(groupPath: "<path>", first: 20) { edges { node { name } } } } and the status a few times with a PAT.

On master, reads that land on the lagging host return the GraphQL error "Secrets manager is not active" and the status alternates. On this branch every read succeeds and the status reads ACTIVE.

The self-managed E2E harness runs exactly this as scenario saas.replica.stale_active (fails on master 7a8e5bd5, passes on this branch); it also asserts that the real worker recorded the position after a natural provisioning.

Specs

• secrets_manager_examples.rb (shared by both model specs): #stick_activation! records under the owner id; #refreshed_after_activation is a no-op when active, re-reads when the primary has the activation, keeps provisioning when nothing has changed.
• Both ProvisionService specs: the activation is recorded for readers on lagging replicas. Fails without the stick_activation! call.
• GroupSecrets::ListService spec: a manager instance that still reads provisioning while the primary is active lists the secrets. Fails without the helper with "Secrets manager is not active".
• API::SecretsManagement::AccessTokens request spec: the same stale instance still mints a token (404 without the endpoint change).
• Shared model examples also cover the no-replica short-circuit and the deleted-row nil.

Screenshots or screen recordings

Reproduced on a GDK with a 10 s delayed replica; the page is loaded inside the window where the primary has the manager active and the replica still reads provisioning (harness scenario saas.replica.stale_active_ui, details in the MR note).

Before (master 2744d309)	After (this branch)

Videos: before , after .

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code