[FF] member_prune_deletion_per_batch_authorized_projects_refresh -- one project authorizations refresh per pruned member batch

Summary

Roll out the feature currently behind the member_prune_deletion_per_batch_authorized_projects_refresh feature flag.

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

When this flag is on for a root namespace, Members::PruneDeletionsWorker (which removes a user from every group and project under that namespace, up to 100 memberships per job) skips the per-membership project_authorizations refresh and instead runs one full recalculation for the user after each batch. Today, each destroyed membership enqueues its own refresh job. In one production sample, removing 1542 memberships produced 1474 refresh jobs where 18 would have done the same job.

The worst case is that the single batch refresh does not run. If that happened, a pruned user could keep project_authorizations rows for access that should have been revoked, until the next periodic recalculation catches up. There is no risk of data loss, and rolling back is just disabling the flag, which restores the previous per-membership refreshes. The code also has a few safeguards already in place: the batch refresh only runs after all destroys in the batch have committed, it also runs when the worker only cleans up the schedule (recovery after the worker is killed partway through), and the flag is read once per batch so behavior stays consistent for the whole batch.

Signals to watch:

  • Run this Kibana query: log.gprd.gitlab.net. The caller_id filter matches every job the prune worker enqueues; the class list narrows it to the authorization refresh workers.

    json.meta.caller_id: "Members::PruneDeletionsWorker" AND json.class: ("AuthorizedProjectUpdate::UserRefreshWithLowUrgencyWorker" OR "AuthorizedProjectsWorker" OR "AuthorizedProjectUpdate::ProjectRecalculatePerUserWorker")

  • With the flag off, only AuthorizedProjectsWorker, AuthorizedProjectUpdate::ProjectRecalculatePerUserWorker, and AuthorizedProjectUpdate::UserRefreshFromReplicaWorker (safety net) appear, many per pruning run. As the percentage rises, AuthorizedProjectUpdate::UserRefreshWithLowUrgencyWorker lines start appearing (about one per user per pruning run) and the other three shrink in proportion. At 100%, only UserRefreshWithLowUrgencyWorker should remain and the other three should drop to zero.

  • Warning sign: no authorization refresh workers starts/finishes (same query as above) while pruning. Disable the feature flag.

Rollout

Run all production /chatops in #production and cross-post the results to #g_authorization. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set member_prune_deletion_per_batch_authorized_projects_refresh 50 --actors --dev --pre --staging --staging-ref
/chatops gitlab run feature set member_prune_deletion_per_batch_authorized_projects_refresh true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set member_prune_deletion_per_batch_authorized_projects_refresh <percentage> --actors

The first production step is enabling the flag for one root namespace that runs regular dormant-member sweeps, since the actor for this flag is the root namespace itself. Compare that namespace's pruning runs against the same window's flag-off runs before moving on to a percentage rollout, then 100%.

/chatops gitlab run feature set --group=<root-namespace-path> member_prune_deletion_per_batch_authorized_projects_refresh true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check https://gitlab.com/gitlab-org/gitlab/-/merge_requests/254125 19.4
/chatops gitlab run feature delete member_prune_deletion_per_batch_authorized_projects_refresh --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set member_prune_deletion_per_batch_authorized_projects_refresh false                                         # production
/chatops gitlab run feature set member_prune_deletion_per_batch_authorized_projects_refresh false --dev --pre --staging --staging-ref     # non-production
/chatops gitlab run feature delete member_prune_deletion_per_batch_authorized_projects_refresh --dev --pre --staging --staging-ref --production  # remove entirely
Edited by Eugie Limpin