Hand refresh events to Sidekiq while its refreshes are off

What does this MR do and why?

Once the refresh queue computes a group of event types, the old Sidekiq refreshes for those changes can be switched off. After that, the queue is the only thing that knows about those changes. If computing is then turned off again, as in a rollback, the queue must not drop those events, or the access changes behind them are lost.

This adds two ops flags, one per group, both on by default: authz_refresh_sidekiq_projects and authz_refresh_sidekiq_memberships. On, they keep the old Sidekiq refreshes running, so the queue drops the events it doesn't compute, as today. Off, events of that group that the queue doesn't compute go to the old Sidekiq workers instead of being dropped.

The producers themselves switch off in the next MR, and only while the group computes for every target as well, so turning a flag off early does nothing. That gives a safe rollback order: turn computing off first (the producers come back straight away and pending events are handed off), then enable the Sidekiq flag again once the queue has drained. Leave at least 5 minutes between changing computing and changing a Sidekiq flag, in either direction, because refresh jobs keep flag values for their whole run (up to 4 minutes) on top of the 1-minute process cache.

They're ops flags rather than derisk flags because they must default to on. Specs treat every flag as on until a hook stubs it, so a default-off "switch Sidekiq off" flag would turn off the old refreshes while before_all fixtures are built. That broke about one in eight finder and policy examples locally. Default-on also keeps the old refreshes on self-managed instances.

References

Screenshots or screen recordings

Not applicable, no UI change.

How to set up and validate locally

  1. In the Rails console, turn the projects Sidekiq flag off, queue a project event and drain it with computing off:
    Feature.disable(:authz_refresh_compute_projects)
    Feature.disable(:authz_refresh_sidekiq_projects)
    Authz::ProjectAuthorizations::RefreshEvent.enqueue(:project, [Project.first.id])
    Authz::ProjectAuthorizations::RefreshService.new(Gitlab::Metrics::RuntimeLimiter.new(5.minutes)).execute.payload
  2. The payload shows the event under handed_off_counts, a ProjectRecalculateWorker job is queued, and log/application_json.log has a Handed off project authorization refresh events line.
  3. With Feature.enable(:authz_refresh_sidekiq_projects), the same steps show it under dropped_counts.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Ian Anderson

Merge request reports

Loading
Loading