Hand refresh events to Sidekiq while its refreshes are off
What does this MR do and why?
Once the refresh queue computes a group of event types, the old Sidekiq refreshes for those changes can be switched off. After that, the queue is the only thing that knows about those changes. If computing is then turned off again, as in a rollback, the queue must not drop those events, or the access changes behind them are lost.
This adds two ops flags, one per group, both on by default: authz_refresh_sidekiq_projects and authz_refresh_sidekiq_memberships. On, they keep the old Sidekiq refreshes running, so the queue drops the events it doesn't compute, as today. Off, events of that group that the queue doesn't compute go to the old Sidekiq workers instead of being dropped.
The producers themselves switch off in the next MR, and only while the group computes for every target as well, so turning a flag off early does nothing. That gives a safe rollback order: turn computing off first (the producers come back straight away and pending events are handed off), then enable the Sidekiq flag again once the queue has drained. Leave at least 5 minutes between changing computing and changing a Sidekiq flag, in either direction, because refresh jobs keep flag values for their whole run (up to 4 minutes) on top of the 1-minute process cache.
They're ops flags rather than derisk flags because they must default to on. Specs treat every flag as on until a hook stubs it, so a default-off "switch Sidekiq off" flag would turn off the old refreshes while before_all fixtures are built. That broke about one in eight finder and policy examples locally. Default-on also keeps the old refreshes on self-managed instances.
References
- Closes https://gitlab.com/gitlab-org/gitlab/-/work_items/637588
- Parent: https://gitlab.com/gitlab-org/gitlab/-/work_items/637561
- Epic: https://gitlab.com/groups/gitlab-org/-/work_items/24326
- Proof of concept: !266827
Screenshots or screen recordings
Not applicable, no UI change.
How to set up and validate locally
- In the Rails console, turn the projects Sidekiq flag off, queue a project event and drain it with computing off:
Feature.disable(:authz_refresh_compute_projects) Feature.disable(:authz_refresh_sidekiq_projects) Authz::ProjectAuthorizations::RefreshEvent.enqueue(:project, [Project.first.id]) Authz::ProjectAuthorizations::RefreshService.new(Gitlab::Metrics::RuntimeLimiter.new(5.minutes)).execute.payload - The payload shows the event under
handed_off_counts, aProjectRecalculateWorkerjob is queued, andlog/application_json.loghas aHanded off project authorization refresh eventsline. - With
Feature.enable(:authz_refresh_sidekiq_projects), the same steps show it underdropped_counts.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.