Add kill switch to GitLab.com Duo Core todo notification worker
What does this MR do and why?
Makes GitlabSubscriptions::GitlabCom::DuoCoreTodoNotificationWorker stoppable while it runs. The worker creates a Duo Core access todo for every eligible member of a hierarchy, 1,000 users per batch. Today nothing can stop a running job. The generic run_sidekiq_jobs_* and drop_sidekiq_jobs_* flags are only checked when a job starts, and unlike the self-managed worker, it never re-checks the setting between batches.
- Before each batch, the worker re-reads a new ops kill switch,
process_duo_core_todo_notifications(enabled by default), and the namespace'sduo_core_features_enabledsetting. It stops if either is off. - The job is deferred when the database health signal for
todos(gitlab_main_org) says to stop. including_scheduled: truededuplicates theperform_in(7.days)jobs, so turning the setting off and on again repeatedly keeps one pending job per namespace.
Implementation notes
- Flag values are memoized for the whole Sidekiq job, because
Feature.flipperis cached in the request store withmemoize: true. A plain per-batchFeature.enabled?would never see a change, so the worker callsFeature.resetbefore each check.Feature.resetalso clears stubbed flags, so the kill switch specs set the flag for real. Because of the per-process L1 cache, a running job sees a flag change within about a minute. - The loaded namespace caches its
namespace_settings, so the per-batch check reads a fresh, uncached record. - The health signal and deduplication only act when a job starts or is scheduled. The per-batch checks are what stop a running job.
- A job stopped by the kill switch isn't rescheduled. That namespace's members get the todo after the next time the setting is turned off and on again.
- With deduplication, re-enabling the setting within seven days reuses the pending job. The todo can then arrive less than seven days after the latest enable.
- Not in this MR: a per-hierarchy member cap or per-batch jobs. Neither reduces the cost of the member query, which
find_in_batchesre-runs for every batch, and a cap would leave members past it without the todo. A follow-up will read the eligible user IDs once, behind agitlab_com_deriskflag, with database review. This MR only adds a primary-key lookup per batch.
References
Screenshots or screen recordings
Not applicable: no UI change.
How to set up and validate locally
-
In a Rails console, enable Duo Core on a root group that has members:
group = Group.find_by_full_path('<root-group>') group.namespace_settings.update!(duo_core_features_enabled: true) worker = GitlabSubscriptions::GitlabCom::DuoCoreTodoNotificationWorker.new -
Disable the kill switch and run the worker. No todos are created:
Feature.disable(:process_duo_core_todo_notifications) worker.perform(group.id) Todo.where(action: Todo::DUO_CORE_ACCESS_GRANTED).count -
Enable the kill switch and run the worker again. Eligible members get a Duo Core access todo:
Feature.enable(:process_duo_core_todo_notifications) worker.perform(group.id) Todo.where(action: Todo::DUO_CORE_ACCESS_GRANTED).count
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.