Add kill switch to GitLab.com Duo Core todo notification worker

What does this MR do and why?

Makes GitlabSubscriptions::GitlabCom::DuoCoreTodoNotificationWorker stoppable while it runs. The worker creates a Duo Core access todo for every eligible member of a hierarchy, 1,000 users per batch. Today nothing can stop a running job. The generic run_sidekiq_jobs_* and drop_sidekiq_jobs_* flags are only checked when a job starts, and unlike the self-managed worker, it never re-checks the setting between batches.

  • Before each batch, the worker re-reads a new ops kill switch, process_duo_core_todo_notifications (enabled by default), and the namespace's duo_core_features_enabled setting. It stops if either is off.
  • The job is deferred when the database health signal for todos (gitlab_main_org) says to stop.
  • including_scheduled: true deduplicates the perform_in(7.days) jobs, so turning the setting off and on again repeatedly keeps one pending job per namespace.
Implementation notes
  • Flag values are memoized for the whole Sidekiq job, because Feature.flipper is cached in the request store with memoize: true. A plain per-batch Feature.enabled? would never see a change, so the worker calls Feature.reset before each check. Feature.reset also clears stubbed flags, so the kill switch specs set the flag for real. Because of the per-process L1 cache, a running job sees a flag change within about a minute.
  • The loaded namespace caches its namespace_settings, so the per-batch check reads a fresh, uncached record.
  • The health signal and deduplication only act when a job starts or is scheduled. The per-batch checks are what stop a running job.
  • A job stopped by the kill switch isn't rescheduled. That namespace's members get the todo after the next time the setting is turned off and on again.
  • With deduplication, re-enabling the setting within seven days reuses the pending job. The todo can then arrive less than seven days after the latest enable.
  • Not in this MR: a per-hierarchy member cap or per-batch jobs. Neither reduces the cost of the member query, which find_in_batches re-runs for every batch, and a cap would leave members past it without the todo. A follow-up will read the eligible user IDs once, behind a gitlab_com_derisk flag, with database review. This MR only adds a primary-key lookup per batch.

References

Screenshots or screen recordings

Not applicable: no UI change.

How to set up and validate locally

  1. In a Rails console, enable Duo Core on a root group that has members:

    group = Group.find_by_full_path('<root-group>')
    group.namespace_settings.update!(duo_core_features_enabled: true)
    worker = GitlabSubscriptions::GitlabCom::DuoCoreTodoNotificationWorker.new
  2. Disable the kill switch and run the worker. No todos are created:

    Feature.disable(:process_duo_core_todo_notifications)
    worker.perform(group.id)
    Todo.where(action: Todo::DUO_CORE_ACCESS_GRANTED).count
  3. Enable the kill switch and run the worker again. Eligible members get a Duo Core access todo:

    Feature.enable(:process_duo_core_todo_notifications)
    worker.perform(group.id)
    Todo.where(action: Todo::DUO_CORE_ACCESS_GRANTED).count

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Roy Liu

Merge request reports

Loading
Loading