Add incremental backoff to database health job deferrals

Imanpal Singhrequested to merge
536042-incremental-db-health-defer-delay into master

What does this MR do and why?

Jobs deferred by defer_on_database_health_signal are re-enqueued at the worker's fixed delay_by for as long as the stop signal lasts. During long signals (autovacuum on a large table can run for hours), a job with the 5-second delay_by default (WorkerAttributes::DEFAULT_DEFER_DELAY; distinct from SkipJobs::DELAY, the feature-flag deferral branch's flat delay, 5 minutes by default) cycles through Redis thousands of times, re-evaluating the health indicators on every attempt, and all deferred jobs re-check in lockstep and execute together when the signal clears.

This MR makes the delay grow while the stop signal persists, using the deferred_count the job hash already carries:

  • First deferral: unchanged, the worker's delay_by (no behavior change for any of the ~185 opted-in workers unless a job is deferred repeatedly).
  • Each consecutive deferral: delay_by * 2^(deferred_count - 1), capped at an absolute 30 minutes, so a job always retries within 30 minutes of the database recovering.
  • Up to 10% jitter subtracted from backed-off delays, so jobs deferred together do not retry at the same moment (thundering herd on recovery was observed in !205521 (merged)) while the 30-minute cap remains a hard ceiling.
  • Only the :database_health_check branch backs off. The :feature_flag deferral branch (run_sidekiq_jobs_*, an incident-response lever) keeps its flat delay.

Behind the incremental_database_health_defer_delay feature flag (gitlab_com_derisk, disabled by default). No changelog entry per the changelog guidelines for default-off flags.

Notes for reviewers:

  • No job payload changes: deferred_count already round-trips through ApplicationWorker.deferred, so mixed-version nodes are safe during rolling upgrades (an old-version Sidekiq node simply applies the flat delay).
  • Backoff reduces re-enqueue churn and the recovery herd. It intentionally does not address the per-execution health-check cost (Prometheus/Mimir 429s); that is complementary work tracked in #413961.
  • Cross-reason deferred_count: a job previously deferred by the run_sidekiq_jobs_* flag enters health-check backoff with its existing count, so its first health-check delay can be deeper than delay_by. Accepted intentionally (the cap bounds it); pinned by a spec.

References

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Imanpal Singh

Merge request reports