[FF] incremental_database_health_defer_delay -- Incremental backoff for database health job deferrals

Summary

Roll out the feature currently behind the incremental_database_health_defer_delay feature flag.

  • DRI: @imanpalsingh
  • Team Slack channel: #g_database_architecture

Note

Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. Feature flag controls · Feature flag lifecycle

What could go wrong?

The flag adds exponential backoff (doubling from the worker's delay_by, capped at 30 minutes, plus up to 10% jitter) to the re-enqueue delay of Sidekiq jobs deferred by defer_on_database_health_signal. Roughly 185 low-urgency workers are opted in.

  • Jobs can sleep up to 30 minutes past database recovery (by design, bounded by the cap). Watch completion latency of opted-in workers after long stop signals.
  • Feature.current_request is a fresh actor per Sidekiq execution, and every deferral is a new execution, so at intermediate percentages a single job's consecutive deferrals blend backoff and flat delay. Metrics are clearest at 0% and 100%; plan to move through intermediate percentages quickly.
  • Rollback is instant: disabling the flag restores the flat delay_by on the next deferral. No job payload changes are involved (deferred_count already exists on all deferred jobs).

Dashboards to watch on https://dashboards.gitlab.net:

  • sidekiq_jobs_skipped_total{action="deferred", reason="database_health_check"} rate per worker (feeds the SidekiqJobsDeferredByDBHealthCheck alert; the deferral rate should drop during stop signals)
  • Redis Sidekiq scheduled set size and memory saturation

Rollout

Run all production /chatops in #production and cross-post the results to #g_database_architecture. Background: incremental rollout process, feature actors.

Non-production

/chatops gitlab run feature set incremental_database_health_defer_delay 50 --actors --dev --pre --staging --staging-ref


/chatops gitlab run feature set incremental_database_health_defer_delay true --dev --pre --staging --staging-ref

Production — percentage rollout (wait ≥15 min between steps, watch dashboards):

/chatops gitlab run feature set incremental_database_health_defer_delay <percentage> --actors

Or target specific actors instead:

/chatops gitlab run feature set --project=gitlab-org/gitlab,gitlab-org/gitlab-foss incremental_database_health_defer_delay true


/chatops gitlab run feature set --group=gitlab-org,gitlab-com incremental_database_health_defer_delay true


/chatops gitlab run feature set --user=imanpalsingh incremental_database_health_defer_delay true

Before global rollout

Confirm the relevant gotchas before going to 100% — see enabling a feature for GitLab.com:

Cleanup

Remove the flag once deemed stable — see cleaning up. Track it here, or open a follow-up Feature Flag Cleanup issue. Remove the flag and its YAML definition from the codebase, then:

/chatops gitlab run release check <merge-request-url> 19.3


/chatops gitlab run feature delete incremental_database_health_defer_delay --dev --pre --staging --staging-ref --production

Rollback

/chatops gitlab run feature set incremental_database_health_defer_delay false                                         # production


/chatops gitlab run feature set incremental_database_health_defer_delay false --dev --pre --staging --staging-ref     # non-production


/chatops gitlab run feature delete incremental_database_health_defer_delay --dev --pre --staging --staging-ref --production  # remove entirely
Edited by Imanpal Singh