Shorten the database-health defer delay for messaging turns
What this changes
Two workers in ee/app/workers/ai/messaging/:
ServerSideTurnWorkerProgressDeliveryWorker
Both declared defer_on_database_health_signal :gitlab_main, [:duo_workflows_workflows], 1.minute. The explicit 1.minute is removed so both fall back to DEFAULT_DEFER_DELAY, which is 5 seconds.
Why
duo_workflows_workflows is written on every checkpoint and every status transition, so a database-health stop signal on that table is routine rather than exceptional. When the signal fires, Gitlab::SidekiqMiddleware::SkipJobs reschedules the job with perform_in(delay_by).
Measured on gprd over 24 hours: 216 ProgressDeliveryWorker jobs were enqueued more than 5 seconds after they were due. Sampled documents showed deferred_by: database_health_check with deferred_count: 1, meaning they waited out the full minute. One example:
| field | value |
|---|---|
created_at |
13:28:17.999 |
scheduled_at |
13:28:19.999 |
enqueued_at |
13:29:01.360 |
The scheduled_at is exactly created_at plus 2 seconds, which is the worker's own debounce interval working as intended. The remaining ~41 seconds is the deferral.
Both workers sit on a path where a person is waiting:
ServerSideTurnWorkerproduces the answer for a Slack mention, so a deferral delays the answer itself. The session sits increatedwhile it waits. The equivalent worker for a CI-executed flow,Ai::Messaging::CallbackWorker, declares no database health check at all.ProgressDeliveryWorkerdebounces at 2 seconds, so a frame held for a minute is stale when it arrives, and the worker's own terminal-status and already-delivered guards will usually discard it on arrival. The long delay therefore costs the render without sparing the table much.
Why shorten rather than remove
The stop signal is still respected, and consecutive deferrals still back off by a factor of 2 up to DATABASE_HEALTH_MAX_DELAY (30 minutes). Only the base delay changes. Load shedding under a sustained stop signal still works; a single deferral simply stops being visible to the user.
Ai::DuoWorkflows::CancelAssociatedPipelinesWorker already guards the same table with the default delay, so this brings these two workers in line with it.
Risk
Low. The change is to a base delay only, with no logic changes. Under a sustained stop signal the exponential backoff reaches comparable delays within a few deferrals.
How to verify
This is an operational change with no user-facing behaviour to click through. After deploy, confirm in Kibana that json.deferred_by: database_health_check entries for these two classes show json.enqueue_latency_s in the seconds rather than tens of seconds, using the pubsub-sidekiq-inf-gprd* data view.