Shorten the database-health defer delay for messaging turns

What this changes

Two workers in ee/app/workers/ai/messaging/:

  • ServerSideTurnWorker
  • ProgressDeliveryWorker

Both declared defer_on_database_health_signal :gitlab_main, [:duo_workflows_workflows], 1.minute. The explicit 1.minute is removed so both fall back to DEFAULT_DEFER_DELAY, which is 5 seconds.

Why

duo_workflows_workflows is written on every checkpoint and every status transition, so a database-health stop signal on that table is routine rather than exceptional. When the signal fires, Gitlab::SidekiqMiddleware::SkipJobs reschedules the job with perform_in(delay_by).

Measured on gprd over 24 hours: 216 ProgressDeliveryWorker jobs were enqueued more than 5 seconds after they were due. Sampled documents showed deferred_by: database_health_check with deferred_count: 1, meaning they waited out the full minute. One example:

field value
created_at 13:28:17.999
scheduled_at 13:28:19.999
enqueued_at 13:29:01.360

The scheduled_at is exactly created_at plus 2 seconds, which is the worker's own debounce interval working as intended. The remaining ~41 seconds is the deferral.

Both workers sit on a path where a person is waiting:

  • ServerSideTurnWorker produces the answer for a Slack mention, so a deferral delays the answer itself. The session sits in created while it waits. The equivalent worker for a CI-executed flow, Ai::Messaging::CallbackWorker, declares no database health check at all.
  • ProgressDeliveryWorker debounces at 2 seconds, so a frame held for a minute is stale when it arrives, and the worker's own terminal-status and already-delivered guards will usually discard it on arrival. The long delay therefore costs the render without sparing the table much.

Why shorten rather than remove

The stop signal is still respected, and consecutive deferrals still back off by a factor of 2 up to DATABASE_HEALTH_MAX_DELAY (30 minutes). Only the base delay changes. Load shedding under a sustained stop signal still works; a single deferral simply stops being visible to the user.

Ai::DuoWorkflows::CancelAssociatedPipelinesWorker already guards the same table with the default delay, so this brings these two workers in line with it.

Risk

Low. The change is to a base delay only, with no logic changes. Under a sustained stop signal the exponential backoff reaches comparable delays within a few deferrals.

How to verify

This is an operational change with no user-facing behaviour to click through. After deploy, confirm in Kibana that json.deferred_by: database_health_check entries for these two classes show json.enqueue_latency_s in the seconds rather than tens of seconds, using the pubsub-sidekiq-inf-gprd* data view.

Merge request reports

Loading
Loading