Stop deferring Slack turns on the database health signal

What does this MR do and why?

A Slack turn can wait up to an hour before it starts, and its answer can then be dropped. Resolves #632565 (closed).

Two changes:

  1. ServerSideTurnWorker and ProgressDeliveryWorker no longer opt into defer_on_database_health_signal. With incremental_database_health_defer_delay enabled on GitLab.com, a turn deferred while the WAL archive queue is too big backs off 5 s → 10 s → … → 30 min per step. Over 7 days, 21% of Slack turns were deferred, the worst for 99 minutes, while the user only saw 👀.
  2. A turn whose session is already terminal is skipped. The stuck-session cleanup fails a session whose turn is still queued and tells the user "Something went wrong". Running the turn anyway resumed the failed session, produced an answer, and the delivery claim discarded it (Duo Messaging: Terminal outcome already delivered).

What this does not fix

A turn can still start late when the shared Sidekiq lane is saturated (https://gitlab.com/gitlab-org/gitlab/-/work_items/631876) or after a deploy re-push. In that case the 30-min cron still fails the session with the generic message, but now without a wasted DWS resume or a silently dropped answer. A messaging-specific start deadline with a clearer message (idea D in the issue) is a follow-up.

Hold on, why not just bump the urgency?

Actually we can't bump up the urgency:

  • Too long. :high is a promise that the job finishes within 10 s, and GitLab.com alerts when it doesn't. A turn takes 25 s at the median and up to 196 s, so every turn would trip that alert.
  • External dependency. The job holds a Workhorse → DWS stream, so it is worker_has_external_dependencies!; every_sidekiq_worker_spec.rb fails any worker that is both that and high urgency.
  • Wouldn't help. Urgency only changes queue routing; it does not touch the deferral or the catchall saturation. The lever for that is a dedicated queue (option A in https://gitlab.com/gitlab-org/gitlab/-/work_items/631876).

Verification on GitLab.com

After deploy, this Kibana query should return nothing: json.class:"Ai::Messaging::ServerSideTurnWorker" AND json.job_status:deferred.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Jannik Lehmann

Merge request reports

Loading
Loading