Stop deferring Slack turns on the database health signal
What does this MR do and why?
A Slack turn can wait up to an hour before it starts, and its answer can then be dropped. Resolves #632565 (closed).
Two changes:
ServerSideTurnWorkerandProgressDeliveryWorkerno longer opt intodefer_on_database_health_signal. Withincremental_database_health_defer_delayenabled on GitLab.com, a turn deferred while the WAL archive queue is too big backs off 5 s → 10 s → … → 30 min per step. Over 7 days, 21% of Slack turns were deferred, the worst for 99 minutes, while the user only saw👀 .- A turn whose session is already terminal is skipped. The stuck-session cleanup fails a session whose turn is still queued and tells the user "Something went wrong". Running the turn anyway resumed the failed session, produced an answer, and the delivery claim discarded it (
Duo Messaging: Terminal outcome already delivered).
What this does not fix
A turn can still start late when the shared Sidekiq lane is saturated (https://gitlab.com/gitlab-org/gitlab/-/work_items/631876) or after a deploy re-push. In that case the 30-min cron still fails the session with the generic message, but now without a wasted DWS resume or a silently dropped answer. A messaging-specific start deadline with a clearer message (idea D in the issue) is a follow-up.
Hold on, why not just bump the urgency?
Actually we can't bump up the urgency:
- Too long. :high is a promise that the job finishes within 10 s, and GitLab.com alerts when it doesn't. A turn takes 25 s at the median and up to 196 s, so every turn would trip that alert.
- External dependency. The job holds a Workhorse → DWS stream, so it is
worker_has_external_dependencies!;every_sidekiq_worker_spec.rbfails any worker that is both that and high urgency. - Wouldn't help. Urgency only changes queue routing; it does not touch the deferral or the catchall saturation. The lever for that is a dedicated queue (option A in https://gitlab.com/gitlab-org/gitlab/-/work_items/631876).
Verification on GitLab.com
After deploy, this Kibana query should return nothing: json.class:"Ai::Messaging::ServerSideTurnWorker" AND json.job_status:deferred.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.