## Problem When someone mentions GitLab Duo in Slack, the answer can take minutes to an hour to start, or never arrive. While nothing happens, the user only sees 👀. In the worst case they get *"Something went wrong while running the task. Try again."* while the agent is still going to answer, and the answer that eventually comes is thrown away. Two things combine: 1. **The turn waits like background work.** Every Slack turn runs in `Ai::Messaging::ServerSideTurnWorker`. The worker opts into `defer_on_database_health_signal :gitlab_main, [:duo_workflows_workflows]`. When a database health indicator signals Stop, the job is pushed back: 5 s, then doubling up to 30 minutes (`incremental_database_health_defer_delay`, enabled on GitLab.com since 2026-08-11, https://gitlab.com/gitlab-org/gitlab/-/work_items/608150). Once a job has reached a long step it sleeps that long, even if the signal cleared seconds later. The mechanism exists so bulk writers back off; a Slack turn is a user waiting. 2. **Nobody owns "this session didn't start in time".** Three parts decide independently: - `CleanStuckWorkflowsService` (cron, every 30 min) fails any session in `created`/`running` for more than 30 min. It doesn't know a turn is still queued. So a turn can wait up to ~60 min before anyone reacts, and then the reaction is a generic failure. - The queued turn job runs whenever Sidekiq lets it, even if the session was failed in the meantime. DWS resumes the failed session and answers. - Delivery is first-come: the cron's failure claims delivery, so the late answer is discarded (`Duo Messaging: Terminal outcome already delivered`). Deferral is one way a turn starts late; a saturated shared Sidekiq lane is another (2026-10-01, https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23075, see https://gitlab.com/gitlab-org/gitlab/-/work_items/631876). The user experience problem is the same: no feedback while waiting, and a misleading outcome afterwards. ## Evidence 7 days to 2026-10-06, `slack_assistant/v1` only (105 sessions, 131 turns): - 28 of 131 turn jobs (21%) were deferred by `database_health_check`; 16 waited more than 1 min, 7 more than 5 min (worst: 99 min). Clusters on 2026-10-01 and 2026-10-05. - On 2026-10-05 every health check for the turn worker returned `WriteAheadLog` → Stop (*"WAL archive queue is too big"*). - `ProgressDeliveryWorker` and `SyncSessionArtifactWorker` on the same chain were deferred too, so progress messages stall with the turn. <details><summary>Example: session 9016825 (2026-10-05, UTC)</summary> | UTC | Event | |---|---| | 12:59:10 | Session created, turn job enqueued | | 12:59 → 14:08 | Turn deferred 11× (5 s doubling to 30 min) | | 13:30–13:41 | Cron `duo_workflow_stuck_cleanup_failed` fails the session → `CallbackWorker` posts "Something went wrong…" | | 14:38:17 | Turn finally runs; DWS resumes the failed session (`retry_request_duo_workflow`) and answers correctly | | 14:38:33 | `Duo Messaging: Terminal outcome already delivered` → answer dropped | Kibana: [turn deferrals, 7 days](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:'2026-09-29T00:00:00Z',to:'2026-10-06T08:00:00Z'))&_a=(dataSource:(dataViewId:'AWNABDRwNDuQHTm2tH6l',type:dataView),query:(language:lucene,query:'json.class:%22Ai::Messaging::ServerSideTurnWorker%22%20AND%20json.job_status:deferred'))) · [session 9016825](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:'2026-10-05T12:55:00Z',to:'2026-10-05T15:00:00Z'))&_a=(dataSource:(dataViewId:'AWNABDRwNDuQHTm2tH6l',type:dataView),query:(language:lucene,query:'json.args:%229016825%22'))) · [health status](https://log.gprd.gitlab.net/app/discover#/?_g=(time:(from:'2026-10-05T12:55:00Z',to:'2026-10-05T14:45:00Z'))&_a=(dataSource:(dataViewId:'AWNABDRwNDuQHTm2tH6l',type:dataView),query:(language:lucene,query:'json.subcomponent:database_health_status%20AND%20json.job_class_name:%22Ai::Messaging::ServerSideTurnWorker%22'))) </details> ## Context and constraints - Deferring the turn saves the database little: the worker itself barely writes; the session's writes (checkpoints, status, whatever the agent creates) go through the Rails API, which is never deferred. Web Agentic Chat writes the same tables without any deferral, and the CI path's `CallbackWorker` doesn't defer either. Still, the health signal exists for a reason (WAL archiving falling behind affects backups and recovery), so **a database maintainer should review whatever we change here.** - https://gitlab.com/gitlab-org/gitlab/-/merge_requests/256206 already lowered the base delay from 1 min to 5 s for the same reason; the doubling still applies. - `ServerSideTurnWorker` holds a Sidekiq thread for the whole turn (`urgency :low`, `concurrency_limit 50`). Changing that is a larger design question, tracked in https://gitlab.com/gitlab-org/gitlab/-/work_items/631876; this issue is about the start delay and the lifecycle, not about the runtime. - https://gitlab.com/gitlab-org/gitlab/-/work_items/632217: re-running a turn job re-sends the turn's message. Any fix here that re-runs or resumes turns must not make that worse. ## Implementation ideas (open, not decided) Both parts are needed. Removing the deferral fixes most of today's delays, but a turn can still start late for other reasons: a saturated shared lane (https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23075), a re-push on deploy, retries. Without the lifecycle part, any of those still ends in "Something went wrong" while the answer is on its way, followed by a dropped answer. Suggested minimum: A (or B) + C + a clear message; D can follow. **Start delay** - **A. Don't defer interactive turns.** Drop `defer_on_database_health_signal` from the turn worker (and the progress worker), or override `defer_on_database_health_signal?` to skip it for messaging turns. Smallest change; doesn't help when the lane itself is saturated. - **B. Cap the deferral.** Allow at most one short deferral, then run. Keeps some back-pressure, bounds the delay. **Lifecycle: who decides that a session failed to start, and what the user sees** - **C. The turn checks before it runs.** If the session is already terminal when the job starts, don't resume it; tell the user instead. Cheap, prevents dropped answers and double messages, but only reacts once the job runs. - **D. A start deadline for messaging sessions.** A session that hasn't started its turn within N minutes is failed with a specific reason, which flows through `WorkflowFailedEvent` to the adapter with a clear message ("I couldn't start on this; try again in a few minutes"). Variants, each with trade-offs: - a messaging-specific threshold in the existing stuck-session cleanup (no new worker; today it runs every 30 min with a 30-min threshold, so a stuck session can wait up to ~60 min), - a scheduled check per turn (precise, but one more job per turn), - a generic "deadline" option in the Sidekiq skip/defer middleware (reusable, larger change). Other approaches welcome. ## Done when - A Slack turn does not wait on the database health signal for more than a bounded, short time (agreed with database maintainers). - If a turn can't start within an agreed time, the user gets a clear message in the thread, and no later answer for that turn is silently dropped or posted twice. - The turn worker's deferrals and start latency are visible (log fields or metrics) so we can confirm the fix on GitLab.com.