Fix workflow lock and connection lifecycle bugs in duoworkflow

What does this MR do and why?

Fixes several lifecycle bugs in workhorse/internal/ai_assist/duoworkflow found during an audit of the package. These are small fixes that come before the larger refactor and feature work. The one change clients can see is the close code on initialization errors, described below.

  • Workflow lock stuck for 2 hours on the server-side execution path. The lock was released using the request context. When the caller disconnects, that context is already cancelled, so the Redis unlock failed. Release now uses a context detached from cancellation, with a 5s timeout.
  • Lock released before the DWS stream was closed. Release moved from the Execute defer to the end of Close(), so a reconnect on another Workhorse can't run the workflow alongside the old stream. Shutdown still releases early on purpose, because the instance is draining.
  • Lock released twice during shutdown. Release now takes the mutex under a lock and clears it, so only one of Shutdown and Close releases. A lock acquired after an earlier release is still released.
  • Close() could block forever when DWS never ended the stream after a stop request. The wait now ends one stop timeout (10s) after the stop request, which stopWorkflow has usually spent waiting for the acknowledgment already. Closing the connection afterwards unblocks the pending Recv and releases the lock. The lock is released in a defer, so a panic while closing can't leave it held.
  • gRPC connection leak when the self-hosted cloud tracking client or stream failed to open: the main client and stream are now closed on those paths.
  • WebSocket initialization errors were written as an HTTP response to an already upgraded connection. That write failed, so the client saw an abnormal closure (1006). It now receives a 1011 close frame with the reason "Failed to initialize agent platform client":
    • The web chat treats both codes the same way: as a retryable error.
    • gitlab-lsp (IDE and CLI) retries both. It now shows "the server encountered an internal error" instead of "the connection was dropped".
    • Error reporting is unchanged. logConnectionError already reports the error to Sentry. fail.Request reported it a second time.
  • Close frames were sent with conn.WriteMessage. That is unsafe while the agent goroutine writes actions (gorilla/websocket panics on concurrent writes), and it bypassed the transport's closed state, so Close() sent a second frame. Close frames now go through wsManager.SendClose, which uses WriteControl.
  • A transport with a keepalive interval of 0 panicked in time.NewTicker. The keepalive goroutine is now skipped. The coming detached transport needs this.

References

Screenshots or screen recordings

Not applicable: no UI changes.

How to set up and validate locally

  1. Enable a flow that uses the Duo Workflow WebSocket endpoint with LockConcurrentFlow enabled (for example, Agentic Chat in the web UI).
  2. Start a chat, then close the browser tab mid-turn.
  3. In Redis, confirm the workhorse:duoworkflow:lock:<workflow_id> key is removed shortly after the tab is closed, and that reopening the chat resumes without a "Failed to acquire lock on workflow" (1013) close.
  4. For the server-side execution endpoint (POST .../execute), interrupt the request mid-stream and confirm the same lock key is removed instead of staying until its TTL.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Igor Drozdov

Merge request reports

Loading
Loading