Add duoworkflow action dispatcher, metrics and per-action logging

Igor Drozdovrequested to merge
observability/duoworkflow-actions-metrics into master

What does this MR do and why?

This MR adds the logging and metrics that detached execution and parallel actions need, and it is MR 1 of #634393. It builds on the runner restructure from !263066 (merged) and the action pool from !263826 (merged), both merged.

Action classification. A new actionDispatcher replaces the runner's action handling. It classifies each action by who executes it: Workhorse (http, mcp, self_hosted_tracking) or the client. Client-bound actions stay on the receiving goroutine, in receive order. When the run has an action pool, the actions Workhorse executes go to it, and their metrics and log line come from the worker, so the action duration excludes the time spent waiting for a free worker. The same errors end the run as before.

Logging. Each action Workhorse executes now produces one Handled action line, with:

  • type, outcome and duration
  • request_id and response size
  • method, path and status code for HTTP actions

This replaces three or four lines per action. Other logging changes:

  • Client-bound actions such as checkpoints are logged only when forwarding them fails.
  • The HTTP path is logged without its query string, which can carry user search text.
  • Response size is measured once, outside the stream's send lock.
  • Truncated responses and writes to a closed WebSocket are logged once, not on every write.
  • MCP init failures are logged per server at warning level, with the server name. A run without MCP servers no longer logs a failure.

Metrics (all gitlab_workhorse_duo_workflow_*):

Metric Labels
actions_total type, outcome (ok, error, fatal)
action_duration_seconds, actions_in_flight type
mcp_calls_total server, outcome
mcp_call_duration_seconds, mcp_init_failures_total server
session_duration_seconds transport, teardown_reason
session_setup_duration_seconds stage (grpc, mcp)
lock_acquire_total outcome
lock_release_failures_total
stop_requests_total outcome

Label sets are bounded:

  • server is gitlab, orbit or slack, or catalog for any AI Catalog server, because users choose those names.
  • method on the existing HTTP action metrics is now one of the standard methods or other.

The pool's own metrics, action_queue_wait_seconds and action_pool_saturated_total, came with !263826 (merged). The detached-run metrics arrive with #634394, because they need detached runs to exist first.

Behaviour changes:

  • A failed MCP call is logged at warning level instead of error, so it no longer reports an exception. These failures come from external servers.
  • Removed log lines: Executing HTTP request, Sending HTTP response event, Successfully sent HTTP response event, Sending action response, Calling an MCP tool, Sending MCP tool response and failed to initialize MCP server(s). Nothing in this repository refers to them; dashboards or saved searches outside it might.

MCP init failures are reported after the concurrent setup from !263067 (merged) finishes, one per failed server, so the setup goroutines don't log or count anything themselves.

Review follow-ups from !263066 (merged) (separate commit):

  • Stop timeout. A caller that joins a stop already in progress waits only for what is left of the timeout since the stop was sent, so the handshake takes at most one timeout. An acknowledgment that already arrived is checked before the timer.
  • Lock error names. errFailedToAcquireLockError is now errFailedToAcquireLock, and both lock errors name workflowLease, where they come from.
  • Unreachable Redis in tests. Tests that point Redis at a refused port share a helper with retries turned off, so they fail at once.

Review follow-ups from !263826 (merged):

  • Panics without a pool. A panic while Workhorse executes an action is answered with an error whether or not the run has an action pool. Before, it was only recovered on pool workers, so with MaxConcurrentActions at 1 or less it took down Workhorse.
  • Test helper. The pool tests read counters through the shared counterValue helper.

References

Screenshots or screen recordings

Not applicable: no UI changes.

How to set up and validate locally

  1. Open Agentic Chat in the web UI and ask something that triggers GitLab API and MCP tool calls.
  2. In the Workhorse logs, confirm each tool call produces one Handled action line with action_type, outcome, duration_s and workflow_id, and that checkpoints produce none.
  3. Fetch /metrics from Workhorse's Prometheus listener (-prometheusListenAddr) and filter for duo_workflow_. Confirm actions_total, mcp_calls_total{server="gitlab"} and session_setup_duration_seconds are present.
  4. Close the tab. Confirm session_duration_seconds and stop_requests_total{outcome="acked"} are recorded.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Igor Drozdov

Merge request reports