fix: prevent checkpoint memory growth in outbox

Duo Developerrequested to merge
workloads/addcf3e9399 into main

What does this merge request do and why?

Modifies duo_workflow_service/executor/outbox.py to exclude actions from response tracking dictionaries if they don't expect a response.

This fix prevents memory accumulation in long-running workflows by ensuring newCheckpoint messages are sent to clients but not tracked in response dictionaries (since they don't receive responses).

For all action responses, if it doesn't have a request ID it will behave as before:

  • it will still call legacy_set_action_response
  • it will still log Request ID not found. as an error

But if the request ID isn't found in _action_response, it now won't attempt to set an action response. It will log Received response for action that doesn't expect responses. Discarding. This should only happen if a client sends an action response with a request ID for an action that shouldn't be tracked (i.e., NewCheckpoint).

Relates to issue #1673 (closed)

How to set up and validate locally

Perhaps the easiest way would be to add a breakpoint at line 44 in duo_workflow_service/executor/outbox.py and confirm that is doesn't get hit when checkpoints are sent to the outbox. That happens a lot when responses are streamed while using Agentic Chat.

Alternatively, as described in #1578 (closed) and !3925 (comment 2917049478), you can run a fix_pipeline flow and check that tool calls are executed as expected (no empty results when there should be content).

  1. Edit gitlab/DUO_WORKFLOW_EXECUTOR_VERSION to be 0.0.51 (this version returns a response to NewCheckpoint actions).
  2. Configure GDK to use the branch from this MR for DWS: 
    gdk config set gitlab_ai_gateway.version workloads/addcf3e9399
gdk reconfigure
  3. Create an MR with a pipeline failure
  4. In the MR or pipeline, click the Fix pipeline with Duo button.
  5. Check the flow session (or Langsmith trace, if enabled) and confirm that tool calls that should have a response, do have one.

Merge request checklist

  • Tests added for new functionality. If not, please raise an issue to follow up.
  • Documentation added/updated, if needed.
  • If this change requires executor implementation: verified that issues/MRs exist for both Go executor and Node executor or confirmed that changes are backward-compatible and don't break existing executor functionality.
Edited by Mark Lapierre

Merge request reports