Cap checkpoint write batches for Duo Workflow sessions

What does this MR do and why?

Caps what one request to the internal POST /ai/duo_workflows/workflows/:id/checkpoint_writes_batch endpoint can write. Until now, the only bound was the generic JSON middleware limit, and the endpoint accepted writes to finished, stopped and failed sessions.

A batch now goes over the limits when any of these holds:

  • The session is finished, stopped or failed.
  • The batch has more than 10 writes.
  • Any write's data is over 1 KB.

Over-limit batches are always logged (Duo Workflow checkpoint write batch exceeds limits, with limit_type, enforced, the session ID and the batch size). They are rejected with 400 only when duo_workflow_enforce_checkpoint_write_limits is enabled for the session's top-level group.

Why these limits, and why log-only first
  • On a production clone, 99% of checkpoints have 1 write, the largest has 183, and the longest data is 260 characters. The gateway sends 1 to 2 writes per request.
  • The gateway ignores rejected writes silently, so enforcing a limit that real traffic hits would lose checkpoint data without any error. The logs, with the flag off, show whether real traffic hits any limit before enforcement. The terminal-session check matters most here: no other checkpoint path refuses writes to terminal sessions.
  • Rollback: disable the flag. With the flag off, behaviour matches today's apart from the log line.

References

Screenshots or screen recordings

Not applicable: no UI change.

Testing

Tested end to end at 2d0a9387fa3 against the GDK development database. All 17 requests behaved as expected:

  • Flag off: every batch is stored as before, and each over-limit batch logs one warning.
  • Flag on (enabled for the gitlab-org group only): over-limit batches return 400 and store nothing, and batches within the limits are stored.
Method and results

Method. Each request was a real POST /api/v4/ai/duo_workflows/workflows/:id/checkpoint_writes_batch sent through this branch's full Rails middleware and API stack. Each used a temporary api-scoped personal access token for root and a signed Workhorse internal API header. The sessions were throwaway ones in gitlab-org/duo-workspace, set to each state directly. Afterwards, the sessions, their writes and the token were deleted and the flag was removed.

Flag Request Response Rows stored Log (limit_type, enforced)
off 2 writes × 100 B, running session 200 2 none
off 10 writes × 1,024 B (at both limits), running 200 10 none
off 11 writes, running 200 11 too_many_writes, false
off 1 write × 1,025 B, running 200 1 data_too_large, false
off 1 write, finished / stopped / failed session 200 each 1 each terminal_session, false
off 5,000 writes (the issue's reproduction) 200 5,000 too_many_writes, false
on 2 writes × 100 B, running 200 2 none
on 10 writes × 1,024 B, running 200 10 none
on 11 writes, running 400 Too many checkpoint writes: 11 exceeds the limit of 10 0 too_many_writes, true
on 1 write × 1,025 B, running 400 Checkpoint write data exceeds the limit of 1024 bytes 0 data_too_large, true
on 1 write, finished / stopped / failed 400 Cannot write checkpoints to a <state> session 0 terminal_session, true
on 5,000 writes (the issue's reproduction) 400 0 too_many_writes, true
on 11 writes × 2,048 B, finished (three limits hit) 400, terminal-session message 0 terminal_session, true
  • Log fields. Each warning carries limit_type, enforced, duo_workflow_id, gl_user_id, workflow_status, checkpoint_writes_count and largest_data_bytesize, and the values matched each request. When a batch hits more than one limit, only the first is logged, but the count and size fields still show the rest.
  • Reproduction timing. Locally, the 5,000-row request took 945 ms with the flag off and 148 ms with it on. Most of the 148 ms is the API parsing and validating the 5,000 rows before the service rejects them, so the flag stops the writes but not that parsing cost.

Not covered:

  • Workhorse and nginx. Requests went to the Rails app in-process, with the Workhorse header signed directly.
  • Real Duo Workflow Service traffic.
  • A second top-level group with the flag off. Flag scoping comes from Feature.enabled? with the session's top-level group as the actor.
  • The dev database was behind this branch, so the harness skipped Rails' pending-migration check. None of the pending migrations touch code this endpoint uses.

How to set up and validate locally

  1. Start a Duo Agent Platform session in GDK and note its ID. In the Rails console, mark it finished:

    Ai::DuoWorkflows::Workflow.find(<id>).update!(status: 3)
  2. Send a checkpoint write batch for it through Workhorse. The write is stored, and log/application_json.log shows "limit_type":"terminal_session","enforced":false.

  3. Enable enforcement:

    Feature.enable(:duo_workflow_enforce_checkpoint_write_limits)
  4. Send the batch again. It returns 400 Bad request - Cannot write checkpoints to a finished session and stores nothing.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Charlie Linacre

Merge request reports

Loading
Loading