Cap checkpoint write batches for Duo Workflow sessions
What does this MR do and why?
Caps what one request to the internal POST /ai/duo_workflows/workflows/:id/checkpoint_writes_batch endpoint can write. Until now, the only bound was the generic JSON middleware limit, and the endpoint accepted writes to finished, stopped and failed sessions.
A batch now goes over the limits when any of these holds:
- The session is finished, stopped or failed.
- The batch has more than 10 writes.
- Any write's
datais over 1 KB.
Over-limit batches are always logged (Duo Workflow checkpoint write batch exceeds limits, with limit_type, enforced, the session ID and the batch size). They are rejected with 400 only when duo_workflow_enforce_checkpoint_write_limits is enabled for the session's top-level group.
Why these limits, and why log-only first
- On a production clone, 99% of checkpoints have 1 write, the largest has 183, and the longest
datais 260 characters. The gateway sends 1 to 2 writes per request. - The gateway ignores rejected writes silently, so enforcing a limit that real traffic hits would lose checkpoint data without any error. The logs, with the flag off, show whether real traffic hits any limit before enforcement. The terminal-session check matters most here: no other checkpoint path refuses writes to terminal sessions.
- Rollback: disable the flag. With the flag off, behaviour matches today's apart from the log line.
References
Screenshots or screen recordings
Not applicable: no UI change.
Testing
Tested end to end at 2d0a9387fa3 against the GDK development database. All 17 requests behaved as expected:
- Flag off: every batch is stored as before, and each over-limit batch logs one warning.
- Flag on (enabled for the
gitlab-orggroup only): over-limit batches return400and store nothing, and batches within the limits are stored.
Method and results
Method. Each request was a real POST /api/v4/ai/duo_workflows/workflows/:id/checkpoint_writes_batch sent through this branch's full Rails middleware and API stack. Each used a temporary api-scoped personal access token for root and a signed Workhorse internal API header. The sessions were throwaway ones in gitlab-org/duo-workspace, set to each state directly. Afterwards, the sessions, their writes and the token were deleted and the flag was removed.
| Flag | Request | Response | Rows stored | Log (limit_type, enforced) |
|---|---|---|---|---|
| off | 2 writes × 100 B, running session | 200 | 2 | none |
| off | 10 writes × 1,024 B (at both limits), running | 200 | 10 | none |
| off | 11 writes, running | 200 | 11 | too_many_writes, false |
| off | 1 write × 1,025 B, running | 200 | 1 | data_too_large, false |
| off | 1 write, finished / stopped / failed session | 200 each | 1 each | terminal_session, false |
| off | 5,000 writes (the issue's reproduction) | 200 | 5,000 | too_many_writes, false |
| on | 2 writes × 100 B, running | 200 | 2 | none |
| on | 10 writes × 1,024 B, running | 200 | 10 | none |
| on | 11 writes, running | 400 Too many checkpoint writes: 11 exceeds the limit of 10 |
0 | too_many_writes, true |
| on | 1 write × 1,025 B, running | 400 Checkpoint write data exceeds the limit of 1024 bytes |
0 | data_too_large, true |
| on | 1 write, finished / stopped / failed | 400 Cannot write checkpoints to a <state> session |
0 | terminal_session, true |
| on | 5,000 writes (the issue's reproduction) | 400 | 0 | too_many_writes, true |
| on | 11 writes × 2,048 B, finished (three limits hit) | 400, terminal-session message | 0 | terminal_session, true |
- Log fields. Each warning carries
limit_type,enforced,duo_workflow_id,gl_user_id,workflow_status,checkpoint_writes_countandlargest_data_bytesize, and the values matched each request. When a batch hits more than one limit, only the first is logged, but the count and size fields still show the rest. - Reproduction timing. Locally, the 5,000-row request took 945 ms with the flag off and 148 ms with it on. Most of the 148 ms is the API parsing and validating the 5,000 rows before the service rejects them, so the flag stops the writes but not that parsing cost.
Not covered:
- Workhorse and nginx. Requests went to the Rails app in-process, with the Workhorse header signed directly.
- Real Duo Workflow Service traffic.
- A second top-level group with the flag off. Flag scoping comes from
Feature.enabled?with the session's top-level group as the actor. - The dev database was behind this branch, so the harness skipped Rails' pending-migration check. None of the pending migrations touch code this endpoint uses.
How to set up and validate locally
-
Start a Duo Agent Platform session in GDK and note its ID. In the Rails console, mark it finished:
Ai::DuoWorkflows::Workflow.find(<id>).update!(status: 3) -
Send a checkpoint write batch for it through Workhorse. The write is stored, and
log/application_json.logshows"limit_type":"terminal_session","enforced":false. -
Enable enforcement:
Feature.enable(:duo_workflow_enforce_checkpoint_write_limits) -
Send the batch again. It returns
400 Bad request - Cannot write checkpoints to a finished sessionand stores nothing.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.