Draft: Add session legs API for cross surface DAP sessions (hidden refs)

Mikołaj Wawrzyniakrequested to merge
duo/feature/cross-surface-dap-sessions-hidden-refs into master

What does this MR do and why?

Prototype — not intended to merge as is. It has no tests, and its last commit is local-only test scaffolding.

Prototype of the Rails side of the hidden refs storage option from the cross surface DAP sessions ADR, the counterpart to the bespoke package store already prototyped in !252979 (closed). The ADR defers the decision between the two options until a hidden-refs prototype of the same scope exists; together with the client half in gitlab-org/editor-extensions/gitlab-lsp!4032 (closed), this is that prototype.

The difference from the bespoke store is where the package lives. Here it is git data in the project's own repository: the client pushes a synthetic leg commit to refs/duo-session/<workflow_id> and the next surface fetches it, so no bytes pass through Rails and there is no uploader, no object storage location, no workhorse route and no expiry-on-read. What Rails keeps is a thin metadata layer, so clients do not have to discover session state from the repository, and so registration is a point at which the store's invariants can be enforced.

What Rails checks at registration

Registration happens after the client's push has landed, and is the only place Rails can say anything about a leg:

  • the session has a project — hidden refs presuppose a repository, so a group-scoped or containerless session cannot use this store at all;
  • refs/duo-session/<workflow_id> currently points at the submitted leg_sha, read through Gitaly. This is what makes the claim checkable rather than trusted;
  • the leg commit descends from the last registered leg (its parents include that leg's SHA), otherwise 409. This is the server-side half of the client's push --force-with-lease: a surface that overwrote the ref concurrently cannot then record itself as the newest leg;
  • a cap on legs per session, as backpressure against a client looping on registration;
  • re-registering the leg already on record is the same fact, not a conflict, so it succeeds without writing.

Notable design decisions

  • The chain between legs is not stored in Rails. The leg commit already carries the previous leg as one of its parents — that octopus parent is what makes the chain git data, and ancestry is what orders legs. An earlier revision of this branch had a parent_leg_sha column, mirroring the ADR's table shape; 54185cd39c9d removes it. It was a second tip that could disagree with the repository, and nothing in Rails read it back: the concurrency check reads the parents from git. The row now holds only what git does not have, or what Rails must not take from it. The ADR's duo_workflow_session_legs table wants the same edit.
  • thread_ts is required and never derived. The bespoke prototype left it optional and best-effort. Only the client's executor knows which checkpoint its leg ended at, and "latest" is exactly the value that races with another leg or with a checkpoint still being flushed — so Rails records it and refuses a leg without it.
  • GET …/legs/latest reports checkpoint drift. A leg records the boundary a client chose to package, while checkpoints advance whenever the session runs anywhere, so the two tips drift apart (the "packages and checkpoints can drift apart" risk in the ADR). The endpoint returns the session's newest thread_ts alongside the leg's, plus a checkpoint_drift flag, so a restoring client can detect it before resetting a worktree to a boundary the agent has already talked past. This is the one fact in the response that exists nowhere in git.
  • No ref column, no manifest parsing. The leg commit's message is the manifest and the ref names the leg, so the receiving surface discovers branches and HEAD from git itself. Rails stays free of semantic knowledge of what the agent did, and the package format evolves without a Rails release.
  • Composite identity is resolved to the session owner. Carried over from the bespoke prototype: a CI leg authenticates as a composite-identity service account acting for the person who started the session, so the session is looked up as that person while the read_duo_workflow ability check still runs as the token's user.
  • No eligibility gate, for the same reason as the bespoke prototype: Ai::DuoWorkflows::Workflow#environment records the originating surface rather than the flow, so a gate on it answers differently for the same flow depending on which code path created the session. See the "environment names two different things" section of the ADR.

Endpoints:

Method Path
POST /ai/duo_workflows/workflows/:workflow_id/legs
GET /ai/duo_workflows/workflows/:workflow_id/legs
GET /ai/duo_workflows/workflows/:workflow_id/legs/latest

The last commit must be dropped before this merges

96bf810cb891 ("Override GDK's DAP image for testing") is local-only scaffolding for exercising the CI leg, mirroring 84817e1e87ed on the bespoke branch. It points StartWorkflowService::IMAGE_PATH at a locally built image carrying the prototype CLI (duo-cli-hiddenrefs:local, built by the build_image.ts invocation in the client MR) and adds --package-session to the executor command. Both changes carry PROTOTYPE OVERRIDE markers with revert instructions. The runner needs a pull_policy including if-not-present, or it tries to pull the local tag from a registry.

Unlike the bespoke store, packaging in a CI leg is a push rather than an upload, so it needs a credential carrying push_code and an origin that is not the job-token URL (which cannot push code). Both already hold for a coding-environment run — GIT_PASSWORD, which the CLI's credential helper reads, is the workflow OAuth token, and oauth_remote_commands already rewrites origin away from the job-token URL — which is why the commit changes nothing else. That path is unverified: no CI leg has pushed a leg commit yet, on either side of the prototype, and it is the one part of the mechanism neither MR has exercised.

Deliberately incomplete

  • No specs at all. This is the main reason the MR is a draft. It also means the permissions-verify check fails: each new endpoint needs its own it_behaves_like 'authorizing granular token permissions'. The pre-push hook was skipped for that check knowingly.
  • No ref cleanup lifecycle. Repository::RESERVED_REFS_NAMES gains duo-session, but nothing deletes a session ref. The ADR's plan is deletion on workflow destroy and on archive (30 days) through Repository#async_delete_refs → the durable BatchedGitRefUpdates queue, plus a low-frequency sweep over refs/duo-session/* for refs Rails never learned about, following Ci::PersistentRef and MergeRequests::CleanupRefsService. This is the main operational cost of the option and it is not exercised here: verified locally that deleting a session cascades its leg rows away and leaves the ref behind.
  • thread_ts is not verified to be a checkpoint of the session. The ADR asks for that check; it is orthogonal to proving the store works.
  • No hideRefs. Whether the namespace can be hidden from advertisement and still fetched by SHA is a Gitaly question, and it only matters once the read-access product question is answered.
  • No changelog entry, since nothing here is meant to ship.
  • Registration is not self-healing. The client MR reports that a leg landing on the ref without being registered leaves every later registration returning 409, because the check compares direct parents rather than ancestry while pushes keep advancing the ref. Both halves of the fix are named there; the Rails half is to check ancestry, which still rejects the case the check exists for.

References

Database

New table duo_workflow_session_legs, sharded on namespace_id, 7 columns, small, no blob columns. Every read is a single-session lookup:

-- list, and latest (LIMIT 1)
SELECT * FROM duo_workflow_session_legs
WHERE workflow_id = $1
ORDER BY created_at DESC, id DESC;

Covered by index_duo_wf_session_legs_on_workflow_id_and_created_at. The unique index on (workflow_id, leg_sha) backs the idempotent retry. namespace_id serves no query and is denormalized only because organization-level tables must carry a non-nullable sharding key referencing namespaces directly. Query plans are not attached: this is a prototype and the table is empty everywhere but a local GDK.

How to set up and validate locally

Validated on a local GDK in a DAP-enabled project, driving the endpoints the way a client would, with real git data. For the same flow through the actual CLI, use the client MR's instructions instead.

  1. bundle exec rails db:migrate, then restart Rails so the new API is mounted.

  2. Create a session:

    curl --request POST --header "PRIVATE-TOKEN: $TOKEN" --header "Content-Type: application/json" \
  --data '{"project_id":"<project>","goal":"prototype","environment":"chat","workflow_definition":"developer/v1"}' \
  --url "$GDK/api/v4/ai/duo_workflows/workflows"

  3. Build and push a leg commit the way a client would — tree from a throwaway index seeded from the real one, parents = HEAD plus the branches the leg moved (plus the previous leg, from the second leg on), message = the manifest:

    cp .git/index /tmp/leg.index
GIT_INDEX_FILE=/tmp/leg.index git add -A
TREE=$(GIT_INDEX_FILE=/tmp/leg.index git write-tree)
LEG=$(git commit-tree "$TREE" -p "$(git rev-parse HEAD)" -m '{"schema":1,"head":"refs/heads/duo/feature"}')
git push origin "$LEG:refs/duo-session/<id>" --force-with-lease="refs/duo-session/<id>:"

  4. Register it, list the session's legs, and resolve the newest one:

    curl --request POST --header "PRIVATE-TOKEN: $TOKEN" --header "Content-Type: application/json" \
  --data "{\"leg_sha\":\"$LEG\",\"thread_ts\":\"1f0a0001-0000-7000-8000-000000000001\"}" \
  --url "$GDK/api/v4/ai/duo_workflows/workflows/<id>/legs"
curl --header "PRIVATE-TOKEN: $TOKEN" --url "$GDK/api/v4/ai/duo_workflows/workflows/<id>/legs"
curl --header "PRIVATE-TOKEN: $TOKEN" --url "$GDK/api/v4/ai/duo_workflows/workflows/<id>/legs/latest"

  5. Restore on a "different surface" — a fresh clone:

    git fetch origin "refs/duo-session/<id>:refs/duo-session-local/<id>/incoming/leg"
git log -1 --format=%B "$LEG"                      # the manifest
git update-ref refs/heads/duo/feature "$LEG^1"     # branches from it
git checkout "$LEG" && git reset --mixed "$LEG^1"   # dirty worktree, git stash style

What this established

A full CLI → CI → register round trip: the fresh clone reproduced the committed work, the unstaged edit and the untracked file, with Rails having stored nothing but a SHA and a thread_ts — every object arrived through ordinary fetch negotiation, with no bundle prerequisite chain to replay. That clone then produced a second leg (-p HEAD -p <leg1>), pushed it with --force-with-lease and registered it, and Rails accepted it because git's ancestry, not a stored pointer, showed it chaining onto leg 1.

Both halves of the concurrency control hold: a stale-lease push was rejected client side (! [rejected] … (stale info)), and force-pushing past it and registering anyway returned 409.

Also checked: a SHA the ref does not point at, and a SHA absent from the repository, both 422; a malformed SHA and a missing thread_ts, 400; unknown session 404; unauthenticated 401; a retried registration returning the same row rather than a duplicate; the leg cap; a group-scoped session rejected for having no repository; and with a later checkpoint header present, legs/latest returning checkpoint_drift: true. The two-leg round trip and the 409 were re-run after parent_leg_sha was removed.

Two of the ADR's open questions get answers:

  • refs/duo-session/* is pushable. A real push through GitLab succeeded — pre-receive did not refuse the synthetic leg commit. Gitlab::GitAccess#check('git-receive-pack', …) allows the ref for a Developer, and the negative control (the same push without push_code) raises ForbiddenError, which also confirms the ADR's "packaging requires push access" cost.
  • Multi-leg sessions need no chain management, which is the row where the bespoke store needs cumulative bundles or client-side replay. The client MR takes this further, restoring a two-leg session into a third, previously uninvolved clone.

Still unverified: the CI leg's push, as described above; whether DWS resumes cleanly when the second leg arrives from a different surface than the one that suspended the session; and whether JiraVerificationCheck (which runs on all changes, unlike the branch/tag-only push-rule checks) rejects a leg commit.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Mikołaj Wawrzyniak

Merge request reports