Draft: Add session packages API for cross surface DAP sessions

Mikołaj Wawrzyniakrequested to merge
mwaw/cross_surfaces_dap_sessions_prototype into master

What does this MR do and why?

Prototype — not intended to merge as is. It has no tests.

Prototype of the Rails side of the cross surface DAP sessions ADR: Rails stores and serves stable boundary packages, so a DAP session started on one surface (Duo CLI, IDE chat) can be resumed on another compatible one (CI runner) and back.

Packaging stays a client responsibility — the client builds a git bundle against its own baseline ref namespace — so the blob is opaque here and Rails needs no semantic knowledge of what the agent did. The record holds only what serving it requires: session, thread_ts, the blob pointer and its retention metadata, following the artifact-with-TTL pattern of Ci::JobArtifact#expire_at.

Notable design decisions:

  • No ref column. A bundle already describes the refs it carries (git bundle list-heads), and a single column could not represent the several divergent refs a boundary may span. The earlier ADR draft had one; it was dropped.
  • No eligibility gate. An earlier revision rejected anything but chat/chat_partial with a 400, mirroring the ADR constraint that a migrated session must be resumable. It was removed in 72d9ea7ab009. The only field available to check, Ai::DuoWorkflows::Workflow#environment, records the originating surface rather than the flow's declared environment, and its writers are a client-supplied value, a value hardcoded in Rails' own foundational flow definitions, and a 'web' constant. The same flow is therefore judged eligible or not depending on which path created the session, and any client could opt any flow in by asserting chat. A check with that shape enforces nothing while looking like it does, so it is left out until a flow-level fact is persisted to gate on — see the "environment names two different things" section of the ADR.
  • Composite identity is resolved to the session owner. Packages are scoped to the session owner, but a CI leg authenticates as a composite-identity service account acting for the person who started the session, so looking the session up under the token's own user made every upload from a runner 404. cb50efd44198 resolves the owner through Gitlab::Auth::Identity#scoped_user for that case only; the read_duo_workflow ability check still runs as the token's user. This answers one of the ADR's open questions.
  • Expiry is honoured on read. Expired packages are invisible in listings and return 404 on download, so no client can rely on a package whose blob may already be gone.
  • New duo_session_packages storage location rather than reusing uploads, to keep bundles of repository content out of the public/ tree.

Endpoints:

Method Path
POST /ai/duo_workflows/workflows/:workflow_id/packages
GET /ai/duo_workflows/workflows/:workflow_id/packages
GET /ai/duo_workflows/workflows/:workflow_id/packages/:package_id|latest/download

A note on the commit history

545324611429 ("Normalize session environment before eligibility check") predates the decision above, so its subject now advertises a gate that no longer exists: 72d9ea7ab009 removes both the gate and the #normalized_environment helper that commit added for it. Left in history rather than rewritten, since the reasoning is part of what the prototype found out.

Deliberately incomplete

  • No specs at all. This is the main reason the MR is a draft. It also means the permissions-verify check fails: each new endpoint needs its own it_behaves_like 'authorizing granular token permissions'.
  • No expiry worker. expire_at is set and enforced on read, but blobs are never reaped. A cron worker deleting expired rows is the follow-up.
  • Uploads are not workhorse accelerated. This route is not in workhorse's route table, so Rack parses the multipart in Ruby (hence the new API::Validations::Types::MultipartFile). A production implementation would add the workhorse route plus direct upload to object storage and switch to WorkhorseFile.
  • No config/gitlab.yml.example entry for the new duo_session_packages storage location yet.

Client side

The client half of the prototype is gitlab-org/editor-extensions/gitlab-lsp!3994 (closed) (draft). It records the baseline refs, builds the package (manifest.json + session.bundle + uncommitted.patch) and drives these three endpoints, which is what establishes that an opaque blob plus a thread_ts is enough for Rails to move a session between surfaces.

To exercise the CI leg, the DAP runner needs an image carrying that CLI build. From a gitlab-lsp checkout:

E2E_IMAGE_TAG=duo-cli-crosssurface:local mise exec -- bun packages/cli/test/production-headless-e2e/docker/build_image.ts

Pointing StartWorkflowService::IMAGE_PATH at that tag (and adding --package-session to the executor command so a CI session packages itself) is local-only test scaffolding and is intentionally not part of this MR. The runner needs a pull_policy including if-not-present, otherwise it tries to pull the local tag from a registry.

References

Database

New table duo_workflow_session_packages, sharded on namespace_id, ~7 columns, small. Queries introduced are all single-session lookups on (workflow_id, created_at DESC):

-- list (API list endpoint)
SELECT * FROM duo_workflow_session_packages
WHERE workflow_id = $1 AND (expire_at IS NULL OR expire_at >= $2)
ORDER BY created_at DESC, id DESC;

-- latest (download endpoint)
SELECT * FROM duo_workflow_session_packages
WHERE workflow_id = $1 AND (expire_at IS NULL OR expire_at >= $2)
ORDER BY created_at DESC, id DESC LIMIT 1;

Both are covered by index_duo_wf_session_packages_on_workflow_id_and_created_at. Query plans are not attached yet, since this is a prototype and the table is empty everywhere but a local GDK — happy to add them when this turns into a real MR.

How to set up and validate locally

Validated on a local GDK in a DAP-enabled project.

  1. bundle exec rails db:migrate, then restart Rails. Note the new storage location means every Rails process needs restarting, Sidekiq included — a stale worker raises Gitlab::Configs::MissingConfig: option 'duo_session_packages' not defined in anything touching ObjectStorage::Config.

  2. Create a session:

    curl -s -X POST -H "PRIVATE-TOKEN: $TOKEN" -H "Content-Type: application/json" \
  -d '{"project_id":"<project>","goal":"prototype","environment":"chat","workflow_definition":"developer/v1"}' \
  "$GDK/api/v4/ai/duo_workflows/workflows"

  3. Build a bundle the way a client would, and upload it:

    git bundle create /tmp/session.bundle --all
curl -s -X POST -H "PRIVATE-TOKEN: $TOKEN" \
  -F "file=@/tmp/session.bundle" -F "thread_ts=thread-aaa" \
  "$GDK/api/v4/ai/duo_workflows/workflows/<id>/packages"

  4. List them, then download the newest and rehydrate it on a "different surface":

    curl -s -H "PRIVATE-TOKEN: $TOKEN" "$GDK/api/v4/ai/duo_workflows/workflows/<id>/packages"
curl -s -o /tmp/out.bundle -H "PRIVATE-TOKEN: $TOKEN" \
  "$GDK/api/v4/ai/duo_workflows/workflows/<id>/packages/latest/download"
git clone --bare /tmp/out.bundle /tmp/restored
git --git-dir=/tmp/restored for-each-ref

Observed: the download is byte-identical (md5) to the upload, git bundle verify reports a complete history, and the clone reconstructs every branch and file the session produced. Also checked: multiple packages per session, latest resolution, the thread_ts filter, unauthenticated → 401, unknown session → 404, empty file → 400, and a back-dated expire_at makes the package vanish from listings and 404 on download. Since the eligibility gate was removed, sessions in every environment (web, ambient, ide, chat) return 201, where web/ambient returned 400 before.

End to end, CI → Duo CLI

One full round trip was exercised with the client prototype and a locally built CLI image, which is also what proves the composite-identity upload path:

  1. Started a developer/v1 session with start_workflow: true (session 1169), which dispatched a DAP CI pipeline.
  2. The CI leg logged Snapshotted baseline (99 refs), then Bundled 1 commit, and uploaded package 15 (21504 B, hasBundle:true hasPatch:true) as the composite-identity service account.
  3. GET .../packages/latest/download returned a tar holding manifest.json, session.bundle and uncommitted.patch.
  4. Restoring into a fresh clone fetched into refs/duo-session/1169/incoming/*, leaving main untouched, then reproduced both the committed file and the untracked one from the CI leg. The commit the CI leg made exists nowhere in the canonical repository.

Still unverified: whether DWS resumes cleanly when the second leg arrives from a different surface than the one that suspended the session. That is a DWS-side question rather than a Rails one, but it is what decides whether the mechanism is usable end to end.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Mikołaj Wawrzyniak

Merge request reports