## Summary A workflow can fetch its Catalog item and the item's references from Rails at runtime. This issue adds the GET request for that: `GET /api/v4/ai/duo_workflows/workflows/:id/catalog_items`. Saving the flow is [#628153](https://gitlab.com/gitlab-org/gitlab/-/work_items/628153). The DWS side is [ai-assist#3306](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/3306) and [ai-assist#2870](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2870). ## Who calls it At the start of a session, DWS calls this endpoint before it builds the flow ([ai-assist#3306](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/3306)). The call goes through Workhorse's HTTP request action, with the run's token. It returns the flow definition and its agents. Some callers need the items later, after the session has started, and they use the GET request too: - A child session, to resolve its own `include` ([&55](https://gitlab.com/groups/gitlab-org/modelops/applied-ml/code-suggestions/-/work_items/55), [#630476](https://gitlab.com/gitlab-org/gitlab/-/work_items/630476)). - A `SubFlowComponent`, or any component, to claim items after the graph is built ([ai-assist#2749](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2749), [ai-assist#3307](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/3307)). - The session page, to list the imported agents ([#634337](https://gitlab.com/gitlab-org/gitlab/-/work_items/634337)). - Skills and flows as items, later, with the same shape. {width=900 height=260} [:information_source: Research, notes and source of this diagram](https://claude.ai/artifact/W2PnqJaNUAbtvDDuKB7wxe). Execution trace of the earlier preauthorization design: [artifact](https://claude.ai/artifact/A1isD9HHoe3m3CwRv85ATK). ## How it works <details> <summary>An example user workflow</summary> 1. A user triggers a flow with subagents in its definition. This sends `POST /api/v4/ai/duo_workflows/workflows` with `ai_catalog_item_consumer_id` and `start_workflow: true` 1. Rails authorizes the user against the project and the consumer, creates the workflow record with `ai_catalog_item_version_id`, and starts a CI job that runs `duo run --existing-session-id <id> --connection-type websocket` ([`StartWorkflowService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/ai/duo_workflows/start_workflow_service.rb)). With `ai_catalog_fetch_flow_config` on, the job gets `DUO_WORKFLOW_FETCH_FLOW_CONFIG=true` instead of the flow definition ([#634359](https://gitlab.com/gitlab-org/gitlab/-/work_items/634359)). 1. Duo CLI opens `GET /api/v4/ai/duo_workflows/ws` (websocket). Workhorse preauthorizes with Rails, which mints the execution session's OAuth token and returns the DWS connection config ([`ConnectionConfigService`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/ai/duo_workflows/connection_config_service.rb)). 1. Workhorse opens the gRPC stream to DWS and relays the `startRequest`. 1. :star2: THE ENDPOINT :star2: DWS calls [`GitlabHttpClient.aget`](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/main/duo_workflow_service/gitlab/executor_http_client.py) for `GET /api/v4/ai/duo_workflows/workflows/:id/catalog_items`, before it builds the graph. The call becomes a `RunHTTPRequest` action on the gRPC stream. 1. Workhorse executes the action against Rails with the execution's token ([`actions.go`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/workhorse/internal/ai_assist/duoworkflow/actions.go)). This functionality already exists. 1. Rails reads the saved item version from `workflow.ai_catalog_item_version` and resolves its dependency records with [`ReferenceResolver`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/ai/catalog/flows/reference_resolver.rb), for the triggering user and the executing project. _It does not read item ids from the request and does not re-parse the `include` section._ 1. Rails returns the item with its resolved `sub_items` 1. The response returns to DWS on the same stream as `ClientEvent.actionResponse.httpResponse`. 1. DWS matches each `include` entry to a node by `item_id` and `version`. It builds one `AgentComponent` per agent, adds it to the claiming component's `subagents`, and builds the graph ([`bind_catalog_items`](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/main/duo_workflow_service/agent_platform/v1/catalog/binding.py)) 1. Execution begins at the first node on the graph. The node sends its prompt to the LLM and the results go back to the user </details> ## Scope <details><summary>Deliverable checklist</summary> - [x] Add `GET /api/v4/ai/duo_workflows/workflows/:id/catalog_items` to [`workflows_internal.rb`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/lib/api/ai/duo_workflows/workflows_internal.rb). OAuth scope `ai_workflows`. - [x] Load the dependency records of the saved item version. Check each one with [`Ai::Catalog::Flows::ReferenceChecks`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/ai/catalog/flows/reference_checks.rb), for the triggering user and the executing project. `ReferenceResolver` uses the same class at save time, so save and execution share one set of rules. - [x] Return the root `item` with `sub_items`. Shape below. - [x] Agent `definition`: `prompt`, `toolset`, `mcp_tools` (empty until [#632032](https://gitlab.com/gitlab-org/gitlab/-/work_items/632032)). - [x] Add `MAX_REFERENCE_DEPTH` (1) to [`ReferenceLimits`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/ai/catalog/flows/reference_limits.rb). The service asks `ReferenceLimits.sub_items_allowed?` rather than comparing depth itself. - [x] Drop-and-continue: when a sub-item fails a check, it comes back with an `error` node instead of a `definition`. The call still returns 200. - [x] Whole-call status codes: 401 no token, 403 not through Workhorse, 404 workflow not owned by the user, 422 no saved Catalog item on the workflow. 200 with empty `sub_items` when the flag or the Feature preview setting is off. - [x] Error nodes keep `item_id`, `version`, `name` and `description`. The execution error can then say which agent. - [x] Specs for each status code and error code. Query-count spec for the preload. </details> MRs: [!261254](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/261254) (endpoint), [!262286](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/262286) (skip reason), [!263003](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/263003) (events). End-to-end proof: [!262838](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/262838). ## API Request and Response <details> <summary>RESPONSE SHAPE</summary> ### Request ```json GET /api/v4/ai/duo_workflows/workflows/:id/catalog_items :id // the Ai::DuoWorkflows::Workflow id of this session. Rails reads ai_catalog_item_version from that record Authorization: Bearer // the execution's OAuth token, scope: ai_workflows, set by Workhorse when it executes the RunHTTPRequest // No body. Rails resolves from the dependency records of the saved item version, not from the request. ``` ### Response ```json 200 { "schema_version": 1, // integer - response shape version "item": { // object - the item this workflow runs "item_type": "flow", // enum string - Ai::Catalog::Item.item_type - only flow for now "item_id": "12", // string, digits - Ai::Catalog::Item id "version": "1.3.0", // string, semver - exact Ai::Catalog::ItemVersion.version "name": "...", // string "description": "...", // string, nullable "definition_kind": "inline", // enum string - only inline for now "definition": {...}, // object - flow config as JSON (flow_v2.json) "sub_items": [ // array of type node - one per dependency record, and empty when none or when a FF/beta setting is off { "item_type": "agent", "item_id": "34", "version": "2.0.1", // must equal the "include" pin "name": "...", // must not be the same as another flow component "description": "...", // shown to the supervisor model in delegate_task "definition_kind": "inline", "definition": { // object - agent+inline (agent_v1.json) "prompt": "...", // string - system prompt "toolset": ["..."], // array of string - Ai::Catalog::BuiltInTool.name "mcp_tools": [] // array of string - empty until #632032 is figured out } }, { "item_type": "agent", "item_id": "56", "version": "0.9.0", // nullable - nil when the dependency record has no pinned version "name": "...", // kept so the execution error can show the agent name to the user "description": "...", "error": { // object - replaces definition "code": "location", // enum string, below "message": "..." // string } } ] } } ``` ### Error codes/response shapes ```json // Whole request: 401 { "message": "401 Unauthorized" } // no or invalid token 403 { "message": "403 Forbidden" } // request did not come through Workhorse 404 { "message": "404 Not Found" } // workflow unknown or not owned by the user 422 { "message": "This session does not run a saved AI Catalog item" } // no ai_catalog_item_version on the workflow, e.g. an inline flow 200 { "schema_version": 1, "item": { …, "sub_items": [] } } // flag or Feature preview setting off: not an error // Per sub-item: the node has error instead of definition. The call stays 200, so one bad agent never blocks the session. unknown_item // item gone or soft-deleted, item is not an agent, or user cannot read it location // item is private to another project, restricted to another root group, or blocked by a namespace restriction visibility // item is less visible than the flow unknown_version // no released version of that number ``` </details> <details> <summary>PROPOSED FOR A FOLLOW UP</summary> These fields and values could be cool for later. ```json GET.../catalog_items?depth=1 // integer, optional - layers of sub_items to resolve. For now though, we always returns one layer only "item_type": "agent" | "flow" | "skill" // skill once skills can be referenced, but third_party_flow can't be returned here "definition_kind": "inline" | "registry_ref" // registry_ref for a foundational flow: { flow_config_id, flow_config_schema_version, flow_version } "definition": { ..., "origin": { path, git_ref, blob_sha, commit_sha } } // repo-backed items only "mcp_tools": ["..."] // provide when we figure out #632032 // Per sub-item error codes that would come with deeper walks: limit_exceeded // depth or byte budget reached at this node cycle // the ref is already in the node's ancestor chain ``` </details> ## Limits `ReferenceLimits` sets the limits on the reference graph. Save and the GET request both read them: ```json MAX_REFERENCES_PER_KIND { agent: 20 } // existing, enforced at save MAX_PAYLOAD_SIZE_PER_KIND { agent: 112 KiB } // existing, enforced at save MAX_REFERENCE_DEPTH 1 // new - ReferenceLimits.sub_items_allowed? reads it ``` <details> <summary>PROPOSED FOR A FOLLOW UP</summary> - Audit event on each refused sub-item, through `Gitlab::Audit::Auditor` with the project as scope and the item as target. New audit event type YAML. - Internal events: one per request with sub-item count and entry point, one per refused sub-item with the error code. New definitions under `ee/config/events/`. - Byte budget on the GET response. The response returns as `HttpResponse` on the gRPC stream, so 4 MiB is the hard limit. The CI job variable reason for 112 KiB no longer applies to this path. - Cycle rule and `limit_exceeded` / `cycle` error codes, once `MAX_REFERENCE_DEPTH` goes above 1. </details> Session chain caps such as `MAX_TRIGGER_DEPTH` and `MAX_CHILD_WORKFLOWS_PER_PARENT` are in `Ai::FlowTriggers::CallerContext` ([#630476](https://gitlab.com/gitlab-org/gitlab/-/work_items/630476)) and are not part of this issue. ## Documentation - [ ] Development docs, DWS source, and error codes ([#633101](https://gitlab.com/gitlab-org/gitlab/-/work_items/633101)). ## Out of scope - The DWS side ([ai-assist#3306](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/3306), [ai-assist#2870](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2870)). - MCP tools of referenced agents ([#632032](https://gitlab.com/gitlab-org/gitlab/-/work_items/632032)). - A workflow with no saved Catalog item version. It gets a 422. A saved Catalog flow is a different case. The Duo CLI sends it as an inline `flowConfig`, and DWS accepts items on it ([ai-assist#3305](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/3305)). With `ai_catalog_fetch_flow_config` on, DWS fetches it from this endpoint instead ([#634359](https://gitlab.com/gitlab-org/gitlab/-/work_items/634359)). - Session chaining and its caps ([#630476](https://gitlab.com/gitlab-org/gitlab/-/work_items/630476)).