## Summary A saved (and therefore released) custom flow that lists AI Catalog agents in its YAML's `include` entries will run them as subagents during execution. This epic covers the execution pathway. The save pathway was [gitlab-org&23526](https://gitlab.com/groups/gitlab-org/-/work_items/23526), now closed. Two items from it are children here. ### References - Parent epic: [gitlab-org&21832](https://gitlab.com/groups/gitlab-org/-/work_items/21832). - [Diagrams of the system topology](https://claude.ai/artifact/W2PnqJaNUAbtvDDuKB7wxe). - [A visualization of the whole execution trace](https://claude.ai/artifact/A1isD9HHoe3m3CwRv85ATK). - End-to-end integration branch: [!262838](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/262838) with ai-assist branch [`poc/ai-catalog-pull`](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/compare/main...poc/ai-catalog-pull). ## Internal cross-team comms channel :compass: :slack: We are using this channel for cross-team communication: https://gitlab.enterprise.slack.com/archives/C0BUB4J0DPD <details> <summary>Decisions from gitlab-org&23526 and the planning issue</summary> - Every reference pins an exact version. No support for the "latest". ([source](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3805911237)) - Two rule sets: user permission and visibility hierarchy, for all catalog items, using the triggering user. ([source](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3785717552)) - Checks run again during execution against the container the flow executes from. ([source](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319#note_3892081394)) - A failed agent check blocks the execution and the error says which agent. Blocked MCP servers are dropped. ([source](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319)) - Twenty agent references per flow. Depth is flow, agent, the agent's MCP servers. ([source](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/256851)) - MCP tools of referenced agents are not in the first release. Built-in tools only. ([source](https://gitlab.com/gitlab-org/gitlab/-/work_items/632032)) - Rails resolves the agents when Workhorse preauthorizes the session. Workhorse puts them in the start request. DWS makes no request for them at start. The GET request stays for child sessions and mid-run claims. ([source](https://gitlab.com/groups/gitlab-org/modelops/applied-ml/code-suggestions/-/work_items/75#note_4031629422), [proof](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/262838)) - Rails resolves agent ids from the saved flow's dependency rows, never from the request. A workflow with no saved Catalog item gets 422 from the GET request. ([source](https://gitlab.com/gitlab-org/gitlab/-/work_items/628315)) - Workspace agents and AI Catalog agents are two lists in the same message, `catalog_items.catalog_items_v1`. Workhorse decodes the message Rails returns once. We add the AI Catalog list and leave the workspace list to [!262949](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/262949). ([source](https://gitlab.com/gitlab-org/gitlab/-/work_items/634357)) - Released at Experiment tier behind the Feature preview setting, the same shape as workspace agents. No new setting. ([source](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/259664)) - Still open for AppSec: which identity the checks use under a service account, and whether enabling a flow authorizes its whole chain. ([source](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319#note_3902006973)) </details> ## How it works - Every entry point opens a session through Workhorse: the Duo CLI and IDE on `GET /api/v4/ai/duo_workflows/ws`, the CI job that the web UI starts on the same route, and headless callers on `POST /api/v4/ai/duo_workflows/workflows/:id/execute`. - Workhorse preauthorizes with Rails. Rails has the workflow row, reads `workflow.ai_catalog_item_version`, and resolves the `include` entries from the dependency rows with [`ReferenceResolver`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/ai/catalog/flows/reference_resolver.rb) for the current user. It returns the flow definition and the agents beside the MCP servers ([#634357](https://gitlab.com/gitlab-org/gitlab/-/work_items/634357)). - Workhorse copies them into the `StartWorkflowRequest`, in the same code that attaches MCP tools ([#634358](https://gitlab.com/gitlab-org/gitlab/-/work_items/634358)). No client sends a definition or an agent for a Catalog flow, and no environment variable holds them. - DWS reads the agents from the start request and binds one `AgentComponent` per agent under the component that claims it ([ai-assist#2870](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2870)). - `GET /api/v4/ai/duo_workflows/workflows/:id/catalog_items` ([#628315](https://gitlab.com/gitlab-org/gitlab/-/work_items/628315)) uses the same resolver. It serves callers that need the items after the session has started: child sessions, `SubFlowComponent`, the session page, and Workhorse as a fallback. ## First release A custom flow with Catalog subagents executes from the web UI, the Duo CLI, Agentic Chat, and headless callers, with built-in tools, at the Experiment tier, behind the Feature preview setting. The `ai_catalog_flow_agent_references` flag is on only for development. The child items are ordered most critical first. Stretch items are last and have the ~Stretch label. ## Related efforts | Effort | What it does | Relation | |---|---|---| | [&75](https://gitlab.com/groups/gitlab-org/modelops/applied-ml/code-suggestions/-/work_items/75), workspace agents GA (Alexander) | [!262949](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/262949) returns workspace agents for CI runs at preauthorization and attaches them in Workhorse. The Duo CLI checks local agents and lists them with `/agents`. DWS runs the deep check. | Same message, `catalog_items_v1`. [#634357](https://gitlab.com/gitlab-org/gitlab/-/work_items/634357) and [#634358](https://gitlab.com/gitlab-org/gitlab/-/work_items/634358) target that MR and extend it. We do not build the workspace-agent side. | | [&55](https://gitlab.com/groups/gitlab-org/modelops/applied-ml/code-suggestions/-/work_items/55) / [&80](https://gitlab.com/groups/gitlab-org/modelops/applied-ml/code-suggestions/-/work_items/80), parent-child sessions (Mikołaj, Experiment in 19.6) | Session lifecycle as a contract: MRTR elicitation for human input, MCP Tasks semantics for child sessions, task handles on `start_flow` | A child session resolves its own `include` with the GET request ([ai-assist#3307](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/3307)). | | [#630476](https://gitlab.com/gitlab-org/gitlab/-/work_items/630476), bounded flow chaining (Igor, MRs 0–5 merged) | `parent_workflow_id`, `MAX_TRIGGER_DEPTH = 3`, `MAX_CHILD_WORKFLOWS_PER_PARENT = 5`, root human as the identity limit (`SA-n ∩ root human`) | Counts workflow rows. [`ReferenceLimits`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/services/ai/catalog/flows/reference_limits.rb) counts references. Each graph has its own limit. | ## Not in this epic Stretch items stay under [gitlab-org&21832](https://gitlab.com/groups/gitlab-org/-/work_items/21832) until they are promoted: - MCP tools of referenced agents ([gitlab-org/gitlab#632032](https://gitlab.com/gitlab-org/gitlab/-/work_items/632032)) - Dependency information in the UI, "uses" and "used by" tables ([gitlab-org/gitlab#628318](https://gitlab.com/gitlab-org/gitlab/-/work_items/628318)) - The GET request from DWS for child sessions and mid-run claims ([ai-assist#3307](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/3307)) - Flows as subagents or components, in-band ([ai-assist#2749](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2749)) and flows triggering flows, out-of-band (above). Both wait for parent-child sessions.