## Summary **Deliverable:** > A custom flow that specifies one or more catalog agents as sub-agents, using its ID and an exact version reference. It should be validated during save, resolved correctly at run time, and the dependency between the agent and the parent flow recorded in the DB. This is the child epic that contains the AI Catalog work to support custom sub-agents defined within custom flows. This is one part of [&21832 Custom Flows: Invoking sub agents and other catalog agents/flows](https://gitlab.com/groups/gitlab-org/-/work_items/21832). Ultimately we will support at least 4 pathways for allowing custom flows to execute nested items. This epic relates ONLY to the first of the list in the collapsable below: <details> <summary>The 4 Paths</summary> 1. :green_circle: **THIS EPIC** | Custom flows to reference custom agents as sub agents (Agent Foundations team has built support for this already) 2. Custom flows to reference custom agents as agent components (Needs new top-level component definition/extension in YAML schema) 3. Custom flows to invoke custom flows (`@dmishunov` has a [complete version of this](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2749), and it implements it as a top-level component) 4. Custom flows to invoke foundational flows (Not discussed nor built yet) </details> **Much of the outcome from _this_ epic will contribute to the other three deliverables. When this is refined enough, we will create issues for the others as well, as sub-epics and child issues.** ## Internal cross-team comms channel :compass: :slack: We are using this channel for cross-team communication: https://gitlab.enterprise.slack.com/archives/C0BUB4J0DPD --- ## Architectural Overview <details> <summary>The Diagram.</summary> ```mermaid sequenceDiagram autonumber actor User participant Rails participant CLI as CI job<br/>(Duo CLI) participant DWS participant LLM User->>Rails: Start flow Rails->>Rails: Load flow YAML<br/>with include entries Rails->>Rails: (new) Resolve each agent:<br/>item, version, permission Rails->>Rails: (new) Build agent list Rails->>CLI: Create CI job<br/>env: flow config, (new) agents, token Rails-->>User: Session view opens CLI->>DWS: Open gRPC stream CLI->>DWS: StartWorkflowRequest<br/>flowConfig + catalog_items DWS->>DWS: Normalize request<br/>(new) inline v1 accepts items DWS->>DWS: Parse FlowConfig<br/>and include list DWS->>DWS: (new) Match each claimed entry<br/>to an agent by id + version DWS->>DWS: Claimant becomes supervisor,<br/>one component per agent DWS->>DWS: Compile graph, start run DWS->>LLM: Coordinator prompt LLM-->>DWS: Delegate to sub-agent DWS->>LLM: Sub-agent prompt + toolset LLM-->>DWS: Tool call DWS->>CLI: Action CLI->>CLI: Run command or<br/>call GitLab API CLI-->>DWS: ActionResponse DWS->>DWS: Write checkpoint DWS-->>CLI: NewCheckpoint CLI->>Rails: Store checkpoint Rails-->>User: First output shown ``` Notes: - Step 4: the list has `item_id`, `version`, `name`, `description`, `system_prompt`, `toolset` per agent. - Step 5: the env vars are `DUO_WORKFLOW_FLOW_CONFIG`, `DUO_WORKFLOW_FLOW_CONFIG_SCHEMA_VERSION`, a new variable for the agent list, and the OAuth token. - Step 8: `catalog_items` is `{catalog_items_v1: {ai_catalog_agents: [...]}}`. - Step 16: the agent's `system_prompt` is passed to the prompt as a literal input value, not spliced into the template. - Step 18: an Action is `runCommand` or `runHTTPRequest`. - Step 23: the CLI stores the checkpoint through a GraphQL mutation. </details> --- ## Decisions These came out of the discussion threads on the [original planning issue #627641](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641). 1. **Every reference pins an exact version.** No "latest" option, even for the owner project. Dependencies are recorded as `ItemVersion` to `ItemVersion`. Loose pinning such as `v1.2.*` is not supported. Confirmed by `@knejad` and `@.luke`. ([thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3805911237), [confirmation](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3810620023)) **Schema, in review in [!256807](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/256807):** `dependency_id` stays set on every row, and a new `dependency_version_id` pins the exact version. A composite foreign key guarantees the pinned version belongs to the referenced item. This replaces the earlier design of a nullable `dependency_id` with an exactly-one check constraint, which allowed duplicate dependencies on the same item. 1. **Permission checks use two rule sets, and both must pass.** User permission (`read_ai_catalog_item`) and visibility hierarchy rules, following enablement: a private item can only be referenced by a flow in the same owner project, a restricted item only by a flow in the same root group. Checks run against the whole reference chain using the triggering user. A failed check blocks saving. ([thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3785717552)) **Proposed in [#628319](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319#note_3892081394), pending AppSec:** add a third rule, `blocked_by_namespace_restriction?`, and re-check at run time against the container the flow runs from, not only the flow's project. 1. **A limit on references is required.** Every reference is resolved and permission-checked, so the list cannot be unbounded. **Proposed in [!256851](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/256851) (draft):** one limit per kind of reference, starting at 20 agent references per flow, plus a 112 KiB cap on the serialized agent list, because the list is one CI job variable and Linux allows 128 KiB. Depth is fixed in iteration 1: flow, agent, the agent's MCP servers. This supersedes the earlier proposal of 10. 1. **Deleting a referenced item soft-deletes it.** Same behaviour as deleting an item that another project has enabled. `BaseDestroyService` already does this when dependency rows exist. ([thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3785549572), [gitlab#628170](https://gitlab.com/gitlab-org/gitlab/-/work_items/628170) applies the same pattern to MCP servers) 1. **Visibility changes on referenced items are handled in [gitlab#616785](https://gitlab.com/gitlab-org/gitlab/-/work_items/616785).** That issue is general enough to cover both MCP servers and agent references. 1. **MVC for dependencies is recording them.** "Uses" and "used by" UI, and newer-version notices, are iteration 2. 1. **MCP servers are not part of iteration 1.** A referenced agent runs with its built-in tools only. The MCP tools the agent selected, and the `read_ai_catalog_mcp_server` check per server, are in [#632032](https://gitlab.com/gitlab-org/gitlab/-/work_items/632032). MCP servers as direct `include` entries are not planned. --- ## Open Questions | No. | Question | Reference/Discussion Area | Answer w/ Reference | | ------ | ------ | ------ | ------ | | 1 | What does Rails send for an `ai-catalog` item: the full resolved agent, or only `item_id` and `version` for DWS to fetch at run time? | [thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3802794670), [ai-assist#2833](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2833) | **Answered.** Rails resolves each agent when the flow starts and sends the full agent data in the `catalog_items` field of the start request, as it currently does for workspace agents. [Comment](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2833#note_3823083771). | | 2 | Is the YAML syntax confirmed and ready? | [ai-assist#2768](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2768), [gitlab!6694](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/6694) | **Yes.** Merged in [`reference.py`](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/main/duo_workflow_service/agent_platform/v1/catalog/sources/reference.py): `source`, `item_type`, `item_id`, optional `version`. `version` is optional there, so Rails enforces the pin rule. | | 3 | How do we enforce namespace and identity boundaries on agents run from custom flows? | [gitlab#628319](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319), [gitlab#616785](https://gitlab.com/gitlab-org/gitlab/-/work_items/616785) | Partly settled by Decision 2. Proposal and gaps are in [this thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319#note_3888839841). AppSec consultation pending. | | 4 | What behaviour do we want when an item referenced in the flow YAML has been disabled? | [thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3785717552) | **Answered.** Nothing changes. Disabling an item deletes its enablement row, so a disabled item and a never-enabled item are the same state. References do not check enablement (item 2 below), so the flow keeps working. Question 7 revisits whether flow enablement should authorize the chain. | | 5 | What happens when a permission check fails at run time? Block the flow, or drop the dependency and log? | [thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3785717552) | **Answered.** A failed agent check blocks the run, the error is presented to the user and names the item. Blocked MCP servers are dropped, as `McpServers::ListService` already does. DWS already fails a run on an unmatched claim, so dropping an agent is not an option. Decision 3 on [#628319](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319). Two settings are not checks on the agent and do not block: the feature flag off for the flow project, or **Allow custom agents** off in the running top-level group. Then Rails removes the `include` section and the claims from the config it sends, and the flow runs without its sub-agents. An audit event records the reason ([!258327](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/258327)). | | 6 | Which identity do the checks use when a flow runs under a service account? | [thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319#note_3892081394) | Open, for AppSec. | | 7 | Does enabling a flow authorize every agent it references? | [comment](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319#note_3902006973) | Open, for AppSec. Proposal: flow enablement authorizes the whole chain, if the chain is visible at enable time and cannot change under an existing authorization. Owner projects that float to latest need a stated decision. | --- ## Issue breakdown ### 1. PRE-ISSUE: Agree on the Catalog reference contract for Flow Registry YAML <details> <summary>Resolved and delivered w/ one open question that doesn't impact _this_ work</summary> **Owner:** ~"category:flow components" with input from ~"category:ai catalog creation" **Purpose:** Agree on the final meaning of `include` before Rails or the UI implements anything. **Scope:** Confirm the top-level YAML shape. Define `source`, `item_type`, `item_id`, and `version`. Define how it works alongside the workspace-agent syntax. **Settled:** `latest` is not allowed. Every reference names an exact version (Decision 1). **Still open:** Should foundational items use the same syntax as custom Catalog items? **Related records:** [Flow Registry subagent schema](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2768), [Phase 1 MR gitlab!6694](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/6694), [composition epic](https://gitlab.com/groups/gitlab-org/-/work_items/21832). </details> ### 2. Validate Catalog references when a flow is saved → [#628153](https://gitlab.com/gitlab-org/gitlab/-/work_items/628153) **MRs:** [!257686](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/257686) (merged, MR 1: `include` in the flow schema behind the `ai_catalog_flow_agent_references` feature flag), [!258326](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/258326) (in review, MR 2 and 4: the resolver and the dependency rows). <details> <summary>Resolved. No enablement needed. Permission plus visibility rules, whole chain checked at save, reference limit applies. Work is in #628153.</summary> **Owner:** ~"category:ai catalog creation" and ~"category:ai catalog curation" **Purpose:** Prevent a user from saving a flow that names an unavailable, disabled or invalid agent. **Scope:** Extend `flow_v2.json` for `include`. Build the shared resolver. Apply the permission rule sets (Decision 2). Enforce the reference limit (Decision 3). Store the YAML as written. **Settled:** A referenced agent does not need to be enabled in the flow's project. The saving user needs `read_ai_catalog_item` on it and the visibility rules in Decision 2 apply. Validation checks the whole reference chain at save time, within the limit in Decision 3. ([thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3785717552)) </details> ### 3. Resolve references at run time → Rails half in [#628315](https://gitlab.com/gitlab-org/gitlab/-/work_items/628315), DWS half in [ai-assist#2870](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2870), save-time acceptance in [ai-assist#2833](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2833), Duo CLI half in [gitlab-lsp#3082](https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/work_items/3082) **Delivered:** [ai-assist#2833](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2833) via [ai-assist!6919](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/merge_requests/6919), merged 2026-09-18. <details> <summary>Resolved. Rails sends the full agent data when the flow starts. Rails half in #628315, DWS half in ai-assist#2870, save-time acceptance in ai-assist#2833.</summary> **Owner:** ~"category:ai catalog creation" **Purpose:** Turn Catalog references into the data DWS needs at execution time. **Scope:** Rails resolves each referenced agent when the flow starts, checks permissions for the triggering user, and sends the full agent data in the `catalog_items` field of the start request. It reuses the resolver from `#2`. DWS binds the pushed agents ([decision](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2833#note_3823083771)). </details> ### 4. Record Catalog dependencies → schema in [#628830](https://gitlab.com/gitlab-org/gitlab/-/work_items/628830), lifecycle in [#628317](https://gitlab.com/gitlab-org/gitlab/-/work_items/628317), rows written in [#628153](https://gitlab.com/gitlab-org/gitlab/-/work_items/628153), UI in [#628318](https://gitlab.com/gitlab-org/gitlab/-/work_items/628318) (iteration 2, moved to the parent epic) **MRs:** [!256807](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/256807) (merged, schema for #628830). <details> <summary>Resolved. Version-pinned rows with a required pin, schema in #628830. Deletion soft-deletes. Visibility changes in #616785. UI is iteration 2.</summary> **Owner:** ~"category:ai catalog creation" with ~"category:ai catalog curation" **Settled:** `ItemVersion` to `ItemVersion`, pin required (Decision 1). Deletion soft-deletes (Decision 4). Visibility changes handled in [gitlab#616785](https://gitlab.com/gitlab-org/gitlab/-/work_items/616785) (Decision 5). **Iteration 2:** "Uses" and "used by" UI, newer-version notices (Decision 6). ~UX ~frontend UX shape from `@michaelmoyers` ([tables](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3811453778)) is recorded in [#628318](https://gitlab.com/gitlab-org/gitlab/-/work_items/628318). The "newer version available" column and the replacement offered on a failed save both read one suggested version set by the item's manager. That field does not exist yet. </details> ### 5. Define permissions for nested Catalog and MCP dependencies → [#628319](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319) **MRs:** [!256851](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/256851) (approved, in the merge queue, reference limits). Check strategy proposal: [thread](https://gitlab.com/gitlab-org/gitlab/-/work_items/628319#note_3888839841). **Owner:** ~"category:ai catalog creation" with ~"category:ai catalog curation", consulting [AI Governance](https://gitlab.enterprise.slack.com/archives/C0A1YDNC67R) and [SSCS Auth](https://gitlab.enterprise.slack.com/archives/CLM1D8QR0) where needed **Settled:** Two rule sets, whole chain, triggering user, save-time failures block (Decision 2). MCP servers are iteration 2 (Decision 7). **Proposed, pending AppSec:** check every referenced agent at save and again at run. The save check gives the author a fast error. The run check is the security boundary, using the triggering user and the executing container. One resolver takes a `(user, container)` pair and runs four checks per agent, plus an existence and permission check per MCP server. **Still open:** Open Questions 3, 5, 6 and 7. ### 6. Error contract and user experience → folded into the issues above, no separate issue ~UX ~frontend **Owner:** ~"category:ai catalog creation" with ~"category:flow components" for error source changes **Purpose:** Give users clear feedback when a reference cannot be used. **Scope:** Define behaviour and messages for invalid YAML, unknown source or item type, missing item or version, item not visible, item not approved or enabled, nested dependency failure, MCP server blocked or not authenticated, DWS validation failure, DWS unavailable. The existing flow UI already distinguishes DWS validation failures from DWS unavailability :ok_hand_tone1: **Answered by `@michaelmoyers`** in [this comment](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3811287822). Recorded in the child issues: - Save blocks on anything checkable without running: invalid YAML, unknown source or item type, missing item or version, not visible, not enabled, broken nested reference. Run-time only: MCP blocked or not authenticated, DWS validation failure, DWS unavailable. See [#628153](https://gitlab.com/gitlab-org/gitlab/-/work_items/628153) and [#628315](https://gitlab.com/gitlab-org/gitlab/-/work_items/628315). - Errors always name the item. The YAML line is required only for structural errors. - Offer the item's suggested version if one is set, otherwise the plain list of versions. - Revoked access shows as a badge in the flow list and details on the flow page. See [#628318](https://gitlab.com/gitlab-org/gitlab/-/work_items/628318). **Still open:** Can a broken dependency be detected before a run, or only when a run fails? ### 7. Add Catalog-aware support to the YAML editor → [#628321](https://gitlab.com/gitlab-org/gitlab/-/work_items/628321) ~UX ~frontend (OUT OF SCOPE FOR ITERATION 1, moved to the parent epic) **Owner:** ~"category:ai catalog creation" **Purpose:** Make the feature usable without requiring users to memorize IDs and version syntax. **Scope:** Autocomplete or selectable Catalog agent references. Show item name, owner, type, and version. Warn before inserting an inaccessible item. Inline errors for invalid references. **Answered by `@michaelmoyers`** in [this comment](https://gitlab.com/gitlab-org/gitlab/-/work_items/627641#note_3811540085). Recorded in [#628321](https://gitlab.com/gitlab-org/gitlab/-/work_items/628321): - The YAML stores the raw `item_id`. The schema does not change. - The editor shows the item's readable name as a hint or hover label over the ID. - The editor links to the item's detail page. It does not show the agent definition inline. **Still open:** How extensible is the off-the-shelf Monaco editor? ~UX ### 8. Documentation → folded into the issues above, no separate issue **Owner:** whoever writes each change, in the same merge request as the code. Two rules make this part of the work rather than a follow-up. GitLab requires documentation for a milestone when a feature changes the user experience ([workflow](https://gitlab.com/gitlab-org/gitlab/-/blob/b46ed43f3151a50f92b6f43175657a00ac07d90d/doc/development/documentation/workflow.md#L23)) and asks for it in the same merge request as the code. The Flow Registry requires its version page to be updated in the same merge request as a framework change, because the Flow Creator and the other foundational agents read that page when they answer ([contribution guidelines](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/blob/45e709c3b7d2ea496aae2e7b10e1ad32a1d4ab79/docs/flow_registry/contribution_guidelines.md#keeping-framework-documentation-in-sync)). Who owns which page: | Issue | Pages | | ------ | ------ | | [#628153](https://gitlab.com/gitlab-org/gitlab/-/work_items/628153) | `flows/custom_flows_schema.md`, `flows/custom.md`, `flows/claude-edit-v1-flow-registry.md`, `agents/custom.md`, `ai_catalog.md`, the feature flag note | | [#628315](https://gitlab.com/gitlab-org/gitlab/-/work_items/628315) | a new development page for the `catalog_items` payload and the job variable, plus optional troubleshooting entries | | [#628317](https://gitlab.com/gitlab-org/gitlab/-/work_items/628317) | the delete sections in `flows/custom.md` and `agents/custom.md`, which currently say deletion is permanent | | [ai-assist#2870](https://gitlab.com/gitlab-org/modelops/applied-ml/code-suggestions/ai-assist/-/work_items/2870) | `docs/flow_registry/v1.md`, a new "Including AI Catalog Agents" section | | [gitlab-lsp#3082](https://gitlab.com/gitlab-org/editor-extensions/gitlab-lsp/-/work_items/3082) | the CLI reference table row for the new flag | No documentation-only issue is needed. Each issue names its own pages and line numbers.