AI Pipeline Builder – Phase 2: Initial Scoping Discussions
<!--IssueSummary start-->
<details>
<summary>
Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards.
</summary>
- [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=592921)
</details>
<!--IssueSummary end-->
## Planning update
- 1. **This issue has being renamed** to *AI Pipeline Builder – Phase 2: Initial Scoping Discussion* to reflect what it actually is — a scoping artifact, not a delivery epic.
- 2. **A [meta epic](https://gitlab.com/groups/gitlab-org/-/work_items/21577) has been created** as the canonical home for Phase 2 planning, sub-epics, and success metrics.
---
### Background
Phase 1 delivers an AI‐powered pipeline starter focused on helping new‐to‐CI projects create their first working build + test pipeline, powered by the Pipeline Authoring foundational agent which released as a Beta in 18.11.
## 1. Summary
### 2. Problem
As phase 1 solves the 0 → 1 activation problem, however, a large share of our revenue and usage comes from customers who already have CI, often with:
- Large, complex pipelines (hundreds–thousands of jobs).
- Legacy patterns and anti‐patterns (giant monolithic `.gitlab-ci.yml`, heavy job‐level rules, fragile includes).
- Under‐used best practices (CI Components, security scanners, caching/parallelization) despite AutoDevOps v2 and CI Catalog investments.
Phase 2 addresses the 1 → N optimization problem:
- Help maintainers and staff engineers continuously improve and modernize their pipelines.
- Do so safely, via explainable recommendations and merge requests, not opaque, auto‐applied mutations.
- Make these capabilities available where users work: GitLab UI, VS Code, and AI‐native tools like Cursor and Claude Code.
Phase 2 could extend this work to:
1. Serve existing customers with complex pipelines by analyzing, optimizing, and modernizing existing `.gitlab-ci.yml` according to GitLab's latest best practices and CI Components.
2. Integrate the Pipeline Authoring agent with the [AutoDevOps v2](https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/autodevops_v2/) CI Pipeline Agent so AutoDevOps v2 becomes the best‐practice way to operationalize the agent's CI intelligence across projects.
3. Integrate the CI Pipeline Agent with 3rd‐party AI tooling ecosystems (Cursor, Claude Code, other AI‐native IDEs), building on the CI Pipeline Language Service (LSP) work and VS Code integration.
4. Define a clear next location for ongoing pipeline authoring and optimization once users outgrow the initial agent flow, favoring IDE‐based experiences (for example, VS Code with the CI Pipeline Language Service) over the current Pipeline Editor, which does not work well with includable/component files.
### Opportunities
The opportunities list below should be broken into individual work item and prioritized accordingly using one of the prioritization frameworks (RICE)
#### Optimize existing complex pipelines
A large share of our usage and revenue comes from customers who already have CI and run large, complex pipelines. Today, they get little help from GitLab itself in understanding where their pipelines are inefficient, fragile, or missing critical best practices (for example, security scanning, caching, parallelization). Phase 2 is an opportunity to use the CI Pipeline Agent to analyze existing `.gitlab-ci.yml` files, surface concrete findings (performance issues, missing scanners, outdated patterns), and generate merge requests that safely modernize and optimize these pipelines. This turns GitLab Duo from a one‐time "first pipeline" helper into an ongoing CI advisor for our largest and most sophisticated users.
#### AutoDevOps 2.0
Today, AutoDevOps does not help existing pipelines: it only activates when there is no `.gitlab-ci.yml` and does not provide intelligent recommendations or continuous improvement for existing configs. Using ADO, we can implement the core logic of AutoDevOps v2 as a foundational agent based on a modified version of CI/CD Expert Agent, rather than building discrete microservices or rule‐based systems. This aligns Phase 2 with AutoDevOps v2's direction: a single CI Pipeline Agent that can both generate initial pipelines and act as a continuous improvement assistant for existing ones, reusing its commit‐validate loop and recommendation engine instead of duplicating logic.
#### 3rd‐party AI tooling ecosystems (Cursor, Claude Code, other AI‐native IDEs)
Many engineers now spend a significant portion of their time in AI‐native tools such as Cursor and Claude Code, in addition to VS Code. Those tools can already generate `.gitlab-ci.yml` snippets, but they have no access to GitLab's CI Components, CI agents, or AutoDevOps v2 intelligence. Phase 2 is an opportunity to expose the CI Pipeline Agent behind a well‐defined API / Flow that these tools can call: send the current CI config and intent ("optimize", "add security scans", "modularize with components"), receive back structured findings and a proposed patch, and then rely on GitLab for the final commit/MR. This lets GitLab stay the system of record for pipelines while meeting users where they are, and differentiates us from generic IDE agents that only see files, not pipelines.
#### Next location for ongoing pipeline work (beyond the initial agent flow)
The current Pipeline Editor is not well equipped for modern, modular CI setups that rely on components, includes, and (eventually) functions. It is also not where many engineers naturally prefer to work once the initial pipeline exists. Phase 2 is an opportunity to define and validate the "next home" for ongoing pipeline authoring and optimization: IDE‐based workflows (for example, VS Code) powered by the CI Pipeline Language Service and the CI Pipeline Agent. In this model, the initial "create pipeline with Duo" flow gets users from 0 → 1, and then deeper refactors, componentization, and optimization happen in the IDE using LSP features (validation, structure, visualization) plus agent‐driven suggestions and patches, while the Pipeline Editor remains focused on visualization and high‐level review rather than being the primary authoring tool for complex, includable configs.
### Success metrics (WIP)
* % of existing pipelines analyzed through the Pipeline Authoring agent (20%)
* Percentage of optimization recommendations accepted and merged (50% of the analyzed pipelines
### Related work: AutoDevOps v2
- AutoDevOps v2 issue (Verify stage):
https://gitlab.com/gitlab-org/verify-stage/-/issues/627
- MR adding the AutoDevOps v2 architecture design document:
https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/17498
- AutoDevOps v2 architecture design document (handbook):
https://handbook.gitlab.com/handbook/engineering/architecture/design-documents/autodevops_v2/
## Phase 2 Dependency & User Flow Diagram
```mermaid
graph TD
A["🎯 AI Pipeline Builder - Phase 2<br/>Pipeline Optimization & Ecosystem Integrations"]
subgraph Foundation["Foundation Layer"]
B["CI Best Practices Definition & Taxonomy<br/>#594375"]
end
subgraph Core["Core Agent Development"]
C["Pipeline Optimization Agent<br/>Analysis, Findings & MR Generation<br/>#594374"]
end
subgraph Integration["Integration Layer"]
D["AutoDevOps v2 Integration<br/>Continuous Improvement for Existing Pipelines<br/>#594376"]
E["3rd-Party AI Ecosystem Integration<br/>API, Content Model & MCP<br/>#594377"]
F["IDE-Based Authoring<br/>VS Code + LSP + Agent Experience<br/>#594378"]
end
subgraph UserFlows["User Flows"]
U1["Existing Pipeline Users<br/>Analyze & Optimize via Agent"]
U2["Existing Pipeline Users<br/>Continuous Improvement via AutoDevOps v2"]
U3["IDE Users<br/>Cursor, Claude Code, VS Code"]
end
A --> B
B --> C
C --> D
C --> E
C --> F
D --> U2
E --> U3
F --> U1
F --> U3
style A fill:#4B0082,stroke:#333,stroke-width:3px,color:#fff
style B fill:#FF6B6B,stroke:#333,stroke-width:2px,color:#fff
style C fill:#4ECDC4,stroke:#333,stroke-width:2px,color:#fff
style D fill:#45B7D1,stroke:#333,stroke-width:2px,color:#fff
style E fill:#45B7D1,stroke:#333,stroke-width:2px,color:#fff
style F fill:#45B7D1,stroke:#333,stroke-width:2px,color:#fff
style U1 fill:#FFA07A,stroke:#333,stroke-width:2px,color:#fff
style U2 fill:#FFA07A,stroke:#333,stroke-width:2px,color:#fff
style U3 fill:#FFA07A,stroke:#333,stroke-width:2px,color:#fff
```
issue