CI Expert Agent — Success Metrics (Simplified, GA)
Formerly titled "AI Pipeline Builder — Success Metrics" — renamed to match the shipped product name and avoid confusion with the earlier, descoped canvas-based initiative.
1. Purpose
This replaces the Beta-era metrics framework with a smaller set focused on business outcomes: is the agent getting more projects onto CI, and can users trust what it generates. Everything else is a diagnostic, not a success criterion.
Note on scope vs. commitment: Section 2 defines what GA success looks like. Section 3 is a separate question — what instrumentation work is actually being committed — and "not gating" does not mean "optional." Every Section 3 row with an effort estimate is real engineering work; see 3.1 for what's ready to build vs. blocked vs. unscheduled.
2. North Star metrics (GA success criteria)
| Metric | Definition | Baseline | Target (Q3 FY27) | Status |
|---|---|---|---|---|
| Flow→commit conversion | % of users who initiate the CI Expert Agent flow who go on to commit the agent-generated config | TBC | TBC — set once baseline lands | Effort S — Flow initiated (new click event) + Config committed (mostly in flight via author_source/foundation MRs) |
| Time-to-first-pipeline | Time from project creation to first successful pipeline run, for AI-guided flow | TBC | <15–18 min | In progress — ci_project_metrics.first_pipeline_succeeded_at, targeting 19.0 |
| Config validity rate | % of agent-generated pipelines with no config_error failure (invalid YAML/schema) |
TBC | 95% | In progress — derived from existing failure_reason column, targeting 19.0 |
This replaces platform-wide CI Activation as the top-line adoption signal for this program
Config validity is kept distinct from general pipeline success on purpose: it isolates what the agent is actually responsible for — did it write correct config — from project-level test/environment failures it doesn't control.
3. Supporting diagnostics (not gating for GA success — see 3.1 for build status)
| Metric | Role | Effort | Status |
|---|---|---|---|
| CI Activation (context only) | GitLab's company-wide CI adoption North Star — 34.4% of active projects using CI (Dedicated 65%, GitLab.com 44.4%, Self-Managed 27%). Useful background, not attributable to this program | — | Complete — tracked on the standalone "Verify North Star Metric Dashboard" (not DAP) |
| Pipeline results reviewed | Whether users check what ran | S | Implementation merged (!241602 (merged)) for the pipeline-details-page surface. Scope decision resolved 2026-07-09 — see below |
| Pipeline survival (7d / 28d) | Whether adoption sticks past the first run | M | Blocked — metric definition (rolling-window vs. cohort) needs sign-off; open question was addressed to @dhershkovitch, who has since left GitLab and needs a replacement decision-maker |
| First-run success rate (overall) | Broader than config validity — includes project code/test/env failures, not just config errors. Explains gaps between "valid config" and "pipeline actually worked" | M | Needs status check — new first_pipeline_completed event modeled on !231478 (merged), but source notes give conflicting status for that MR ("merged" vs. "still in review"); confirm before committing a timeline |
| First-run failure reasons | Root-cause breakdown when metrics in Section 2 miss target | 0 incremental | Contingent on First-run success rate landing (same event) |
| Pipeline complexity distribution | Context for interpreting success rate (simple vs. complex configs) | M/L | Scoped, not scheduled — known limitation: parser won't resolve include: directives |
Agent attribution across all of the above rides on two merged foundation MRs (!232453 (merged), !232740 (merged)) that stamp and persist an author_source/ci_config_ai_generated flag per project.
"Pipeline results reviewed" scope — resolved 2026-07-09: kept to the pipeline-details-page surface only, as already merged in !241602 (merged). MR-widget views are not counted, since the agent's flow drops every user onto the MR automatically after committing — counting a widget render would collapse this metric into "reached the MR" (already implied by the prior funnel step, config committed) and erase the signal it exists to capture: whether users go on to actually verify the outcome. If there's real demand later for a genuine "glanced at inline status" signal, that needs its own deliberate interaction event (not a page-load proxy) and should be scoped as a separate follow-up issue.
3.1 Engineering commitment status (for the dev team)
This is the answer to "do we have to build this": every row below except CI Activation requires engineering work if we want the diagnostic to exist. None of it is optional busywork — it's what makes Section 2's numbers explainable when they miss target.
- Done: Pipeline results reviewed (S) — !241602 (merged) merged, scope confirmed to pipeline-details-page only. No further work needed unless a future issue is opened for inline-widget engagement.
- Scoped but blocked on a decision: Pipeline survival (M) — needs a new owner to confirm metric definition, since the original approver left the company
- Scoped but needs status verification first: First-run success rate (M) — confirm real status of !231478 (merged) before committing to the "modeled after" plan or timeline
- Contingent, no separate work: First-run failure reasons — ships automatically once First-run success rate lands
- Scoped, not yet prioritized/scheduled: Pipeline complexity distribution (M/L) — lowest priority of the set given effort vs. its "informative only" role
4. Out of scope for this framework
The activation-vs-expansion namespace analysis (gitlab-data/product-analytics#3411) is a roadmap-prioritization question — which segment to target next — not a success metric for the current program. It should inform planning, not sit in the metrics doc.
5. What changed from the previous work item descripton
- Renamed the doc/work item title from "AI Pipeline Builder — Success Metrics" to "CI Expert Agent — Success Metrics," matching the shipped product name rather than the earlier, descoped canvas-based initiative. Same fix applied to the "Flow→commit conversion" definition, which referenced "the AI Pipeline Builder flow."
- Marked "Pipeline results reviewed" as resolved: scope confirmed to pipeline-details-page only (posted to the issue thread 2026-07-09), moved from "pending decision" to "Done" in 3.1.