Formerly titled "AI Pipeline Builder — Success Metrics" — renamed to match the shipped product name and avoid confusion with the earlier, descoped canvas-based initiative. ## 1. Purpose This replaces the Beta-era metrics framework with a smaller set focused on business outcomes: is the agent getting more projects onto CI, and can users trust what it generates. Everything else is a diagnostic, not a success criterion. Note on scope vs. commitment: Section 2 defines what GA success looks like. Section 3 is a separate question — what instrumentation work is actually being committed — and "not gating" does not mean "optional." Every Section 3 row with an effort estimate is real engineering work; see 3.1 for what's ready to build vs. blocked vs. unscheduled. ## 2. North Star metrics (GA success criteria) | Metric | Definition | Baseline | Target (Q3 FY27) | Status | |---|---|---|---|---| | **Flow→commit conversion** | % of users who initiate the CI Expert Agent flow who go on to commit the agent-generated config | TBC | TBC — set once baseline lands | Effort S — Flow initiated (new click event) + Config committed (mostly in flight via `author_source`/foundation MRs) | | **Time-to-first-pipeline** | Time from project creation to first successful pipeline run, for AI-guided flow | TBC | <15–18 min | In progress — `ci_project_metrics.first_pipeline_succeeded_at`, targeting 19.0 | | **Config validity rate** | % of agent-generated pipelines with no `config_error` failure (invalid YAML/schema) | TBC | 95% | In progress — derived from existing `failure_reason` column, targeting 19.0 | This replaces platform-wide CI Activation as the top-line adoption signal for *this program* Config validity is kept distinct from general pipeline success on purpose: it isolates what the agent is actually responsible for — did it write correct config — from project-level test/environment failures it doesn't control. ## 3. Supporting diagnostics (not gating for GA success — see 3.1 for build status) | Metric | Role | Effort | Status | |---|---|---|---| | CI Activation (context only) | GitLab's company-wide CI adoption North Star — 34.4% of active projects using CI (Dedicated 65%, GitLab.com 44.4%, Self-Managed 27%). Useful background, not attributable to this program | — | Complete — tracked on the standalone "Verify North Star Metric Dashboard" (not DAP) | | Pipeline results reviewed | Whether users check what ran | S | Implementation merged (!241602) for the pipeline-details-page surface. Scope decision resolved 2026-07-09 — see below | | Pipeline survival (7d / 28d) | Whether adoption sticks past the first run | M | Blocked — metric definition (rolling-window vs. cohort) needs sign-off; open question was addressed to @dhershkovitch, who has since left GitLab and needs a replacement decision-maker | | First-run success rate (overall) | Broader than config validity — includes project code/test/env failures, not just config errors. Explains gaps between "valid config" and "pipeline actually worked" | M | Needs status check — new `first_pipeline_completed` event modeled on !231478, but source notes give conflicting status for that MR ("merged" vs. "still in review"); confirm before committing a timeline | | First-run failure reasons | Root-cause breakdown when metrics in Section 2 miss target | 0 incremental | Contingent on First-run success rate landing (same event) | | Pipeline complexity distribution | Context for interpreting success rate (simple vs. complex configs) | M/L | Scoped, not scheduled — known limitation: parser won't resolve `include:` directives | Agent attribution across all of the above rides on two merged foundation MRs (!232453, !232740) that stamp and persist an `author_source`/`ci_config_ai_generated` flag per project. **"Pipeline results reviewed" scope — resolved 2026-07-09:** kept to the pipeline-details-page surface only, as already merged in !241602. MR-widget views are not counted, since the agent's flow drops every user onto the MR automatically after committing — counting a widget render would collapse this metric into "reached the MR" (already implied by the prior funnel step, config committed) and erase the signal it exists to capture: whether users go on to actually verify the outcome. If there's real demand later for a genuine "glanced at inline status" signal, that needs its own deliberate interaction event (not a page-load proxy) and should be scoped as a separate follow-up issue. ### 3.1 Engineering commitment status (for the dev team) This is the answer to "do we have to build this": every row below except CI Activation requires engineering work if we want the diagnostic to exist. None of it is optional busywork — it's what makes Section 2's numbers explainable when they miss target. - **Done:** Pipeline results reviewed (S) — !241602 merged, scope confirmed to pipeline-details-page only. No further work needed unless a future issue is opened for inline-widget engagement. - **Scoped but blocked on a decision:** Pipeline survival (M) — needs a new owner to confirm metric definition, since the original approver left the company - **Scoped but needs status verification first:** First-run success rate (M) — confirm real status of !231478 before committing to the "modeled after" plan or timeline - **Contingent, no separate work:** First-run failure reasons — ships automatically once First-run success rate lands - **Scoped, not yet prioritized/scheduled:** Pipeline complexity distribution (M/L) — lowest priority of the set given effort vs. its "informative only" role ## 4. Out of scope for this framework The activation-vs-expansion namespace analysis (gitlab-data/product-analytics#3411) is a roadmap-prioritization question — which segment to target next — not a success metric for the current program. It should inform planning, not sit in the metrics doc. ## 5. What changed from the previous work item descripton - Renamed the doc/work item title from "AI Pipeline Builder — Success Metrics" to "CI Expert Agent — Success Metrics," matching the shipped product name rather than the earlier, descoped canvas-based initiative. Same fix applied to the "Flow→commit conversion" definition, which referenced "the AI Pipeline Builder flow." - Marked "Pipeline results reviewed" as resolved: scope confirmed to pipeline-details-page only (posted to the issue thread 2026-07-09), moved from "pending decision" to "Done" in 3.1.