## Problem When we track `runner_system_failure` from CI job execution in Grafana, we do **not** distinguish between **customer-hosted (self-managed) runners** and **GitLab-hosted (SaaS) runners**. Both populations are folded into the same counter, even though ownership of the failure is fundamentally different: - A `runner_system_failure` on a **GitLab-hosted** runner is **GitLab's fault** — it should consume our error budget, fire our alerts, and (on Dedicated Hosted Runners) potentially count against contractual SLAs. - A `runner_system_failure` on a **customer-hosted** runner is generally **not attributable to GitLab** — the customer owns the runner host, executor environment, network, and autoscaler. Mixing them together means: - **Error budget leak / false positives** — customer infrastructure problems (bad K8s nodes, OOM, customer-side network/egress issues) inflate GitLab's infra-attributable error rate and can trip EOC alerts for failures we don't own. - **No clean "GitLab-hosted only" view** — there's no first-class way to slice the metric to the fleet GitLab operates without indirect proxies (e.g. shard names) that don't generalize across GitLab.com SaaS, Dedicated Hosted Runners, and self-managed. This is a **different axis** from the failure-reason taxonomy work in #596251 and gitlab-org/gitlab-runner#39355–39358. That effort splits `runner_system_failure` by **fault cause** (infra vs user-config); this issue splits by **runner ownership** (who operates the runner). They are complementary — ideally we want both dimensions. ## Current State Two metrics are in play and they behave differently: - **Rails-side:** `gitlab_ci_job_failure_reasons` powers the `job_infra_failure_ratio` SLI (numerator = positive include-list of infra-attributable reasons incl. `runner_system_failure`; denominator = all jobs). See `gitlab-com/runbooks`: `metrics-catalog/services/ci-orchestration.jsonnet`, `dashboards/ci-orchestration/pipeline-observability.dashboard.jsonnet`, `docs/ci-orchestration/alerts/CiOrchestrationServiceJobInfraFailureRatioErrorSLOViolation.md`. **This metric has no runner-ownership dimension** — every job's failure is counted regardless of whether it ran on a GitLab-hosted or customer-hosted runner. This is the gap. - **Runner-side:** `gitlab_runner_failed_jobs_total{failure_reason="runner_system_failure", ...}`. Hosted-runner alerts implicitly scope to the GitLab-hosted fleet via `job="hosted-runners-prometheus-agent"` and `shard=...` (see `gitlab-com/runbooks`: `reference-architectures/get-hybrid/config/prometheus-rules/hosted-runners.alerts.mixin.yml`, `docs/hosted-runners/runner_system_failure.md`). This works only because we scrape our own runners — `shard` is a fleet-internal concept, not an ownership label, and there is no equivalent for the Rails-side SLI. ## Desired Outcomes 1. **Ownership-aware metric** — `runner_system_failure` (and ideally all infra-attributable reasons) can be filtered to **GitLab-hosted runners only** vs **customer-hosted runners** in Grafana. 2. **Clean infra error budget** — the `job_infra_failure_ratio` SLI is updated (or a parallel GitLab-hosted-only view added) so customer-hosted runner failures do not leak into GitLab's error budget / alerts. 3. **No new blind spots** — the all-jobs view is preserved for regression detection; dashboards and hosted-runner alerts reflect the new dimension. ## Possible Approaches To be evaluated with Pipeline Execution / SLI owners: | Approach | Description | Trade-off | | -------- | ----------- | --------- | | A — New label | Add a runner-ownership label to `gitlab_ci_job_failure_reasons` (e.g. `runner_type` = instance/group/project, or derived `runner_owner` = `gitlab_hosted` vs `self_managed`). Rails knows the runner type at emit time. | Cleanest; cardinality impact to assess. | | B — Derive in rules | Infer ownership from an existing label in recording rules / dashboards, no new metric label. | No metric change; depends on an existing label being sufficient. | | C — Separate SLI | New SLI scoped to GitLab-hosted runners only, leaving the all-jobs metric intact. | Avoids touching the existing SLI; adds a parallel metric to maintain. | ## Acceptance Criteria - [ ] Confirm whether `gitlab_ci_job_failure_reasons` (or an adjacent metric) can carry a runner-ownership dimension, and whether Rails has the data at emit time to populate it. - [ ] Decide on the approach (label vs derived vs separate SLI) with Pipeline Execution and SLI owners. - [ ] `runner_system_failure` can be filtered to GitLab-hosted vs customer-hosted runners in Grafana. - [ ] `job_infra_failure_ratio` SLI updated (or parallel view added) so customer-hosted runner failures do not leak into GitLab's error budget. - [ ] Pipeline Observability dashboard and hosted-runner alerts updated; no new blind spots. - [ ] Runbook docs updated (`docs/ci-orchestration/...`, `docs/hosted-runners/...`). ## Open Questions 1. Does Rails already expose runner type on the job failure event, or is a new field/label required upstream of the metric? 2. Canonical ownership taxonomy — binary (GitLab-hosted vs self-managed), or finer (instance / group / project; GitLab.com SaaS vs Dedicated Hosted Runners vs self-managed)? 3. Coordinate into the in-flight taxonomy work (gitlab-org/gitlab-runner#39358) so ownership + cause land together, or ship independently? 4. Cardinality impact of adding a label on `gitlab_ci_job_failure_reasons`? ## Related Work - gitlab-org/gitlab#596251 — Investigate `runner_system_failure`: Root Cause Taxonomy and Observability Improvements (fault-cause axis) - gitlab-org/gitlab-runner#39358 — Rollout and Validation of New Failure-Reason Taxonomy with SLI Owners - gitlab-org/gitlab-runner#39535 — Reclassify plain/image-pull errors out of `runner_system_failure`