Grafana dashboards and alerting for the .com closed beta

What

Create the Grafana dashboards and alerting for the Artifact Registry Runway services before the .com closed beta.

Runway generates a default service dashboard and alerts (apdex, error rate, traffic cessation, saturation) once a service has a metrics-catalog entry. Anything service-specific on top is our work. See Runway observability and the self-service list in runway/team#965.

Plan

Phase 1: registration (gitlab-com/runbooks)

Split into a 3-MR stack per reviewer request, modeled on the iam-data-access registration (ecd20e44a6). Each MR carries its own generated dashboards, SLO recording rules, and default alerts.

  • runbooks!11379 — artifact-registry-gke (runway-k8s-archetype) + teams.yml team artifact_registry (SLO alerts to #f_artifact_registry_alerts).
  • runbooks!11381 — runway-db-artifact-registry-gke (postgres archetype).
  • runbooks!11382 — runway-redis-artifact-registry-gke (redis archetype); unstacked and merged independently.

Phase 2: SLI feedstock

  • runway_lb_* staleness: no defect. Per-class series only emit when that class's traffic occurs; apdex excludes 4xx/5xx by design and fills in once closed-beta traffic flows.
  • Prod Memorystore metrics missing platform-wide (#f_runway thread): fixed 2026-09-03 by argocd/apps!3416 — the exporter's 2m interval was shorter than Memorystore's ingest delay. Production series verified.
  • Custom (vanity) domain support: asked in #f_runway, tracked in #1000 (closed).
  • GCS + Cloud CDN export: dedicated Stackdriver exporter (ai-framework pattern), live in both environments and validated 2026-09-02 — GCS and CDN LB series for gl-ar-storage-bucket-{stg,prd} flow into the Runway Mimir tenant. config-mgmt!15245 (IAM, applied) + argocd/apps!3344 (gstg) + argocd/apps!3376 (gprd). The dashboard is the Phase 3 "Storage and CDN" row.

Phase 3: custom dashboards

Kind-first structure (hosted / remote, virtual later), not per-format: failure domains follow repository kind, and format lives in metric names so a format template variable is fragile. Multi-format graphs (one series per format, never summed across families) plus per-format rows. Panel inventory: metrics catalog.

  • runbooks!11420 — Overview (extend the generated one with top-level app KPIs)
  • runbooks!11422 — HTTP API (requests, latency, sizes, in-flight, panics by route/method/status)
  • runbooks!11423 — Database (named-query latency and rates, pool state, CloudSQL); pool pressure panels in runbooks!11470 (merged) on the new pgx metrics (AR!2403, AR!2406, AR!2407, all merged); churn and per-query error panels in runbooks!11483, merged
  • Auth and authz (denials by reason, GLAZ and IAM RPCs, relationship cache). The denial_reason vocabulary settled on main (no open MR moves the authz metric surface), so the dashboard is built and playground-validated; its MR opens once the Authentication row is rewritten onto the client-authentication outcome metrics being implemented for #1267 (closed). While that AR change is undeployed the auth panels are empty; they populate on its first production scrape. Follow-up owed here: the upload-session concurrency-cap plan (docs/plans/2026-09-03-upload-session-concurrency-cap.md) adds metric families as its steps land — panel them once the implementing MRs merge.
  • runbooks!11479 — Hosted (cross-format health row, collapsed per-format rows, upload sessions, purge, faults). Format-neutral publish metrics follow-up: #1252.
  • runbooks!11468 — Remote (upstream outcomes, cache verdicts and fills, container upstream auth, proxy faults). Restructured into the cross-format layout in runbooks!11480, merged.
  • runbooks!11502 — Jobs (throughput, duration and in-flight, enqueue path, storage-accounting drains, reconciliation), shaped from the families the jobs code emits across both backends. A runtime row for the Overview (Go memory, goroutines, GC, CPU, FDs) follows in runbooks!11504.
  • Usage data (drops, unattributed events, Snowplow emitter pipeline)
  • runbooks!11421 — Storage and CDN
  • Virtual (when S30-S32 ship)

Phase 4: AR-specific alerts

  • Default SLO burn-rate alerting verified end to end 2026-09-07: the archetype's rules evaluate (apdex/error ratio 0.999, severity s4) and route to #f_artifact_registry_alerts via teams.yml. The staging RunwayBackendApdexSLOViolation flapping traced to single-replica rollouts (GKE gateway NEG lag during pod replacement, invisible to the app); AR!2404 raises minInstances to 2 in both environments to close it.
  • High-signal set on top of the defaults (authz denial spikes, buffered-counter loss rates, usage-data drops, npm deadline kills), thresholds tuned, each linked to a runbook page.

Phase 5: brand-list-validator

Out of scope

  • SM CNG operational alerts and runbooks (#299).
  • New application-side metrics; this issue only consumes what the service already exposes.

Related to Artifact Registry: Provision required infrastructure components for GitLab.com (gl-infra&1999).

Edited by João Pereira