Route cell alerts to incident.io via tenant_type label

Vladimir Glafirovrequested to merge
vglafirov/route-cell-alerts-to-incidentio into master

Summary

Cell alerts arrive at the global Alertmanager with tenant_type=cell and tenant=<cell-name> labels (stamped by alert relabel configs in the k8s-monitoring-stack chart) but without an env label. The existing incidentio route only matches env=~gprd|ops|thanos, so cell alerts were falling through to the default prod_alerts_slack_channel receiver and never reaching incident.io.

This MR adds a new route matching tenant_type: cell that forwards alerts to the incidentio receiver with continue: true, mirroring the existing incidentio route behavior.

Changes

alertmanager/alertmanager.jsonnet

  • New route matching tenant_type: cell → incidentio receiver
  • Positioned right after the existing incidentio route for env=gprd|ops|thanos
  • Uses continue: true so alerts also reach terminal receivers (e.g., prod_alerts_slack_channel or production_slack_channel for pagerduty alerts)
  • Groups by alertname, tenant, tenant_type (cell-specific grouping)
  • Uses repeat_interval: 2h (same as existing incidentio route)

alertmanager/routing-tests.jsonnet

  • Test: basic cell alert (tenant_type=cell) routes to incidentio + prod_alerts_slack_channel
  • Test: cell alert with pager=pagerduty routes to incidentio + production_slack_channel

Background

  • Cell Prometheus sends alerts to Mimir's passthrough endpoint, which forwards them to the global Alertmanager at alerts.gitlab.net
  • The observability team (cc @knottos) confirmed this pipeline works via test alerts (silenced: 5360e279-3201-4e12-906c-bf4ea020ac64)
  • Prerequisite MRs already merged: gitlab-helmfiles!9575, gitlab-helmfiles!9577, charts!891

Verification

All 41 routing tests pass locally (make test-alertmanager), including the 2 new cell alert tests.

Closes gitlab-com/gl-infra/tenant-scale/cells-infrastructure/team#616 (closed)

/cc @devin @sabrams @romongbale

Merge request reports