## What is wrong Artifact Registry has 16 deployed alert rules that carry `team: artifact_registry`. None of them reaches the `#f_artifact_registry_alerts` Slack channel. The team-alert route matches `env` against the exact value `gprd`, and this service emits `env=production`. The channel is the only alert destination this team owns. So Artifact Registry today has alert rules and no alert delivery to its own channel. The four purger alert rules that `docs/dev/alerting.md` proposes inherit the same defect, whatever severity they take. Nothing in this repository is broken. The value that fails the match is correct, and the route that rejects it lives in `gitlab-com/runbooks`. ## How to read this issue Every claim below carries the file, the function or the line, and the sha it was read at. Each negative claim carries a positive control, so an empty result is absence rather than a broken search. Re-derive the chain rather than taking it on authority. Two limits on the evidence, stated up front: - The routing chain was read from the jsonnet source, not from a generated `alertmanager.yml`. `.gitignore` line 20 of `gitlab-com/runbooks` excludes that generated file, so no tree holds it. The section named `## Closing the last gap` gives the command that generates it. - No live Alertmanager instance and no Grafana query were read. Read times and shas: | Repository | Ref | Sha | | --- | --- | --- | | `gitlab-org/ops/artifact-registry` | `origin/main` | `7f766481cd1d175c9b2ba552cf1579088b889655` | | `gitlab-com/runbooks` | `origin/master` | `9c01d543381f43ecfa693bb82d5f8102655c4e6b` | | `gitlab-com/gl-infra/platform/runway/docs` | `master` | `21d298a25287ac4317e22e66a4ea17f97a07d42f` | All three were read on 2026-09-01. ## Step 1: the team route matches on `env` and `team` `alertmanager/alertmanager.jsonnet` lines 515 to 523 at `gitlab-com/runbooks` `origin/master` = `9c01d5433` build one route per team that has a Slack alert channel: ```jsonnet Route( receiver=receiverNameForTeamSlackChannel(team.name), continue=true, matchers={ env: envForTeams(team), team: team.name, }, ) for team in teamsWithAlertingSlackChannels() ``` Line 519 is the matcher line this issue is about: ```jsonnet env: envForTeams(team), ``` `teamsWithAlertingSlackChannels()` is defined at lines 268 to 270 of the same file. It selects every team record with a non-empty `slack_alerts_channel`. The `artifact_registry` record has one, so the route exists. Lines 679 to 682 of the same file build the receiver that route names: ```jsonnet [SlackReceiver({ name: receiverNameForTeamSlackChannel(team.name), channel: team.slack_alerts_channel, }) for team in teamsWithAlertingSlackChannels()] + ``` So the receiver posts to `f_artifact_registry_alerts`. The channel is reachable. Only the `env` matcher stops the alert. ## Step 2: `envForTeams` returns a bare string for this team `alertmanager/alertmanager.jsonnet` lines 353 to 356 at the same sha: ```jsonnet // envForTeams will check if the team is opting-in any aother environment apart // from the required `gprd` environment. local envForTeams(team) = if std.objectHas(team, 'alerts') then { re: std.join('|', ['gprd'] + team.alerts) } else 'gprd'; ``` The function has two arms. With an `alerts` key on the record, it returns `{ re: 'gprd|<the listed values>' }`. Without one, it returns the bare string `'gprd'`. `services/teams.yml` lines 106 to 112 at the same sha hold the whole `artifact_registry` record: ```yaml - name: artifact_registry cloud_cost: cost_owner: NOT_ASSIGNED url: https://handbook.gitlab.com/handbook/engineering/devops/package/ slack_channel: f_artifact_registry_dev slack_alerts_channel: f_artifact_registry_alerts send_slo_alerts_to_team_slack_channel: true ``` Seven lines, and no `alerts:` key. So `envForTeams` takes the second arm and returns `'gprd'`. **Positive control for that absence.** `git grep -n "^ alerts:" origin/master -- services/teams.yml` returns 10 lines: 285, 299, 562, 575, 937, 1011, 1089, 1104, 1119 and 1145. None of them falls between 106 and 112. The pattern matches in that file, so the absence is real. ## Step 3: a bare string is an equality matcher, and a regex is not This step decides the outcome, so the trace below goes down to the serializer. `alertmanager/alertmanager.jsonnet` line 325 at the same sha passes every matcher hash through one function: ```jsonnet [if matchers != null then 'matchers']: selectors.alertManagerMatchers(matchers), ``` `libsonnet/promql/selectors.libsonnet` lines 159 to 161 at the same sha define it, and it calls `serializeHashItem` per key. Lines 4 to 9 of that file state the five value forms: ```jsonnet // serializeItem supports 5 forms for the value: // 1: for string values: -> `label="value"` // 2: for equality values { eq: "value" } -> `label="value"` // 3: for non-equality values { ne: "value" } -> `label!="value"` // 4: for regex-match values { re: "value" } -> `label=~"value"` // 5: for non-regex-match values { nre: "value" } -> `label!~"value"` ``` Lines 68 to 70 of the same file implement form 1: ```jsonnet local serializeHashItem(label, value) = if std.isString(value) || std.isNumber(value) then serializeItems(label, '=', value) ``` Line 58 of the same file implements form 4: ```jsonnet re: function(label, value) serializeItems(label, '=~', value), ``` The operator string is the whole difference. A bare string becomes `=`, which is Alertmanager equality. A `{ re: ... }` object becomes `=~`, which is a regex match. Both forms appear in the same routing tree, a few lines apart. The feature-category route at lines 490 to 493 uses the regex form: ```jsonnet matchers={ env: { re: 'gprd|thanos|production' }, // For now we only send production channel alerts to teams feature_category: featureCategoryTeam.featureCategory, }, ``` The team route at lines 518 to 521 uses the string form, through `envForTeams`: ```jsonnet matchers={ env: envForTeams(team), team: team.name, }, ``` For `artifact_registry`, that renders as `env="gprd"`. An alert carrying `env="production"` does not match it. The feature-category route above admits `production`, and the team route does not. `docs/dev/alerting.md` line 165 in this repository, at `origin/main` = `7f766481c`, already records the equality point: > The team route in `alertmanager/alertmanager.jsonnet` matches `team` against each team's own name and `env` against the value the last paragraph of this section covers, and both are equality matchers, so a rule missing either label matches no team route and lands in no team's Slack channel. The mechanism in this repository is correct. The value is the part that is wrong. ## Step 4: the `env` value is `production` Three sources establish it, and one of them is in this repository. **This repository.** `docs/dev/gitlab-com-infrastructure.md` lines 66 to 68 at `origin/main` = `7f766481c`: > The `kube_pod_container_status_ready` join narrows the result to containers > serving traffic. Add `env="staging"` or `env="production"` to scope it to one > environment; an environment with no matching series has no running workload. That paragraph is about the `mimir-runway` datasource, which is the same tenant these alert rules read. It names two values, and `gprd` is not one of them. Lines 84 and 85 of the same file repeat the pair on the ClickHouse side: `Attributes['env'] = 'staging'` or `'production'`. Line 88 of the same file says why: "AR runs on the GKE runtime". **Runway's own documentation.** `src/content/docs/reference/observability.md` at `gitlab-com/gl-infra/platform/runway/docs` `master` = `21d298a25`, line 447: > Runway Kubernetes services use `env=production` and `env=staging` (not `gprd`/`gstg`). The `stage` label is always `stage=main` — Runway services do not have a canary stage. Cloud Run services use `env=gprd` and `env=gstg`. Line 60 of the same file, in the load-balancer label table: > | `env` | `production`, `staging` (GKE/EKS) / `gprd`, `gstg` (Cloud Run) | Runway environment | Line 358 of the same file, in the label table for a service's own scraped metrics: > | `env` | Environment | `staging`, `production` | Line 358 is the one that governs these rules. The generated Artifact Registry rules and the four proposed purger rules read service metrics, not Cloud Run metrics. **Artifact Registry is a Runway Kubernetes service.** `.runway/deployment.yaml` at `origin/main` = `7f766481c` declares `apiVersion: runway/v2`, `kind: RunwayManifest` and `metadata.name: artifact-registry-gke`. `metrics-catalog/services/artifact-registry-gke.jsonnet` line 9 at runbooks `9c01d5433` passes `runtime='gke'`. Line 7 of the same file passes `team='artifact_registry'`, which is where the `team` label comes from. So the GKE arm of the documented rule applies, and the value is `production`. ## Step 5: the 16 rules `git grep -c "team: artifact_registry" origin/master -- mimir-rules/runway/artifact-registry-gke/` at runbooks `9c01d5433` returns: ```plaintext origin/master:mimir-rules/runway/artifact-registry-gke/autogenerated-runway-artifact-registry-gke-saturation-alerts.yml:6 origin/master:mimir-rules/runway/artifact-registry-gke/autogenerated-runway-artifact-registry-gke-service-level-alerts.yml:10 ``` 6 plus 10 is 16. The same directory holds 17 alert rules in total. The seventeenth is `KubeContainersWaitingInError`, which carries `team: sre_reliability` and is not this team's rule. The 16 rules carry 13 distinct alert names. The table adds the seventeenth rule in its last row, for contrast. | Alert | File | Rules | Severity | | --- | --- | --- | --- | | `component_saturation_slo_out_of_bounds:kube_container_cpu_limit` | saturation-alerts | 1 | `s4` | | `component_saturation_slo_out_of_bounds:kube_container_memory_limit` | saturation-alerts | 1 | `s4` | | `component_saturation_slo_out_of_bounds:kube_container_memory_request` | saturation-alerts | 1 | `s4` | | `component_saturation_slo_out_of_bounds:kube_container_throttling` | saturation-alerts | 1 | `s3` | | `component_saturation_slo_out_of_bounds:kube_horizontalpodautoscaler_desired_replicas` | saturation-alerts | 1 | `s3` | | `component_saturation_slo_out_of_bounds:open_fds` | saturation-alerts | 1 | `s2` | | `ArtifactRegistryGkeServiceRunwayBackendApdexSLOViolation` | service-level-alerts | 2 | `s4` | | `ArtifactRegistryGkeServiceRunwayBackendErrorSLOViolation` | service-level-alerts | 2 | `s4` | | `ArtifactRegistryGkeServiceRunwayBackendTrafficCessation` | service-level-alerts | 1 | `s4` | | `ArtifactRegistryGkeServiceRunwayBackendTrafficAbsent` | service-level-alerts | 1 | `s4` | | `ArtifactRegistryGkeServiceRunwayGkeLbErrorSLOViolation` | service-level-alerts | 2 | `s4` | | `ArtifactRegistryGkeServiceRunwayGkeLbTrafficCessation` | service-level-alerts | 1 | `s4` | | `ArtifactRegistryGkeServiceRunwayGkeLbTrafficAbsent` | service-level-alerts | 1 | `s4` | | `KubeContainersWaitingInError` | kube-cause-alerts | 1 | `s2`, `team: sre_reliability` | Three names appear twice because they carry two burn-rate windows. `git grep -n "^ window:" origin/master -- mimir-rules/runway/artifact-registry-gke/autogenerated-runway-artifact-registry-gke-service-level-alerts.yml` returns six lines, alternating `1h` and `6h`. The `team` label on these rules is not accidental. `libsonnet/slo-alerts/slo-alert-labels.libsonnet` lines 31 to 34 at the same sha add it: ```jsonnet if team != null && team.send_slo_alerts_to_team_slack_channel then { team: sli.team } else {} ``` The `artifact_registry` record sets `send_slo_alerts_to_team_slack_channel: true`, at line 112 of `services/teams.yml`. ## The claim this contradicts [runbooks!11379](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11379) registered this service, and merged on 2026-08-31 as `0d9639ffe`. Its own description states, in the section "Details worth reviewer attention": > - New `artifact_registry` team in `teams.yml` routes SLO alerts to `#f_artifact_registry_alerts`. The route exists, and the alerts do not reach it. Runway's documentation states the same expectation at line 443 of `reference/observability.md` at `21d298a25`: > - To route alerts to a team Slack channel, specify a valid `team` in your metrics catalog entry. The metrics catalog entry does specify a valid team, at line 7 of `metrics-catalog/services/artifact-registry-gke.jsonnet`. The `env` matcher is what stops the delivery, and neither document mentions it. ## Positive controls Each negative claim above, and the command that shows the search works. | Negative claim | Command that returns nothing | Positive control on the same file and tool | | --- | --- | --- | | The `artifact_registry` record has no `alerts:` key | `git grep -n "^ alerts:" origin/master -- services/teams.yml` shows no hit between lines 106 and 112 | The same command returns 10 hits elsewhere in that file | | No routing test covers this team | `git grep -c "artifact" origin/master -- alertmanager/routing-tests.jsonnet` exits 1 with no output | `git grep -c "gitaly" origin/master -- alertmanager/routing-tests.jsonnet` returns 9 | | The team record path resolves | — | `git grep -c "slack_alerts_channel" origin/master -- services/teams.yml` returns 77 | | The rule directory resolves | — | `git grep -c "team: artifact_registry" origin/master -- mimir-rules/runway/artifact-registry-gke/` returns 6 and 10 | The second row matters on its own. `alertmanager/routing-tests.jsonnet` asserts receiver sets for many label combinations, and CI gates the Alertmanager deploy on it. No case in that file names this team, so the test suite asserts nothing about this route. ## The second half: `docs/dev/alerting.md` carries the wrong value The four proposed purger rules in this repository select `env="gprd"`. Under the derived value, every one of them selects nothing. A rule that selects nothing never fires, and nothing reports that it never fires. Measured at `origin/main` = `7f766481cd1d175c9b2ba552cf1579088b889655`. `git grep -n 'env="gprd"' origin/main -- docs/dev/alerting.md` returns eight lines: 59, 60, 62, 66, 88, 128, 130 and 153. This run re-derived those line numbers at that sha, and they are unchanged. | Line | Rule it belongs to | Metric selected | | --- | --- | --- | | 59, 60 | Alert 1, tombstone backlog | `gitlab_artifact_registry_lifecycle_tombstones` | | 62 | Alert 1, the narrowing operand | `gitlab_artifact_registry_database_queries_total` | | 66 | Alert 1, the `retention_window: 0s` variant | `gitlab_artifact_registry_lifecycle_tombstones` | | 88 | Alert 2, sweep errors | `gitlab_artifact_registry_jobs_processed_total` | | 128, 130 | Alert 3, sweep liveness | `gitlab_artifact_registry_jobs_processed_total` | | 153 | Alert 4, purge outcomes | `gitlab_artifact_registry_lifecycle_purge_outcomes_total` | Line 164 of the same file states the delivery condition, and its second half is wrong under the derived value: > That reads as reassuring, and delivered the same way is not the same as delivered: the routing keys on two labels rather than on `severity`, so both severities reach no team channel at all until every rule carries `team: artifact_registry` and `env: gprd` beside its `severity` and its `alert_type`. Lines 228 and 231 of the same file record the value as open and reach the same wrong conclusion: > Delivery to `#f_artifact_registry_alerts` therefore needs `env` to be exactly `gprd` today, and adding an `alerts` key to that record is the change that would widen it. Line 232 names a query to settle it: > Running `count by (env) (runway_lb_request_count{type="artifact-registry-gke"})` against the `mimir-runway` datasource settles which value these rules take. That query returns nothing under either value. `libsonnet/service-archetypes/runway-k8s-archetype.libsonnet` lines 30 to 32 at runbooks `9c01d5433` say why: > // Required because GKE Gateway auto-generates url_map_name with a hash suffix and > // does not support custom labels on forwarding rules — so type label cannot be set > // on frontend LB metrics. Line 38 of `reference/observability.md` at `21d298a25` lists that metric's labels, and `type` is not among them: > | `runway_lb_request_count` | Total requests to load balancer | `runtime`, `env`, `load_balancer` (AWS) / `forwarding_rule_name` (GCP) | An empty result reads as "no data yet", so anyone who runs that query parks the question again. This correction is independent of the on-call decision in work item 1011. It applies under a rotation and without one. ## Where the repair belongs The routing repair is one key on the `artifact_registry` record in `services/teams.yml`, in `gitlab-com/runbooks`. An `alerts:` list widens `envForTeams` from the bare string `'gprd'` to `{ re: 'gprd|production' }`. The value is not novel in that catalog. `services/teams.yml` lines 1089 to 1091 at `9c01d5433` give the `runway` team record: ```yaml alerts: - gprd - production ``` Lines 1104 to 1106 of the same file give `fleet_management` the same pair. `gitlab-com/runbooks` belongs to another team, and this issue does not open a merge request against it. What belongs here is the Artifact Registry impact. 16 deployed rules reach no channel this team watches, and the four proposed purger rules inherit the same gap. The `docs/dev/alerting.md` corrections listed above are this repository's own work and need nobody else. ## Closing the last gap The routing chain above was read from jsonnet source. Generating the Alertmanager configuration closes it, from a `gitlab-com/runbooks` checkout with `jsonnet` and `amtool` installed: ```shell make alertmanager/alertmanager.yml grep -n -A6 'team_artifact_registry_alerts_channel' alertmanager/alertmanager.yml ``` The generated route for this team must show `env="gprd"` as an equality matcher. `Makefile` lines 57 and 58 at `9c01d5433` define that target, and `alertmanager/generate.sh` runs the jsonnet. `.gitlab-ci.yml` line 783 at the same sha runs the same target in the deploy job, with the real secrets. So the generated file describes the deployed configuration. ## Not established here - The value `production` rests on merged documentation, not on a live query. A corrected query against the `mimir-runway` datasource is `count by (env) (gitlab_artifact_registry_jobs_processed_total)`. Nobody has run it. - Whether any `gitlab_artifact_registry_*` series reaches the Mimir tenant at all. `docs/dev/alerting.md` lines 226 and 227 already record that as open. - Whether an `alerts:` key is the repair the `gitlab-com/runbooks` owners prefer. Issue #1003 covers a different defect in the same area: Artifact Registry documentation states that the service has no alerting at all. Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1011 _This is a bot message 🤖 — /smurfit_