## What is wrong `gitlab-com/runbooks` deploys one alert rule that pages for `team: artifact_registry`. The `artifact_registry` team record names no escalation target. So a page fires for this team, and the team catalog names nobody to escalate to. This is not about the lifecycle purger. The purger has no alert rule at all. The rule below is a saturation rule that landed with the metrics-catalog registration. ## What is true today All `gitlab-com/runbooks` reads below are at `origin/master` = `9c01d543381f43ecfa693bb82d5f8102655c4e6b`, read 2026-09-01. ### The deployed rule The rule is `component_saturation_slo_out_of_bounds:open_fds`. It lives in `mimir-rules/runway/artifact-registry-gke/autogenerated-runway-artifact-registry-gke-saturation-alerts.yml`. The `- alert:` line is line 331 of that file at that sha. Its expression is at lines 404 and 405, its `for: 5m` is at line 406, and its labels are at lines 407 to 412: ```yaml labels: alert_type: cause pager: pagerduty rules_domain: general severity: s2 team: artifact_registry ``` Line 404 of the same file selects `type="artifact-registry-gke"`, so the rule is scoped to this service: ```yaml gitlab_component_saturation:ratio{component="open_fds",type="artifact-registry-gke"} > on(component) group_left ``` ### The team record `services/teams.yml` lines 106 to 112 hold the whole `artifact_registry` record at that sha: ```yaml - name: artifact_registry cloud_cost: cost_owner: NOT_ASSIGNED url: https://handbook.gitlab.com/handbook/engineering/devops/package/ slack_channel: f_artifact_registry_dev slack_alerts_channel: f_artifact_registry_alerts send_slo_alerts_to_team_slack_channel: true ``` Seven lines, and none of them is `pagerduty_service:` or `oncall_schedule:`. ### The rule arrived through a registration, not through a decision `git log --diff-filter=A` on that file, the last command in the section named `## How to check every claim`, returns one commit at that sha. That commit is `9fef35ec8eb60437a9ed61b6e0b0786aa2ef59d1` of 2026-08-28, "observability(runway): register artifact-registry-gke in the metrics catalog". It reached `master` through [runbooks!11379](https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11379), merged 2026-08-31 as `0d9639ffe`. That merge request added the whole generated rule set for this service. Its description says nothing about a pager or an escalation target. The `pager` label is not a free choice either. `scripts/validate-alerts` lines 106 and 107 at that sha make it mandatory on an `s2` rule: ```ruby elsif (labels["severity"] == "s1") || (labels["severity"] == "s2") validation_error("s1 and s2 alerts must be configured to send to pagerduty") unless labels["pager"] == "pagerduty" ``` The generator set `severity: s2` for the open file descriptor saturation component. The validator then required `pager: pagerduty` beside it. So the team acquired a paging rule as a side effect of registering its metrics. ## Why this is a gap The `pager: pagerduty` label reaches a human today, and that human belongs to another team. `alertmanager/alertmanager.jsonnet` lines 398 to 406 at that sha send every alert with `env` in `gprd|ops|thanos|production` to the `incidentio` receiver, with `continue=true`. That route groups on `groupByTypeAndPager`, defined at lines 306 to 311 of the same file, which carries `type`, `env`, `environment` and `pager`. So the `pager` label travels to incident.io. Lines 643 to 648 of the same file add the `#production` Slack channel and stop there. Inside incident.io, the alert route "Alertmanager paging alerts to triage" escalates to the "GitLab.com Escalation Policy". `gitlab-com/gl-infra/production` issue 20792, "2025-11-10: Cut over from PagerDuty to incident.io On Call", records that change. The issue is public, closed on 2025-11-10, and labelled `change::complete`. One of its executed steps, marked done in the issue body, reads: > - [x] Enable "Escalate alerts" directly to the "GitLab.com Escalation Policy". Enable escalations to be automatically cancelled if the alert is resolved. **One step in that chain is an inference, not a read.** The filter conditions on the incident.io route were not read, because the route needs an authenticated incident.io session. Every alert with a matching `env` reaches the `incidentio` receiver, and most of those alerts carry no `pager` label. That is a large and continuous volume across the whole fleet, and the GitLab.com on-call gets no page for most of it. So the route filters on something, and `pager` is the label its `group_by` carries. The gap is therefore concrete. A saturation breach on this service escalates to the GitLab.com Production on-call engineer. The `artifact_registry` record names no target that engineer can hand the alert back to. ## What the repair is The key is `pagerduty_service:` on the `artifact_registry` record in `services/teams.yml`. `services/service-catalog-schema.json` lines 182 and 183 at that sha define it: ```json "pagerduty_service": { "description": "The PagerDuty Service name or ID used to escalate to the team.", ``` Do not use `oncall_schedule:`. Lines 149 and 150 of the same file mark it deprecated: ```json "oncall_schedule": { "description": "DEPRECATED: The on-call schedule of the team", ``` The `teams.yml` key is one half of the registration. `services/incident-io-importer.jsonnet` lines 136 to 141 import `pagerduty_service` into the incident.io team catalog as a `PagerDutyService` attribute. Lines 142 to 149 of the same file declare `escalation_path` with `schema_only: true`, so the importer creates that field and never fills it from `teams.yml`. incident.io holds the escalation path itself. **The key is rare in this catalog.** `services/teams.yml` holds 123 team records at that sha. Exactly 2 carry `pagerduty_service`: `sre_reliability` at line 9 and `gitaly` at line 57. For comparison, 77 records carry `slack_alerts_channel`. ## What work item 1011 does not settle Work item 1011 decided that the four proposed purger alerts get no on-call rotation. That decision covers four rules that do not exist yet and carry no `pager` label. The `open_fds` rule is a different rule. It is deployed, it carries `pager: pagerduty` today, and the 1011 decision leaves it exactly where it is. So the answer to 1011 does not close this gap, and folding the two together hides it. ## Questions this issue has to answer 1. Is the `open_fds` page intended for this team? 1. If the page is intended, which PagerDuty service name goes in the record? 1. If the page is not intended, what changes instead: the severity, the saturation threshold, or the `team` label? 1. Does the GitLab.com escalation policy accept an Artifact Registry saturation alert as in scope? Question 4 is a policy question. No file in either repository answers it. ## How to check every claim Run these from a `gitlab-com/runbooks` checkout, after `git fetch origin`. ```shell # The deployed rule and its labels. git grep -n "pager: pagerduty" origin/master -- mimir-rules/runway/artifact-registry-gke/ # The team record, in full. git show origin/master:services/teams.yml | sed -n '106,112p' # Every pagerduty_service key in the catalog. git grep -n "pagerduty_service" origin/master -- services/teams.yml # The schema entries for both keys. git grep -n "pagerduty_service\|oncall_schedule" origin/master -- services/service-catalog-schema.json # The validator rule that forces the pager label on s2. git show origin/master:scripts/validate-alerts | sed -n '100,109p' # How the rule file first appeared. git log --format='%H %cI %s' --diff-filter=A origin/master -- \ mimir-rules/runway/artifact-registry-gke/autogenerated-runway-artifact-registry-gke-saturation-alerts.yml ``` Positive controls, so an empty result is absence rather than a broken search: - `git grep -c "slack_alerts_channel" origin/master -- services/teams.yml` returns 77, so the tree reference and the path resolve. - `git show origin/master:services/teams.yml | grep -c '^ - name:'` returns 123, so the record count is measurable from the same file. - `git grep -c "team: artifact_registry" origin/master -- mimir-rules/runway/artifact-registry-gke/` returns 6 on the saturation file and 10 on the service-level file, so the rule directory resolves. ## Where the repair belongs The file to change is `services/teams.yml` in `gitlab-com/runbooks`. That repository belongs to another team. This issue records the gap and its evidence for Artifact Registry. It does not open a merge request against `gitlab-com/runbooks`, and it does not decide that the page is wrong. Related to https://gitlab.com/gitlab-org/ops/artifact-registry/-/work_items/1011 _This is a bot message 🤖 — /smurfit_