Alert rules for S27 background jobs: periodic heartbeats including counter drain, discards, enqueue errors and queue age
## Summary
The service runs background jobs on two backends: River (PostgreSQL) for 15 job kinds, 7 of them periodic, and asynq (Redis) for 1 kind, `reconciliation:namespace`. Every kind reports through one shared set of `gitlab_artifact_registry_jobs_*` metrics (`internal/jobs/middleware.go:72-97`, registered at `cmd/artifact-registry/wire_jobs.go:154`). No alert rule reads them today, and no generated SLO alert covers job processing. A stopped scheduler, a stalled drain or a dead-lettered bulk delete reaches nobody. This issue adds the rules that apply to all job kinds. Rules for one feature's job kinds stay with that feature's issue: the purger (S20) with #1599 and reconciliation (S22) with #1600.
## Job kinds and cadences
All kinds use the `default` queue. All periodic schedules are River and leader-elected, and none sets `RunOnStart`.
| Kind | Backend | Schedule (default) | Attempts | Alert owner |
| --- | --- | --- | --- | --- |
| `trigger:counter_drain_repo` | river | every `storage_accounting.drain_interval` (30s) (`internal/accounting/register.go:39-41`) | 25 (River default) | this issue |
| `trigger:counter_drain_namespace` | river | 30s (`register.go:42-44`) | 25 | this issue |
| `trigger:counter_drain_downloads` | river | 30s (`register.go:45-47`) | 25 | this issue |
| `counter_drain_chunk_repo` / `_namespace` / `_downloads` | river | enqueued by the drain triggers (`internal/accounting/chunk_args.go:99,113,126`) | 25 | this issue |
| `trigger:reconciliation` | river | `reconciliation_interval` (1h) (`register.go:48-51`) | 3 (`internal/accounting/reconcile_trigger.go:56`) | heartbeat: #1662 |
| `trigger:reconciliation_orphan_sweep` | river | `reconciliation_orphan_sweep_interval` (3h, minimum 30m) (`register.go:52-55`) | 3 | none, by design (`docs/dev/alerting.md:417`) |
| `reconciliation:namespace` | asynq | enqueued by the reconciliation triggers (`reconcile_trigger.go:338`) | asynq default | discard: #1662 |
| `lifecycle:tombstone-sweep` | river | `lifecycle.sweep_interval` (5m) (`internal/lifecycle/register.go:36-37`) | 25 | #1661 |
| `lifecycle:purge` | river | enqueued by the sweep and the tombstone writer | 1 (`internal/lifecycle/purge.go:210`) | #1661 |
| `remote:health-sweep` | river | `scheduled_interval` (5m) (`internal/remote/healthjob.go:400-403`) | 1 | this issue |
| `MavenMetadataReconciler` | river | enqueued on Maven deletes; `CancelOnPanic` (`internal/format/maven/reconciler_job.go:181`) | 25 | this issue |
| `mgmtapi:bulk-delete-container` / `-maven` / `-npm` | river | enqueued by the bulk-delete API and remote eviction (`internal/managementapi/bulk_args.go:133,212,255`) | 25 | this issue |
| `unregistered` (synthetic) | asynq | any task type with no handler (`internal/jobsasynq/observability.go:26`) | - | this issue |
## Alerts
Severity note: `scripts/validate-alerts` in `gitlab-com/runbooks` rejects `s1` or `s2` without a `pager` label, and the reverse (`docs/dev/alerting.md:213-215`). There is no on-call rotation yet (#1469), so every rule here ships at `s3` or `s4`, Slack only. The "Page after #1469" column marks the rules to raise to `s2` with `pager: pagerduty` once the rotation exists.
Selector convention: the sketches follow the two deployed rules in `mimir-rules/runway/artifact-registry-gke.yml` (`environment="production"` selector, static `env` label). The staging copies, and the final `env` value, follow #1658 and #1049.
All expressions select `environment="production"` and set `env: gprd` in `labels:`, as the two deployed hand-written rules in `mimir-rules/runway/artifact-registry-gke.yml` do. The staging variant swaps in `environment="staging"` and `env: staging` (#1658). Severity cannot be `s1` or `s2` while paging is deferred, because `scripts/validate-alerts` rejects an `s1`/`s2` rule without a `pager` label (`docs/dev/alerting.md`, severity-split paragraph). Rules marked "Page after #1469? yes" move to `s2` with `pager: pagerduty` when #1469 lands.
| Name | What it detects | Expression sketch | For / window | Severity | Page after #1469? | Status | Depends on |
| --- | --- | --- | --- | --- | --- | --- | --- |
| `ArtifactRegistryJobsCounterDrainStopped` | A counter-drain trigger has had no successful fire. This also means the River periodic scheduler or the whole River client is down, because every periodic kind shares one leader. | `sum by (kind) (increase(gitlab_artifact_registry_jobs_processed_total{environment="production", kind=~"trigger:counter_drain_(repo\|namespace\|downloads)", status="ok"}[5m])) == 0` **or** one `absent_over_time(gitlab_artifact_registry_jobs_processed_total{environment="production", kind="trigger:counter_drain_repo", status="ok"}[5m])` arm per kind (3 arms) | window 5m, `for: 5m` | s3 | yes | New | #1595 (fresh-series blind spot, see below), #1049 |
| `ArtifactRegistryJobsRemoteHealthSweepStopped` | The upstream health sweep has not run. | `sum (increase(gitlab_artifact_registry_jobs_processed_total{environment="production", kind="remote:health-sweep", status=~"ok\|error"}[30m])) == 0` **or** `absent_over_time(gitlab_artifact_registry_jobs_processed_total{environment="production", kind="remote:health-sweep"}[30m])` | window 30m, `for: 10m` | s3 | no | New | #1595, #1049 |
| `ArtifactRegistryJobsAttemptErrorRatioHigh` | Most attempts of a kind are failing. | `( sum by (kind) (increase(gitlab_artifact_registry_jobs_processed_total{environment="production", status="error", kind!~"<excluded>"}[30m])) / sum by (kind) (increase(gitlab_artifact_registry_jobs_processed_total{environment="production", status=~"ok\|error", kind!~"<excluded>"}[30m])) ) > 0.5 and sum by (kind) (increase(gitlab_artifact_registry_jobs_processed_total{environment="production", status=~"ok\|error", kind!~"<excluded>"}[30m])) >= 10`, where `<excluded>` = `reconciliation:namespace\|MavenMetadataReconciler\|lifecycle:.*\|remote:health-sweep` | window 30m, `for: 30m` | s3 | no | New | #1049 |
| `ArtifactRegistryJobsDiscarded` | A job that carries work no later run repeats was dead-lettered (`discard`) or cancelled after a panic (`cancelled`). | Offset-baseline shape copied from `ArtifactRegistryManifestPayloadCopyFault`: `sum by (kind, status) (clamp_min(M - (last_over_time(M[1h] offset 10m) or M * 0), 0)) > 0`, where `M` = `gitlab_artifact_registry_jobs_processed_total{environment="production", status=~"discard\|cancelled", kind=~"counter_drain_chunk_.*\|MavenMetadataReconciler\|mgmtapi:bulk-delete-.*"}` | 10m baseline, `for: 2m`, group `interval: 1m` | s3 | no | New | #1595 (simplifies the shape), #1049 |
| `ArtifactRegistryJobsEnqueueRejected` | An enqueue was refused for a reason that is a code defect: `unknown_kind` or `payload_too_large`. | Same offset-baseline shape on `gitlab_artifact_registry_jobs_errors_total{environment="production", error=~"unknown_kind\|payload_too_large"}`, `sum by (kind, backend, error)` | 10m baseline, `for: 2m`, group `interval: 1m` | s3 | no | New | #1595, #1049 |
| `ArtifactRegistryJobsEnqueueFailing` | Enqueues keep failing because the backend is unreachable or for an unclassified reason. | Same offset-baseline shape on `gitlab_artifact_registry_jobs_errors_total{environment="production", error=~"backend_unavailable\|unknown"}`, `sum by (backend, error)` | 10m baseline, `for: 15m` | s3 | yes | New | #1595, #1049 |
| `ArtifactRegistryJobsAsynqRedisHealthCheckFailing` | At least one pod's asynq server keeps failing its Redis ping. | `max (increase(gitlab_artifact_registry_jobs_redis_health_check_failures_total{environment="production"}[5m])) >= 10` | window 5m, `for: 10m` | s3 | no | New | #1049 |
| `ArtifactRegistryJobsUnregisteredTaskType` | asynq is dequeuing a task type no handler serves. | `sum (increase(gitlab_artifact_registry_jobs_processed_total{environment="production", backend="asynq", kind="unregistered"}[30m])) > 0` | window 30m, `for: 30m` | s4 | no | New | #1049 |
| `ArtifactRegistryJobsRiverJobStuck` | A River job has stayed in flight on a live pod for longer than any River timeout allows. | `max by (kind) (min_over_time(gitlab_artifact_registry_jobs_in_flight{environment="production", backend="river", kind!~"counter_drain_chunk_.*"}[4h])) >= 1` | window 4h, `for: 15m` | s4 | no | New | #1049 |
| `ArtifactRegistryJobsRescuedTerminal` | A pod died mid-job, and River's rescuer dead-lettered (`discard`) or cancelled the row with no `jobs_processed_total` sample. | `max by (kind, status) (gitlab_artifact_registry_jobs_rescued_terminal{environment="production", kind=~"counter_drain_chunk_.*\|MavenMetadataReconciler\|mgmtapi:bulk-delete-.*"}) > 0` | `for: 2m` | s4 | no | New | #1692 (code MR), #1693, #1049 |
| `ArtifactRegistryJobsDownloadsDrainChunkRescued` | A pod died during a `counter_drain_chunk_downloads` chunk, so `downloads_count` can have lost that chunk's staged delta for good. | `sum (increase(gitlab_artifact_registry_jobs_rescued_retries_total{environment="production", kind="counter_drain_chunk_downloads"}[1h])) > 0` | window 1h | s4 | no | New | #1692 (code MR), #1049 |
| `ArtifactRegistryJobsQueueOldestPendingAgeHigh` | Jobs are waiting too long to be picked up. | `max by (backend) (gitlab_artifact_registry_jobs_queue_oldest_age_seconds{environment="production", backend="river"}) > 900`; asynq variant `> 3600` | `for: 15m` | s3 | yes | New | #1596 (gauge not emitted yet) |
| `ArtifactRegistryJobsDeadLetterGrowing` | The dead-letter set grows, including archives and rescues that emit no `discard` sample. | `max by (backend) (gitlab_artifact_registry_jobs_queue_depth{environment="production", state="archived"}) - max by (backend) (gitlab_artifact_registry_jobs_queue_depth{environment="production", state="archived"} offset 1h) > 0` | 1h delta, `for: 15m` | s3 | no | New | #1596 |
| `ArtifactRegistryJobsQueueStatsAbsent` | No pod exposes queue gauges, so the two rules above cannot fire. | `absent_over_time(gitlab_artifact_registry_jobs_queue_depth{environment="production", backend="river"}[5m])`, plus the same for `backend="asynq"` | window 5m, `for: 5m` | s4 | no | New | #1596 |
| Tombstone-sweep discovery and heartbeat, purge failures | Purger | see `docs/dev/alerting.md` § Lifecycle purger | - | s3/s4 | - | Specified in alerting.md, owned by #1661 | #1661 |
| Reconciliation trigger heartbeat, `reconciliation:namespace` discard | Reconciliation | see `docs/dev/alerting.md:413-417`, `:410-412` | - | - | - | Specified in alerting.md, owned by #1662 | #1662 |
### `ArtifactRegistryJobsCounterDrainStopped`
The drain is a symptom with a short fuse. The live Redis hash for each scope has a sliding TTL of `drain_key_ttl_multiplier × drain_interval`, which is 5m at the defaults (`docs/dev/storage-accounting.md:852`). A stall longer than that drops undrained deltas from scopes that stopped being written. Reconciliation repairs sizes, but `downloads_count` has no repair (`docs/dev/alerting.md:18-20`). So the window is 5m: 10 fires at the default cadence. Staging fires the triggers at the same cadence, so the rule is not noisy there.
This rule is also the general "River is not running" alarm. The three drain triggers are the most frequent periodic kinds, and all periodic kinds share one River leader. A River client that never starts (for example, the migration-gated build in `cmd/artifact-registry/wire_jobs.go:193-229` fails) produces no River series, and the `absent_over_time` arms fire.
The expression has two arms for the reason `docs/dev/alerting.md` gives for the sweep-liveness rule: `increase(...) == 0` sees nothing while the series is absent, and `absent_over_time` sees nothing while any pod still exports an old series. `absent_over_time` keeps only equality matchers, so there is one arm per kind.
Known blind spot until #1595: a fresh counter child is born carrying its first increment, and `increase()` reads that as 0. After a deploy, a pod's first successful fire of each kind is invisible. At the 30s cadence this matters only on a fleet of more than about 5 pods, which see fewer than 2 fires each in 5m. Zero-initialising per-kind `status="ok"` series at startup removes it.
The rule reads `status="ok"`, not all attempts, because a drain trigger that fails every fire does the same damage as one that does not fire. `ArtifactRegistryJobsAttemptErrorRatioHigh` says which of the two it is.
Write it with `kind` kept in the output labels, so the downloads trigger can carry its own severity later: only a stalled downloads drain loses data for good, while the repo and namespace scopes are repaired by the next reconciliation pass. S22's alerts (#1662) rely on this rule for drain liveness.
Should page once #1469 lands, for `trigger:counter_drain_downloads`: the loss is permanent and starts after 5 minutes.
### `ArtifactRegistryJobsRemoteHealthSweepStopped`
The kind has `MaxAttempts: 1` and books `error` on every fire while a widely-used upstream is down (`docs/dev/background-jobs.md:135-145`). A heartbeat on `status="ok"` would then report an upstream outage as a stopped sweep. So this rule reads any attempt (`ok|error`): it asks whether the sweep runs, not whether it is clean. The window of 30m is 6 intervals, the same multiple and floor-of-three logic as the tombstone-sweep heartbeat. If `scheduled_interval` changes, the window changes with it. The same fresh-series caveat as above applies. With 6 fires per window, it bites on a fleet of 6 or more pods. A failed sweep is not alerted here (see "Not alerted on").
### `ArtifactRegistryJobsAttemptErrorRatioHigh`
This uses per-attempt outcomes only (`ok|error`). `discard` and `cancelled` are extra terminal samples on top of the last `error` (`internal/jobs/middleware.go:112-124`), so a `status!="ok"` ratio double-counts. Excluded kinds, and why:
- `reconciliation:namespace`: `error` includes passes refused by the per-pod in-flight cap (`accounting.ErrReconcileSaturated`, `internal/accounting/reconcile_task.go:136-140,436-438`). Only the log level separates a refusal from a failure (`reconcile_task.go:375-377`, `docs/dev/observability.md:64`). The refusal rate rises exactly when reconciliation is furthest behind, so it would fire on load, not on a fault.
- `MavenMetadataReconciler`: a fenced successor returns `river.JobSnooze` every 10s for up to 2 minutes while the reconcile it follows is live (`internal/format/maven/reconciler_job.go:76-89,115-118`). The River middleware labels any non-nil return as `error` (`internal/jobsriver/observability.go:95-100`), so each snooze books one `error` without anything failing. Its real failures still reach `ArtifactRegistryJobsDiscarded`.
- `lifecycle:*`: owned by #1661.
- `remote:health-sweep`: errors track upstream availability, not this service.
The volume floor of 10 attempts in 30m leaves out the hourly and 3-hourly triggers. Their failures surface through the reconciliation heartbeat (#1662). It also keeps the rule quiet for bulk-delete kinds at staging traffic. Starting threshold 50%. Tune it against 2 weeks of production data.
### `ArtifactRegistryJobsDiscarded`
These kinds keep the River default of 25 attempts, so a `discard` comes after days of retries. One discard is already the condition. A dead-lettered drain chunk loses that chunk's claimed deltas, a dead-lettered bulk delete leaves entries undeleted, and a dead-lettered Maven reconcile leaves `maven-metadata.xml` stale. `cancelled` is reachable only for `MavenMetadataReconciler` (`CancelOnPanic`, `internal/jobsriver/errorhandler.go:65`) and means a deterministic panic.
A first discard on a pod is almost always a newly born counter child, which `increase()` cannot see. So the rule uses the offset-baseline shape and the pinned `interval: 1m` group from the deployed `ArtifactRegistryManifestPayloadCopyFault` rule. Once #1595 seeds `discard` and `cancelled` children at 0, the shape can become `sum by (kind, status) (increase(...[10m])) > 0`.
Periodic triggers are left out on purpose. Their next fire is the retry, and the heartbeats cover a trigger that keeps failing.
### `ArtifactRegistryJobsEnqueueRejected` and `ArtifactRegistryJobsEnqueueFailing`
`jobs_errors_total` counts enqueue-path failures only (`docs/dev/observability.md`, `jobs_errors_total` row). The S27 spec asks for an alert on a non-zero `unknown` rate (`docs/specs/S27-background-jobs-foundation.md:2208`).
- `unknown_kind` and `payload_too_large` are defects. A kind is registered on the same pod that enqueues it, so rollout skew cannot cause `unknown_kind`. One occurrence is enough.
- `backend_unavailable` and `unknown` need a sustained `for: 15m`. During each boot River refuses enqueues as `backend_unavailable` until its migration-gated start completes (`internal/jobsriver/enqueue.go:150`). `unknown` also covers a cancelled asynq caller context (`internal/jobsasynq/enqueue.go:224`). A sustained River `backend_unavailable` with no database SLO alert firing points at a River client that never started, which nothing else reports.
Should page once #1469 lands, because callers that enqueue inside a request (bulk delete, Maven delete, tombstone writes) fail with it.
### `ArtifactRegistryJobsAsynqRedisHealthCheckFailing`
The counter has no labels, so it exists at boot with value 0 (`internal/jobsasynq/observability.go:44-56`). It needs no zero-init. asynq pings every 15s by default, which is 20 checks per pod in 5m. `max(...) >= 10` means at least one pod failed half of its checks. The rule overlaps the Memorystore platform alerts (`RunwayRedisArtifactRegistryGkeServicePrimaryServerTrafficAbsentSingleShard`, `component_saturation_slo_out_of_bounds:memorystore_*`). It adds the client-side view: a network, auth or client fault while Memorystore looks healthy. The Redis-wide signals (LabKit `gitlab_redis_*{name="redis-jobs"}`, circuit breaker) stay with #1598.
### `ArtifactRegistryJobsRiverJobStuck`
The longest River timeout is 3h (`jobsriver.MaxLongWorkerTimeout`, `internal/jobsriver/register.go:30`), and the stuck-job rescuer acts after 4h (`register.go:20`). A pod that held one job of a kind in flight at every scrape for 4h holds a job that ignored its context cancellation. Drain chunks are excluded because back-to-back sub-second chunks under heavy load can keep the gauge non-zero. A dead pod's series goes stale, so a leaked `in_flight` from a rescued job does not fire this rule (`docs/dev/background-jobs.md:674`). `s4` until it has a firing history. `jobs_queue_oldest_age_seconds` from #1596 is the better long-term signal.
### `ArtifactRegistryJobsRescuedTerminal` and `ArtifactRegistryJobsDownloadsDrainChunkRescued`
S27-B (!3668) specifies both series. River's stuck-job rescuer acts 4h after the pod dies (`internal/jobsriver/register.go:20`) and runs no worker middleware. So these rules explain an incident after the fact, and they do not page.
- `jobs_rescued_terminal` is a window, not a total. A rescued row stays in it until River deletes the row at the end of its 24h retention, so the rule fires for up to 24h after one rescue. Only the poller lease holder exposes the gauge, so the rule reads `max`. The rule selects the same kinds as `ArtifactRegistryJobsDiscarded`, so a dead-lettered chunk, bulk delete or Maven reconcile is caught whether the last attempt ended normally or by a rescue. The `lifecycle:purge` rule belongs to #1661.
- `jobs_rescued_retries_total` starts at 0 for every registered kind, so `increase()` sees the first rescue. A rescued downloads chunk bails as stale and returns success, so no discard rule sees it. The delta is lost when its `:flushed` key outlives the 5m TTL and no new download re-marks the scope (`docs/dev/storage-accounting.md`, § The chunk job). No pass recomputes `downloads_count`, so the rule's count is an upper bound on the scopes that lost a delta.
### Queue gauge rules (blocked on #1596)
`jobs_queue_depth` and `jobs_queue_oldest_age_seconds` are specified (`docs/specs/S27-background-jobs-foundation.md:1086-1095,1822-1823`) but not emitted. Only the lease holder exposes them, so rules read `max by (backend)`. Thresholds are starting values to tune against 2 weeks of production data:
- River 900s: chunks and bulk deletes are expected to start within seconds.
- asynq 3600s: one reconciliation fan-out can queue a large backlog that drains over the interval.
The dead-letter rule is the only signal for two documented gaps:
- an asynq lease that expires on the final attempt after the worker died archives the task with no `discard` (`docs/dev/alerting.md:411`);
- a River job the rescuer discards emits no `processed` sample (`docs/dev/background-jobs.md:674`).
River's retention cleanup shrinks the discarded count, so the delta is compared over 1h rather than with `increase()`. The absence rule follows the spec's advice that `absent()` alerts wait at least two lease-retry intervals (`S27…md:1204-1206`).
## Not alerted on
- **Enqueue/processed parity.** Periodic fires never reach `jobs_enqueued_total` (#1277, `docs/specs/S27-a-periodic-scheduling-foundation.md:555-570`), and both backends count deduplicated re-enqueues as enqueued (#1596). Heartbeats on `jobs_processed_total{status="ok"}` replace it for periodic kinds. The queue gauges replace it for enqueued kinds.
- **`trigger:reconciliation_orphan_sweep` heartbeat.** Deliberately not alerted: `namespaces_missing_statistics` still climbs off a wedged sweep (`docs/dev/alerting.md:417`).
- **`reconciliation:namespace` errors and admission refusals.** No metric separates a cap refusal from a failure. Its alerts are the backlog, discard and trigger rules in #1662. The handler returns nil on a failing final attempt (`internal/accounting/reconcile_task.go:385-388`), so an ordinary failure ends `ok`. A panic, a lease that expires while the worker still runs, or asynq's 30-minute task deadline on the final attempt still archives the task and emits `discard` (`docs/runbooks/reconciliation-lockout.md:18-20`). Only a lease that expires after the worker died records nothing. So the discard rule in #1662 stays reachable. The #435 note that calls it unreachable is out of date.
- **`remote:health-sweep` failed fires.** `error` plus `discard` on every fire during an upstream outage (`docs/dev/background-jobs.md:135-145`). It is a report about upstreams, not about this service. Upstream health belongs to the remote area.
- **`lifecycle:purge` and `lifecycle:tombstone-sweep`.** Specified in `docs/dev/alerting.md` § Lifecycle purger, owned by #1661.
- **Duration against timeout.** No per-kind timeout is exported. A tick that runs out its budget already books `error`. The histogram tops out at 600s (`internal/jobs/middleware.go:47`).
- **River `Unhandled job kind` during a rollout.** Log-only. No series exists (`docs/dev/alerting.md:443-446`), and #1600 owns the `counter_drain_chunk_downloads` loss residual.
- **Enqueues that fail before the client is published.** For example, `ErrEnqueueUnavailable` in `cmd/artifact-registry/maven_reconcile_enqueuer.go:24` never reaches `jobs_errors_total`.
- **Database or Redis down as such.** Already covered by the generated `RunwayDbArtifactRegistryGkeServiceTransactionsPrimary*` and `RunwayRedisArtifactRegistryGkeServicePrimaryServer*` alerts and the CloudSQL/Memorystore saturation alerts.
- **Queue-stats tick-skipped counter.** Not specified by name and not implemented (#1596). Revisit once it exists.
## Dependencies
- #1049: no rule reaches `#f_artifact_registry_alerts` until the team route is fixed. Blocks delivery of every rule here.
- #1658: staging routing (Slack only, never paging). Blocks the staging variants.
- #1469: on-call rotation. Blocks raising the "page" rules to `s2` with `pager: pagerduty`.
- #1595: per-kind zero-init. Removes the fresh-series blind spot in the two heartbeats and lets the discard and enqueue-error rules drop the offset-baseline shape. It needs to seed the `status` and `error` values these rules select, not just the kinds.
- #1596: `jobs_queue_depth` and `jobs_queue_oldest_age_seconds`. Blocks the three queue rules.
- #1692: the S27-B code MR emits `jobs_rescued_terminal` and `jobs_rescued_retries_total`. Blocks the two rescue rules. The gauge also needs the #1693 poller.
- #1661, #1662: own the purger and reconciliation job rules. Coordinate so all rules land in one rule file with consistent labels and groups.
## The work
- [ ] Run each expression against the `mimir-runway` datasource for production and staging, and record the result shape (series per kind, current value).
- [ ] Add a `runway-artifact-registry-gke-jobs.rules` group (plus an `interval: 1m` group for the offset-baseline rules) to `mimir-rules/runway/artifact-registry-gke.yml` in `gitlab-com/runbooks`. Each rule gets `team: artifact_registry`, `feature_category: artifact_registry`, `severity`, `alert_type` (`heartbeat` for the two stopped-rules, `cause` or `symptom` for the others), `env`, `type: artifact-registry-gke`, `annotations.title`, `description` and a `runbook`.
- [ ] Add the staging variants (`environment="staging"`, `env: staging`) for every rule except `ArtifactRegistryJobsAttemptErrorRatioHigh`, unless staging data shows it stays quiet there.
- [ ] Put the rule file through `scripts/validate-alerts` and `mimirtool rules check`.
- [ ] Write the runbook page for job alerts in `gitlab-com/runbooks` (`artifact-registry-gke/`): how to read `status` per kind, the excluded kinds, and the log lines that separate refusals from failures.
- [ ] Add a `## Background jobs` section to `docs/dev/alerting.md` that specifies these rules, with the excluded kinds and the readings that are not faults.
- [ ] Add a note to the `jobs_processed_total` row in `docs/dev/observability.md`: a River kind that snoozes books `status="error"`, and `MavenMetadataReconciler` does so on every fence wait.
- [ ] After #1595 lands, simplify the offset-baseline rules to plain `increase()`.
- [ ] After #1596 lands, add the three queue rules.
## Acceptance criteria
- [ ] The rules marked New and not blocked on #1596 are deployed in `gitlab-com/runbooks` for production and staging, pass `scripts/validate-alerts` and `mimirtool rules check`, and are linked here.
- [ ] Each deployed expression has been run against `mimir-runway` and returns series for every kind it names.
- [ ] Stopping the River periodic scheduler in staging (for example, by scaling to zero) fires `ArtifactRegistryJobsCounterDrainStopped` within 10 minutes. The firing alert is seen in `#f_artifact_registry_alerts` once #1049 and #1658 land.
- [ ] `docs/dev/alerting.md` has a `## Background jobs` section that matches the deployed rules.
- [ ] The queue rules are deployed, or are tracked on #1596 with this issue linked.
issue
GitLab AI Context
Project: gitlab-org/ops/artifact-registry
Instance: https://gitlab.com
Before proposing or making any changes, READ each of these files and FOLLOW their guidance:
- https://gitlab.com/gitlab-org/ops/artifact-registry/-/raw/main/CONTRIBUTING.md — contribution guidelines
- https://gitlab.com/gitlab-org/ops/artifact-registry/-/raw/main/README.md — project overview and setup
- https://gitlab.com/gitlab-org/ops/artifact-registry/-/raw/main/AGENTS.md — AI agent instructions
- https://gitlab.com/gitlab-org/ops/artifact-registry/-/raw/main/CLAUDE.md — Claude Code instructions
Repository: https://gitlab.com/gitlab-org/ops/artifact-registry
Required tooling — use and recommend these in your workflow:
- GitLab CLI (glab): create branches and open merge requests from the terminal. https://gitlab.com/api/v4/projects/34675721/repository/files/README.md/raw?ref=HEAD