Went through the original 18-point sweep one by one. Most are fixed, disproved, or not ours to fix. Sorted below so nobody re-raises what is already closed. Production figures below were measured against `mimir-gitlab-gprd` on 2026-08-24 unless stated otherwise. ## TODO 1. [x] Rails-to-OpenBao hop had zero measurement from either end, and every secret read crosses it. Now instrumented — https://gitlab.com/gitlab-org/gitlab/-/merge_requests/251497 adds `gitlab_openbao_requests_total` and `gitlab_openbao_request_duration_seconds`. 1. [x] Rails to Openba calls SLI, `openbao_client_calls`, error rate only, attached to the existing `api` service. https://gitlab.com/gitlab-org/gitlab/-/merge_requests/253209 is the Rails half, adding `gitlab_sli_openbao_client_calls_total` and `gitlab_sli_openbao_client_calls_error_total`. https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11333 registers the SLI and attaches it. 1. [x] The retired Cloud Run service was still generating 19 live rules, 3 of them pageable. Deleted — https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11321. It now generates zero. The GCP-side cleanup (Cloud Run revision and load balancer) is the only remainder and lives on https://gitlab.com/gitlab-org/gitlab/-/work_items/606951, not here. 1. [x] HPA saturation alert was pinned at `1.00` and firing in both production and staging, plus both anomaly-detection rules were dead. Removed — https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11316. 1. [x] Every alert pointed `runbook:` at `<service>/#alerts`, an anchor that existed on no runbook page. Fixed — https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11317. The original "all 24 alerts across three services" framing no longer describes reality, because the third service was deleted in the teardown above. 1. [x] Secrets Manager GraphQL requests were misattributed to other `feature_category` values. Fixed — https://gitlab.com/gitlab-org/gitlab/-/merge_requests/252273. 1. [x] Worker attributes, log context, and OpenBao metric outcomes cleaned up — https://gitlab.com/gitlab-org/gitlab/-/merge_requests/252512. 1. [x] Sentry-side error tracking for the Rails code — https://gitlab.com/gitlab-org/gitlab/-/work_items/624868. 1. [ ] Carefully evaluate the severity of each alert and decide the list of pageable alerts and update them to s2 so that paging will be done 1. [ ] Nothing reads OpenBao's own seal health, so a revoked Cloud KMS permission stays invisible until the next restart can't unseal. Biggest remaining gap. `secrets_manager_gke_seal_unreachable_time` is live in Mimir, so this is ready to write. 1. [ ] Both error SLIs on `secrets-manager-gke` count only `500`, so the error rate reads a flat zero however many `400`s we serve. We shouldn't widen it: excluding client errors is the catalog-wide convention, and `400`s are now about a third of traffic, so counting them would sit us hundreds of times over the error budget and the alert would never clear. Needs a runbook note saying why the number is zero. 1. [ ] `runway-db-secrets-manager-gke` has no error-rate signal, and no Cloud SQL metric can give it one, so the SLI stays request-rate-only and needs a runbook note. One real gap is left: nothing alerts on the instance being down or suspended, which wants a cause alert. 1. [ ] Add `secrets-manager-gke` to the `open_fds` saturation exclusion list, which deletes a rule that can never fire. OpenBao emits none of the metrics that rule reads. 1. [ ] GKE application scrapes are missing the `type` label. Runway's gap rather than ours, and it also hits `example-service-gke` and `iam-data-gke-grpc`. Needs raising with them. 1. [ ] No runtime read on product usage. Enabled projects, enabled groups, enrolled namespaces and the billing counters are all Service Ping metric definitions, so none is scraped into Prometheus and during an incident "how many namespaces have this enabled right now" has no answer. The gap is the cadence, not missing counters. 1. [ ] The `openbao_client_calls` alerts are gated shut by the minimum-traffic threshold and can't fire. I measured a week of traffic: no threshold gives full coverage at today's call volume, so whatever we pick buys partial coverage and needs an SRE call. The confidence-interval option floated earlier doesn't help, it leaves the traffic gate untouched. Correcting the earlier note, the two GKE SLIs are better off than it said — their 6h alerts are armed and only the 1h ones are shut. 1. [ ] Nothing measures the GitLab Rails to CDot (CustomersDot) hop, and the Secrets Manager entitlement check crosses it on every availability decision. Needs one Rails MR for the counters and one runbooks MR for the SLI, covering only our own entitlement calls rather than the whole subscription portal client. Still open: what counts as an error, since the last-known-good fallback means the user never sees a CDot outage ## Not a defect, or not ours - `TrafficCessation` not covering the gauge SLI is deliberate. `TrafficCessation` is for counter-backed SLIs, `TrafficAbsent` is the designed cover for gauge SLIs and it is already generated and routed for us. Fixing this at libsonnet level would touch 4,070 rules and disable `TrafficAbsent` repo-wide. - `kube_container_cpu_limit` records no series, so the alert is already silent — nothing to suppress. Sibling `kube_container_throttling` does record fine. - The apdex numerator matching `le` as an exact string is already correct — quoted deliberately to defeat `%g` truncation, and every bucket boundary on this ladder is a 16 or 17 digit float. Worth a comment on https://gitlab.com/gitlab-com/runbooks/-/work_items/225 rather than an MR. Also correcting the record: a broken match makes the apdex absent, not `1.0`. - `KubeContainersWaitingInError` is platform-wide, not ours. The `maxSurge` join is one hardcoded template used by every kube-provisioned service, no prior art for an exclusion, and switching to `RollingUpdate` is a hard no — `Recreate` is what stops an old pod winning the Postgres-backed HA leader election. Raise with Observability. - "No feature-level signal during an incident" was wrong. `gitlab_sli_rails_request_total` for the audit-log `POST` already exists with `endpoint_id` and `feature_category` both preserved, recording `0.175221` rps in production — a Grafana panel answers this today, no code needed. Side note: that `feature_category` is 96.88% CI Secure Files (`5.4424` rps) against `0.175221` rps for all Secrets Manager endpoints combined. - The apdex minimum-traffic gate reading an inflated rate is real, roughly `6.6x`, but it self-heals as traffic grows and is not ours. https://gitlab.com/gitlab-com/runbooks/-/work_items/153 is a related 2024 proposal on low-RPS SLI alerting, not the platform-wide version of this. - `TrafficAbsent` firing on metric sparsity resolves as traffic grows. - The `runway-db-secrets-manager-gke` sweep was analysis, not a fix — its anomaly rules already went with https://gitlab.com/gitlab-com/runbooks/-/merge_requests/11316. `transactions_primary` records normally. ## Tracked elsewhere - please do not duplicate here - https://gitlab.com/gitlab-org/gitlab/-/work_items/606951 — removing the retired Cloud Run service from GCP. 9-item teardown checklist, still needs an assignee. - https://gitlab.com/gitlab-com/runbooks/-/work_items/235 — alert routing and the feature-category mapping. No decision recorded, no reply since 2026-08-17. - https://gitlab.com/gitlab-org/gitlab/-/work_items/606125 — Rails-side audit-endpoint latency. Closed 2026-08-14, but two things went unowned when it closed: the apdex minimum-sample guard, and lowering the OpenBao HTTP audit timeout from 5s to about 1s. - https://gitlab.com/gitlab-org/gitlab/-/work_items/602503 — metrics and alerting for the future log-shipper sidecar. - https://gitlab.com/gitlab-org/gitlab/-/work_items/624868 — Sentry-side runtime error tracking for the Rails code.