_Corrective action from the INC-14518 incident review ([production#23026](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23026), Macro item 4: [note_3915201882](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23026#note_3915201882))._ ## 1. Patroni leadership / replica-detachment alerts ### Problem No alert named the failure ([note_3908190901](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23026#note_3908190901)): - The first page was `PatroniScrapeFailures` at 22:50, which was read as an exporter problem. - 01 was confirmed unreachable at 23:13, and Tier 2 DB was paged at 23:19. - 01's exporter was down, so alerts that depend on the primary's own metrics wouldn't have fired. ### Proposal Evaluate these on the **replicas'** Patroni `/metrics` (metric names from the [Patroni REST API docs](https://patroni.readthedocs.io/en/latest/rest_api.html)). Each should page DBRE with a link to the [assess-before-failover runbook](https://gitlab.com/gitlab-com/runbooks/-/blob/master/docs/patroni/primary-unresponsive-assess-before-failover.md). | Alert | Expression (sketch) | Sep 24 | |-------|---------------------|--------| | Cluster has no leader | `max by (env, type) (patroni_cluster_unlocked) == 1` for 30s | \~22:49 | | Replicas not streaming | `count by (env, type) (patroni_postgres_streaming == 1) < count by (env, type) (patroni_replica == 1)` for 1m | \~22:50 | | Pause left on | `max by (env, type) (patroni_is_paused) == 1` for 15m | guards the pause runbook | First, confirm these series are already scraped in Mimir for main, ci, sec and registry. ## 2. Delayed-replica lag alert ### Problem The delayed DR replica was ~44 hours behind, well outside its intended window, and could not serve as a pre-fork copy when it was needed. ### Proposal Add an alert when a delayed replica's lag leaves its configured band (too far behind *or* unexpectedly caught up), so the gap is caught before an incident needs that replica. ## 3. GCP-level metric heuristics as a fallback signal ### Problem Guest-side metrics (node_exporter) stopped reporting exactly when they were needed, because the exporter itself was on the frozen boot disk. GCP only exposes hypervisor/hardware-level metrics (e.g. `average_io_queue_depth`, disk `performance_status`), not Postgres-level commit activity. ### Proposal Investigate what heuristic over GCP-level disk metrics could serve as a proxy signal for "primary disk may be hanging" when the in-guest exporter is unavailable, and add the relevant metrics to dashboards/alerts. ## 4. Canonical dashboards ### Problem Responders saw conflicting error-rate figures across dashboards during the incident, and one Patroni panel had stopped updating. ### Proposal Designate the canonical 5xx panel to use during incidents, and review/fix the Patroni overview panel that stopped updating. ## 5. Kernel hung-task visibility ### Problem The ext4 journal hang on the primary's boot disk was only visible via the GCE serial console; no kernel stacks were captured automatically for later analysis. ### Proposal Add alerting on `hung_task` kernel messages, and auto-capture `/proc/*/stack` and `sysrq-w` output when they fire, so the evidence survives the serial-console buffer rolling over. ## 6. Kernel load-average artifact (P3) ### Problem Elevated load-average readings during the incident were a known kernel counter artifact and briefly misdirected investigation. ### Proposal Note this known kernel issue in the runbook, and confirm whether it's still present on the current kernel version. ## Related - Incident: [production#23018](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23018) - Incident review: [production#23026](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23026) - GCP alerting/notification design (complementary, provider-side): [production#23029](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23029) - Source discussion: [note_3914210538](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23026#note_3914210538), [note_3915201882](https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/23026#note_3915201882)