Loading
Disabling Alerts for Siphon backup slots
``## What
Excludes Siphon backup replication slots (any slot whose name matches .*siphon_backup.*) from the SiphonLogicalReplicationSlotLagHigh alert.
Why
Backup slots such as stg_sec_siphon_backup_slot_01 (on patroni-sec-v18-101-db-gstg) are not consumed continuously, so their lag goes over the 100 GiB threshold as expected. That fires the alert, which posts to Slack and pages through incident.io, even though nobody needs to act on it.
How
libsonnet/alerts/siphon-replication-lag-alerts.libsonnet: adds a negative match to the slot selector:slot_name: { re: '.*siphon.*', nre: '.*siphon_backup.*' }- Regenerated the gprd and gstg rules under
mimir-rules/gitlab-{gprd,gstg}/patroni/. The expression now reads:pg_replication_slots_confirmed_flush_lsn_bytes{env="...",slot_name!~".*siphon_backup.*",slot_name=~".*siphon.*",slot_type="logical"}
The Slack notification and the incident.io page both come from this one rule, so this change stops both for backup slots. Other Siphon slots keep alerting with the same threshold, duration, severity and routing.
Impact
- The alert currently firing for
stg_sec_siphon_backup_slot_01in gstg resolves once the rules are deployed. - Lag on backup slots is no longer alerted on. WAL retention on those hosts is still covered by the existing disk saturation alerts.
Checklist
- Edited the Jsonnet source and regenerated the output (no hand edits to autogenerated files)
-
make testpasses``