feat(postgresql): add replication slot xmin age dashboard
What / Why
Adds a Grafana dashboard for tracking PostgreSQL replication slot xmin / catalog_xmin age alongside the autovacuum and wraparound signals that make those numbers actionable.
Motivated by INC-13566 and the corrective-action work item gitlab-com/gl-infra/production-engineering#29650 (closed). During that incident, replication slots on patroni-sec pinned catalog_xmin on template0, autovacuum workers were consumed by wraparound-prevention vacuums that could not advance, and auto-analyze effectively stopped — leaving the planner with stale statistics and picking a bad plan on security_findings.
The existing panels for slot catalog_xmin age (Grafana link) show only one side of that story. This dashboard puts the full signal chain in one place.
Signal flow
slot pins xmin/catalog_xmin
-> cluster xmin age climbs
-> datfrozenxid age approaches wraparound
-> autovacuum workers saturate on wraparound-prevention vacuums
-> auto-analyze rate drops to zero
-> planner stats go stalePanels
- Replication slot
catalog_xminage (per slot) —pg_replication_slots_catalog_xmin_age - Replication slot
xminage (per slot) —pg_replication_slots_xmin_age - Cluster
xminage (per node) —pg_txid_xmin_age - Datfrozenxid age (per node, per database) —
pg_database_wraparound_age_datfrozenxid; approaches 2^31 (~2.15B) as wraparound nears - Autovacuum workers running vs max (per node) —
count(pg_stat_activity_autovacuum_age_in_seconds)vspg_settings_autovacuum_max_workers - Longest-running autovacuum age (per node) —
max(pg_stat_activity_autovacuum_age_in_seconds) - Analyze rate (per node) —
rate(pg_stat_user_tables_analyze_count[5m])
Template variables
$type(cluster):patroni,patroni-ci,patroni-registry,patroni-sec— defaults topatroni-sec.$fqdn(node): populated fromlabel_values(pg_replication_is_replica{environment=$environment, type=$type}, fqdn), multi-select, defaults to All.
Preview
Uploaded to the Playground folder for review: https://dashboards.gitlab.net/d/pkanellidis-verify-pg-slot-xmin
All panels return data on gprd / patroni-sec at the time of writing (autovacuum-worker and longest-autovacuum panels are empty when no autovacuum is active — expected).
Validation
jsonnetfmt --test— clean./dashboards/generate-dashboard.sh postgresql/slot-replication-xmin-age.dashboard.jsonnet— compiles
Related to gitlab-com/gl-infra/production-engineering#29650 (closed)