feat(postgresql): add replication slot xmin age dashboard

What / Why

Adds a Grafana dashboard for tracking PostgreSQL replication slot xmin / catalog_xmin age alongside the autovacuum and wraparound signals that make those numbers actionable.

Motivated by INC-13566 and the corrective-action work item gitlab-com/gl-infra/production-engineering#29650 (closed). During that incident, replication slots on patroni-sec pinned catalog_xmin on template0, autovacuum workers were consumed by wraparound-prevention vacuums that could not advance, and auto-analyze effectively stopped — leaving the planner with stale statistics and picking a bad plan on security_findings.

The existing panels for slot catalog_xmin age (Grafana link) show only one side of that story. This dashboard puts the full signal chain in one place.

Signal flow

slot pins xmin/catalog_xmin
  -> cluster xmin age climbs
  -> datfrozenxid age approaches wraparound
  -> autovacuum workers saturate on wraparound-prevention vacuums
  -> auto-analyze rate drops to zero
  -> planner stats go stale

Panels

  • Replication slot catalog_xmin age (per slot) — pg_replication_slots_catalog_xmin_age
  • Replication slot xmin age (per slot) — pg_replication_slots_xmin_age
  • Cluster xmin age (per node) — pg_txid_xmin_age
  • Datfrozenxid age (per node, per database) — pg_database_wraparound_age_datfrozenxid; approaches 2^31 (~2.15B) as wraparound nears
  • Autovacuum workers running vs max (per node) — count(pg_stat_activity_autovacuum_age_in_seconds) vs pg_settings_autovacuum_max_workers
  • Longest-running autovacuum age (per node) — max(pg_stat_activity_autovacuum_age_in_seconds)
  • Analyze rate (per node) — rate(pg_stat_user_tables_analyze_count[5m])

Template variables

  • $type (cluster): patroni, patroni-ci, patroni-registry, patroni-sec — defaults to patroni-sec.
  • $fqdn (node): populated from label_values(pg_replication_is_replica{environment=$environment, type=$type}, fqdn), multi-select, defaults to All.

Preview

Uploaded to the Playground folder for review: https://dashboards.gitlab.net/d/pkanellidis-verify-pg-slot-xmin

All panels return data on gprd / patroni-sec at the time of writing (autovacuum-worker and longest-autovacuum panels are empty when no autovacuum is active — expected).

Validation

  • jsonnetfmt --test — clean
  • ./dashboards/generate-dashboard.sh postgresql/slot-replication-xmin-age.dashboard.jsonnet — compiles

Related to gitlab-com/gl-infra/production-engineering#29650 (closed)

Merge request reports

Loading
Loading