2025-12-05: Prometheus unable to scrape exporter on patroni-sec-v17-03-db-gprd.c.gitlab-production.internal
Prometheus unable to scrape exporter on patroni-sec-v17-03-db-gprd.c.gitlab-production.internal (Severity 3)
Problem: Prometheus failed to collect metrics from at least one exporter on patroni-sec-v17-03-db-gprd.c.gitlab-production.internal, causing gaps in monitoring data.
Impact: Prometheus was unable to scrape metrics from at least one exporter on a Patroni database server, reducing monitoring visibility for those databases. There is no evidence of direct customer impact. A long-running transaction alert on another Patroni node was also detected and merged into this incident, but no customer-facing issues were reported.
Causes: A recent change to the gitlab-monitor cookbook was deployed, which was then reverted as a precaution after Prometheus scraping failures were detected.
Response strategy: We investigated VM health, Chef runs, and metrics dashboards. We reverted the recent gitlab-monitor cookbook change, which has led to recovery of scraping. Both attached alerts have now been resolved.
This ticket was created to track INC-6133, by incident.io