# [INC-2309: Git service errors caused by paused pgBouncer in Patroni sec cluster](https://app.incident.io/gitlab/incidents/2309) Generated by Pierre Guinoiseau on 10 Jul 2025 09:09. All timestamps are local to Etc/UTC # Key Information <!--- Customers Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "All users with repositories on file-01 node" Requests Affected: If numbers are available, use them. Link the number to a metric if possible. If numbers are not easy to generate, a text description is fine - "Any requests from Australia during the CloudFlare outage" Incident Severity: Copy from the incident. Use the severity at the end of the incident, not the beginning. Use the label rather than text - "~Severity::*" Start Time: Use the time when service started to be effected. Not the time the incident was opened (if they are different) End Time: Use the time when the incident was mitigated. Don't include the time in a monitoring state unless the monitoring indicated continuing problems. Total Duration: Use the [Platform Metrics Dashboard](https://dashboards.gitlab.net/d/general-triage/general-platform-triage?orgId=1) to look at appdex and SLO violations. Link to Incident Issue: Link to Incident Issue ---> | Metric | Value | | ------ | ------ | | Customers Affected | 17,383 | | Requests Affected | 445,621 HTTP Git requests, 59,777 Sidekiq jobs | | Incident Severity | ~"severity::2" | | Impact Start Time | Thu, 10 Jul 2025 06:00:00 UTC| | Impact End Time | Thu, 10 Jul 2025 06:15:00 UTC | | Total Duration | 24 minutes | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20150 | # Summary **Problem:** The Git and Sidekiq services experienced errors due to an issue within the load balancer, leading to an SLO violation. **Impact:** Git requests and Sidekiq jobs were failing for about 15 minutes **Causes:** The errors were caused by the PgBouncer nodes of the `sec` Patroni cluster being stuck in a paused state during a Patroni leader switchover executed during the [Consul CA rotation](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20084). The issue was triggered by a Patroni switchover playbook timing out while waiting for the Yubikey. **Response strategy:** The Patroni switchover playbook was re-executed successfully, unpausing PgBouncer in the process. # What went well? * The Patroni failover playbook was able to run without any issue on the second run, leading to a quick resolution. # What was difficult? * It took a long time for the SRE executing the Patroni switchover to realise that the playbook was stuck because of the lack of error in the Ansible output * This incident triggered and surfaced another issue that took longer to resolve and had a higher impact: https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20165 # Investigation Details <details> <summary> Timeline</summary> # Incident Timeline **2025-07-10** **06:35:38** **Incident reported by Tarun Khandelwal** Tarun Khandelwal reported the incident Severity: Severity 3 Status: Investigating **06:47:59** **Image posted by Pierre Guinoiseau** Pierre Guinoiseau posted an image to the channel There were errors from the `git` service for about 15 minutes  **06:50:10** **Message from Pierre Guinoiseau** Pierre Guinoiseau pinned their own message Those errors caused by the PgBouncer nodes of the `sec` Patroni cluster being stuck in a paused state during a leader switchover executed during the Consul CA rotation https://gitlab.com/gitlab-com/gl-infra/production/-/issues/20084 **06:51:20** **Message from Pierre Guinoiseau** Pierre Guinoiseau pinned their own message Because the playbook timed out during that step while waiting for the yubikey **06:59:22** **Incident resolved and entered the post-incident flow** Pierre Guinoiseau shared an update Severity: ~~Severity 3~~ → Severity 2 Status: ~~Investigating~~ → Documenting We experienced errors from the git service for about 15 minutes. This was caused by PgBouncer nodes in the 'sec' Patroni cluster being stuck in a paused state during a leader switchover, which occurred as part of the Consul CA certificate rotation. The switchover process timed out due to a delay while waiting for the yubikey during the playbook execution. The incident was resolved after retrying the leader switchover which unpaused the PgBouncer nodes. # Investigation Notes </details> Any details you may want to add about the investigation can go here. # Follow-ups * ~[Document safe Patroni failovers with the Ansible playbook](https://app.incident.io/gitlab/incidents/2309?tab=follow-ups)~ replaced by... * [Fix Switchover Playbook - issues found during Consul update](https://gitlab.com/gitlab-com/gl-infra/data-access/dbo/dbo-issue-tracker/-/issues/514+) - Database Operations team # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [x] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [x] Close the review before the due date - [x] Go back to the incident channel or page and close out the remaining post-incident tasks