## Summary On 2026-06-11, https://gitlab.com experienced widespread and intermittent HTTP 503 errors across all major services during two short periods. The errors were caused by severe disk IO contention on the storage backing the primary database node, which was traced to a confirmed cloud provider (GCP) storage incident affecting a disk in the `us-east1-b` zone. All user-facing services recovered on their own after each burst, and the platform has been fully operational since 07:25 UTC. - Services affected: Web, frontend, API, git, websockets, web-pages, and CI runners on https://gitlab.com - Team attribution: Database Operations / Infrastructure (triggered by a GCP infrastructure-level storage IO degradation) - Minutes downtime or degradation: Two bursts of ~5–10 minutes each (06:16–06:22 UTC and 07:03–07:12 UTC) ## Impact & Metrics | Question | Answer | | ----- | ----- | | What was the impact? | Service outage / brown-out: widespread and intermittent HTTP 503 errors across all major services. Requests timed out at the 60-second Rack limit, with network calls to Redis and Postgres blocked. | | Who was impacted? | All external customers and users of https://gitlab.com | | How did this impact customers? | Users were unable to reliably access core services (frontend, web, websockets, API, git, web-pages) and CI runners during the two impact windows. | | How many attempts made to access? | All requests during the impact windows were affected | | How many customers affected? | All | | How many customers tried to access? | All traffic to https://gitlab.com during the windows | Additional metrics: - HAProxy 5xx errors spiked during 06:16–06:22 UTC (peak 06:18–06:20) and 07:03–07:12 UTC (peak 07:06–07:08). - Disk IO queue depth on the primary jumped sharply at 06:16 (~219) and 07:00–07:02 (~353 to 405), aligning with the error windows. - Spikes in LWLock contention, `pg_wait_sampling_buffer_mapping`, replication lag, and pgBouncer (sync and async) pool saturation were observed during the impact windows. - Both windows are associated with the primary database instance `patroni-main-v17-01-db-gprd` in project `gitlab-production`. - No DDoS activity was detected by Cloudflare, and no unusual request volumes preceded the outage. ## Detection & Response | Question | Answer | | ----- | ----- | | When was the incident detected? | 2026-06-11 06:16 UTC | | How was the incident detected? | Pingdom alert (reported in triage at 06:16:12 UTC) | | Did alarming work as expected? | Yes for surfacing user-facing 503s; however, there was no direct alerting on disk IO saturation / pgBouncer pool saturation / LWLock contention, which were the leading indicators. | | How long did it take from the start of the incident to its detection? | ~4 minutes (impact start 06:12 UTC → detected 06:16 UTC) | | How long did it take from detection to remediation? | Each burst self-resolved within ~5–10 minutes; no manual remediation was required. Root cause confirmed as a GCP storage incident and services remained stable from 07:25 UTC onward. | | What steps were taken to remediate? | Escalated to Database Operations experts, paused deployments, opened a GCP support case, engaged the GCP TAM, and pre-staged a Patroni primary switchover (to node `04` in `us-east1-d`) as a contingency. A structured 3-hour pause-and-review confirmed no recurrence before resolution. | | Were there any issues with the response? | A lot of time was spent watching and waiting because it was unclear whether the problem would recur. There was no pre-agreed threshold for when to fail over, and the team could not confirm ahead of time which zone was healthy to switch to given the widespread cloud provider issue. | ## Timeline 2026-06-11 (all times UTC) - 06:12 - Impact started - 06:16:12 - Incident reported in triage by Pingdom alert - 06:17:42 - Incident accepted, raised to Severity 1, and escalated to GitLab.com Production, IMOC, and Infrastructure Leadership - 06:20 - First burst fixed (self-resolved) - 06:30:06 - Status changed to Monitoring; initial assessment pointed to a temporary external networking issue - 07:03 - Second burst of widespread HTTP 503 errors begins - 07:10:35 - Status changed back to Investigating; pgBouncer pool saturation observed across sync and async pools - 07:11:35 - Escalated to tier2 - Database Operations Escalation; public status page updated - 07:12 - Second burst ends (self-resolved) - 07:36:45 - Spikes in `pg_wait_sampling_buffer_mapping` observed coinciding with connection drops - 08:00:22 - LWLock contention spike identified during the impact window - 08:13:27 - Two major disk IO queue depth spikes on the primary identified, aligning with the outages; GCP TAM contacted - 08:18:06 - Status changed to Monitoring - 08:56:29 - GCP support case opened - 10:55:41 - Outage windows (06:16–06:22 and 07:03–07:12) and affected resource (`patroni-main-v17-01-db-gprd`) verified for GCP support - 12:15:57 - Root cause confirmed: a cloud provider incident affecting a disk backing the primary database - 12:29:08 - Team prepared (but did not execute) a Patroni primary switchover to another zone as a contingency - 15:19:42 - Incident paused for a 3-hour review window after GCP concern appeared mitigated - 18:07:53 - Incident resumed for documentation; no recurrence observed ## Root Cause Analysis The purpose of this document is to understand the reasons that caused an incident, and to create mechanisms to prevent it from recurring in the future. A root cause can **never be a person**, the way of writing has to refer to the system and the context rather than the specific actors. Following the "**5 whys**" in a blameless manner: 1. **Why did https://gitlab.com return widespread HTTP 503 errors?** Requests across all major services timed out at the 60-second Rack limit because network calls to Postgres (and Redis) were blocked. 2. **Why were calls to the primary database blocked?** The primary database node experienced severe IO contention, manifesting as disk IO queue depth spikes, LWLock contention, replication lag, and pgBouncer connection pool saturation. 3. **Why did the primary database experience severe IO contention?** A disk backing the primary database node in the `us-east1-b` zone suffered storage IO degradation. 4. **Why did the disk suffer storage IO degradation?** A confirmed cloud provider (GCP) storage incident affected that disk, as corroborated by our technical telemetry and the cloud provider's notification. 5. **Why was a single degraded disk able to bring down all of GitLab.com?** The architecture currently routes all primary database traffic through a single primary node, and application DB load balancing did not gracefully shed or reroute requests when the primary became unresponsive; requests piled up to the Rack timeout instead of failing fast. Failover depends on a manual human decision with no pre-agreed threshold. (Root cause) ## What went well - **Fast detection and escalation.** Pingdom caught the first burst within ~4 minutes of impact start, and the incident was accepted and escalated to GitLab.com Production, IMOC, and Infrastructure Leadership within 90 seconds, with multiple responders acknowledging almost immediately. - **Effective tiered escalation to DB experts.** When the second burst pointed at the database/connection layer, the team escalated to tier2 Database Operations and a DB expert engaged quickly, enabling deep diagnosis (LWLock contention, `pg_wait_sampling_buffer_mapping` spikes, replication lag). - **Strong collaborative diagnosis with telemetry.** Responders correlated HAProxy 5xx spikes, disk IO queue depth, and pgBouncer pool saturation to converge on disk IO contention, and correctly ruled out DDoS and slow queries early using Cloudflare data. - **Prepared a concrete contingency without rushing it.** The team pre-staged a Patroni switchover to node `04` in `us-east1-d`, weighing the write-throughput impact, so it could be executed instantly if degradation recurred. - **Disciplined monitoring and resolution.** Deployments were paused, a GCP support case was opened, and a structured 3-hour pause-and-review was used before resolving rather than closing prematurely. - **Good external coordination.** Engaging the GCP TAM and opening a support case got the cloud provider confirming the storage incident, validating the root-cause hypothesis. ## What can be improved - **Reduce blast radius.** A single degraded disk was sufficient to bring down GitLab.com. While Cells are designed to limit the blast radius long-term, a short-term mitigation is needed to ensure service continuity. - **Fail fast.** Application DB load balancing did not gracefully shed or reroute when the primary became unresponsive; requests piled up to the 60s Rack timeout instead of failing fast (echoes prior incident learnings in [#18269](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/18269)). - **Reduce detection time.** Add alerting directly on disk IO queue depth / IO saturation for primary Patroni nodes, and consider alerting on pgBouncer pool saturation and LWLock contention as leading indicators, since both spiked ahead of full impact. - **Reduce resolution time.** Document a clear runbook with decision criteria for executing a Patroni primary switchover during disk IO degradation (which metrics/thresholds justify it, expected impact, exact command), and evaluate automated failover on primary IO degradation. - **Clarify incident state transitions.** For intermittent, self-resolving issues it was hard to decide when to move the incident to "Fixing" or "Monitoring"; clearer guidance is needed. - **MR acceptance checklist.** Not applicable — no code change introduced this incident. ## Corrective actions - [production-engineering#29220](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29220) - Add disk IO saturation / queue-depth alerting for primary Patroni DB nodes (Owner: Database Operations). - [production-engineering#29221](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29221) - Write a Patroni primary switchover runbook with explicit trigger thresholds (Owner: DBRE). - [production-engineering#29222](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29222) - Clarify incident state-transition guidance for intermittent, self-resolving issues (Owner: IMOC/incident process). - [production-engineering#29192](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29192) - Should we have failed over automatically during the disk IO degradation? (Owner: Bob Van Landuyt). ## Guidelines - [Blameless RCA Guideline](https://about.gitlab.com/handbook/customer-success/professional-services-engineering/workflows/internal/root-cause-analysis.html) - [5 whys](https://en.wikipedia.org/wiki/5_Whys)