# [#INC-12449: gitlab.com HTTP timeout over 30s](https://app.incident.io/gitlab/incidents/12449) Generated by Mike Eddington on 27 Jul 2026 23:02. All timestamps are local to Etc/UTC # Key Information | Metric | Value | | ------ | ------ | | Customers Affected | All GitLab.com SaaS users; elevated 503s and timeouts across web, API, git, CI runners, AI-assisted, and package registry traffic. Platform Metrics show ~9.49M HTTP 503 responses during the peak 19:43–19:58 UTC window (see general platform triage dashboard). | | Requests Affected | All GitLab.com web, API, git, CI runner, AI-assisted, and generic package registry requests. PgBouncer waiters peaked at 3,680 and HTTP 503s at ~9.49M during the incident (see [PgBouncer overview dashboard](https://dashboards.gitlab.net/d/pgbouncer-main?from=1785181200000&to=1785182700000&var-PROMETHEUS_DS=Mimir+-+Gitlab+Gprd&var-environment=gprd&viewPanel=60) and [PostgreSQL overview](https://dashboards.gitlab.net/d/postgresql-pgbouncer_overview?from=1785181200000&to=1785181680000&var-PROMETHEUS_DS=mimir-gitlab-gprd&var-environment=gprd&viewPanel=16)). | | Incident Severity | Severity 1 | | Impact Start Time | Mon, 27 Jul 2026 19:46:00 UTC | | Impact End Time | Mon, 27 Jul 2026 21:03:00 UTC | | Total Duration | 1 hour, 17 minutes | | Link to Incident Issue | https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22579 | # Summary **Problem**: GitLab.com experienced HTTP timeouts over 30 seconds during a burst of package registry traffic. **Impact**: Customers saw elevated HTTP timeouts, 503 errors, and load-related failures across GitLab.com. The incident also affected package registry access and caused downstream runner API errors. **Causes**: A burst of Nix generic-package download traffic from a single external user repeatedly updated `packages_packages.last_downloaded_at` on every GET request without throttling, creating a PostgreSQL lock convoy on a small set of package rows. This overwhelmed the primary database connection pools and related web components (PgBouncer, Puma, Workhorse), cascading into broad GitLab.com HTTP failures. A contributing factor was that the application-level RackAttack rate limit for authenticated package API requests (`throttle_authenticated_packages_api_enabled`) was disabled in production, meaning requests bypassed an existing protection that could have mitigated the traffic burst. **Response strategy**: We added temporary rate limiting and blocking controls (Cloudflare rule) to reduce the traffic that was causing the overload. Responders identified and implemented a code fix to throttle `last_downloaded_at` updates and moved the incident to monitoring after the system recovered naturally as the traffic burst stopped. # What went well? 1. Responders quickly correlated Postgres lock metrics, PgBouncer saturation, and HTTP 503 spikes, allowing them to narrow in on the `packages_packages.last_downloaded_at` UPDATE as the dominant SQL fingerprint early in the incident. 2. Cross-functional collaboration between Infrastructure, database experts, and application engineers allowed fast identification of the write-on-read pattern in the generic package registry API and rapid authoring of the throttle MR. 3. A temporary Cloudflare rate-limiting rule was designed and deployed quickly as a preventive measure, then codified into Terraform (`config-mgmt` MR 14886) so the protection would persist across rule updates. 4. The incident channel captured key links (dashboards, Kibana queries, MRs, and Cloudflare configuration) in real time, giving a clear evidence trail for this review and for follow-up actions such as the FCL and GET-write audit work item. # What was difficult? 1. The write-on-read pattern for `last_downloaded_at` on generic package downloads was not clearly documented, and existing dashboards did not highlight it as a risk ahead of time, so responders had to infer the pattern from Postgres logs, locks, and Marginalia sampling under pressure. 2. We lacked bind values in historical telemetry for the problematic UPDATE, which prevented us from confirming the exact contended package IDs or fully quantifying how many distinct packages versus repeated downloads were involved. 3. PgBouncer pool configuration (24 connections per process with `reserve_pool = 0`) left no headroom for short bursts, so contention on a narrow set of rows quickly saturated all Web/API pools and amplified the blast radius across unrelated routes. 4. Workhorse's `ResponseHeaderTimeout` of 5 minutes meant it continued waiting on stalled Puma responses, filling the Workhorse listening socket backlog and causing nginx upstream connect timeouts; we did not have targeted guidance or runbooks for how to respond to this failure mode. 5. The existing Cloudflare package rate-limit rule did not apply effectively to this traffic pattern (deploy-token usage, bursty request profile), and we did not have clear documentation explaining which client types and authentication methods are (or are not) protected by particular rules. 6. Per-pod webservice and accept-queue telemetry was incomplete, so we could not definitively rule out pod-level issues versus DB-wait saturation as the mechanism for nginx connect() timeouts; responders had to rely on indirect evidence. 7. The application-level RackAttack rate limit for authenticated package API requests (`throttle_authenticated_packages_api_enabled`) was disabled in production (`=> false`). Due to the interaction between our rate limiting frameworks, this meant requests effectively bypassed application-level throttling entirely. This was not surfaced during the incident response and was only identified in post-incident review. See [rate limiting team investigation](https://gitlab.com/gitlab-com/engineering-division/pmo/portfolios/gitlab-com-reliability/-/merge_requests/179). If Capacity Overload Performance Issues was a contributing factor of the incident: - There is now a Feature Change Lock (FCL) scoped to the Package Registry for INC-12449 ([FCL work item #111](https://gitlab.com/gitlab-com/feature-change-locks/-/work_items/111)), which includes planning for capacity-related improvements. - Improved Tamland-style coverage for PgBouncer saturation and DB lock patterns (especially in combination with Workhorse backlog signals) could have provided earlier, more actionable alerts; this is being considered as part of the follow-up capacity and observability work. - Extending capacity planning to explicitly cover PgBouncer reserve pools, Workhorse timeouts, and "write-on-read" hotspots should help prevent similar database-induced global saturation in the future. # Investigation Details # Incident Timeline **2026-07-27** • **19:46:00** **Impact started** • **19:47:18** **Incident reported in triage by Pingdom alert** • **19:47:25** **Escalation acknowledged** • **19:51:40** **Incident accepted** • **19:51:40** **Escalated to GitLab.com Production** • **19:51:40** **Escalated to Incident Manager Oncall (IMOC)** • **19:51:51** **Escalation acknowledged by IMOC** • **19:52:55** **Escalation acknowledged by SRE** • **19:53:07** **Severity upgraded from Severity 2 → Severity 1** • **19:53:08** **Escalated to Infrastructure Leadership** • **20:23:16** **Status changed from Investigating → Monitoring** • **20:38:07** **Temporary Cloudfare rate-limit was applied** • **21:04:30** **Incident resolved and entered the post-incident flow** # Investigation Notes During 19:43–19:58 UTC, a burst of Nix generic-package downloads from a single customer's CI job repeatedly hit the generic package registry download API. Each GET request issued a synchronous `UPDATE "packages_packages" SET "last_downloaded_at" = $1 WHERE "packages_packages"."id" = $2` without throttling, causing heavy contention on a small set of package rows. PostgreSQL observed new `ExclusiveLock` and `ShareLock` queues on the `packages_packages` table, along with 764 statement timeouts (SQLSTATE 57014) and 177 aborted transactions (SQLSTATE 25P02) tied to this SQL fingerprint. These lock queues drove PgBouncer saturation across the `gitlabhq_production` Web/API sync pools (up to 3,680 waiting clients and 8.03 hours of aggregated wait time). As Rails requests backed up behind PgBouncer, Puma workers saturated across web, git (cny), and AI-assisted services. Workhorse, configured with a `ResponseHeaderTimeout` of 5 minutes, continued to wait on stalled Puma responses, filling its listen backlog. Nginx then logged `connect() failed (110: Operation timed out)` to webservice upstreams, manifesting as 502/503 errors and >30s timeouts across unrelated routes (e.g., `/api/v4/runners/jobs`, `/api/v4/user`, `POST /api/graphql`, artifact uploads, repository APIs). The traffic source was traced via Postgres logs and Marginalia to legitimate Nix cache usage, not abuse. The system recovered naturally once the Nix download burst completed and the PgBouncer and worker queues drained, before the Cloudflare rate-limit rule became active. Stan Hu then authored [MR !247205](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/247205) to throttle `last_downloaded_at` updates to at most once per minute per record, turning most repeated GETs into cheap no-op UPDATEs. We validated that: - The production rollout range did not include changes touching generic-package downloads, `last_downloaded_at`, or Web/API DB connection behavior. - Patroni storage and IO metrics showed no disk or replication issues at onset, and Cloudflare/http-router remained healthy while origin-side 5xxs spiked. - The dedicated PgBouncer configuration (pool size 24, `reserve_pool = 0`) was a pre-existing capacity boundary rather than a fresh regression. Post-incident review identified that `throttle_authenticated_packages_api_enabled` was set to `false` in production, meaning the RackAttack application-level rate limit for authenticated package API requests was not active. Due to the interaction between rate limiting frameworks, this allowed the high-volume Nix download traffic to pass through without application-level throttling. This was a pre-existing configuration state and was not a regression introduced at the time of the incident. There is still some uncertainty about which exact package IDs were most heavily contended (due to missing bind values in long-term telemetry) and the precise per-pod accept-queue behavior during the peak saturation window. # Lessons Learned ### General lessons _What did we learn from this incident that we want to carry forward?_ 1. Even seemingly low-cost "last accessed" writes on GET paths can create severe lock convoys under bursty, repeat-download patterns; these must be treated as capacity-sensitive and either throttled, made asynchronous, or removed. 2. PgBouncer pool headroom (`reserve_pool`) and Workhorse/Puma timeout behavior form a critical part of GitLab.com's blast-radius control; tuning them purely for steady-state efficiency leaves the platform exposed to narrow hot-spot contention. 3. Observability that correlates DB lock types, SQL fingerprints, PgBouncer waiters, and HTTP 5xx/latency is essential for quickly distinguishing database contention from infrastructure or network issues. 4. Cloudflare rate-limit rules need to be evaluated against all supported client types (including deploy tokens and CI-based tooling like Nix) to ensure protections actually apply to high-volume automated access patterns. 5. The incident confirms that self-managed and Dedicated instances using the generic package registry share the same code path; fixes like MR !247205 should be backported to supported self-managed/Dedicated releases to reduce the chance of similar outages in those environments. 6. Application-level rate limit flags (e.g., `throttle_*_enabled` settings in RackAttack) must be audited to ensure they are actually enabled in production. A rate limit that is configured but disabled provides no protection, and the interaction between limiting frameworks can mean that a disabled application-level limit allows traffic to bypass controls entirely. ### What can we do to reduce detection time by half? _Consider: alerting gaps, missing metrics, lack of dashboards, unclear ownership, delayed escalation paths._ 1. Add or refine alerts that directly detect patterns of DB lock buildup and PgBouncer saturation (especially on `packages_packages` and similar "last accessed" tables), with clear runbook links describing how to distinguish lock convoys from infra faults. 2. Create focused dashboards and runbooks that tie together DB locks, PgBouncer waiters, Workhorse backlog, Puma saturation, and HTTP 5xx/latency, so responders can quickly recognize this specific failure mode without reconstructing the story from multiple views. 3. Add targeted alerts or Tamland coverage on high-error-rate generic package registry requests (e.g., sudden increases in 5xxs or latency on Nix/generic paths), with ownership clearly assigned to the Package team. ### What can we do to reduce resolution time by half? _Consider: missing runbooks, unclear mitigation steps, tooling gaps, communication delays, lack of automation._ 1. Document a standard mitigation playbook for DB lock convoy incidents that includes: (a) how to identify the dominating SQL fingerprint, (b) how to safely apply short-term traffic shaping (e.g., per-path Cloudflare rate limits), and (c) how to assess whether Workhorse and PgBouncer tuning changes are appropriate in the moment. 2. Extend the package registry runbooks to explicitly call out any synchronous writes on GET paths, how they are throttled, and how to disable or override them temporarily if they misbehave. 3. Improve Cloudflare configuration documentation and templates for package registry traffic, including tested patterns for handling deploy-token and CI-driven access, so responders can more quickly create safe, targeted rate-limiting rules during incidents. 4. Conduct an audit of all `throttle_*_enabled` application settings to identify any rate limits that are configured but currently disabled, and establish a process to review and enable them or document the rationale for keeping them disabled. # Actions - [codify temporary CF rule into TF](https://app.incident.io/gitlab/incidents/12449?tab=actions) - [Throttle `packages_packages.last_downloaded_at` updates for generic packages](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/247205) - Audit and enable `throttle_authenticated_packages_api_enabled` and review all other `throttle_*_enabled` flags to ensure application-level rate limits are active where intended (identified post-incident by @swiskow; see [rate limiting team investigation](https://gitlab.com/gitlab-com/engineering-division/pmo/portfolios/gitlab-com-reliability/-/merge_requests/179)) # Follow-ups - https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29434+ - https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/29433+ # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [x] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] For all S1 and S2 incidents, begin the [Feature Change Lock (FCL) process](https://handbook.gitlab.com/handbook/engineering/#feature-change-locks) and [open an issue in the FCL project](https://gitlab.com/gitlab-com/feature-change-locks/-/issues/new?description_template=feature-change-lock). (Done: [Package Registry FCL for Incident INC-12449](https://gitlab.com/gitlab-com/feature-change-locks/-/work_items/111)) - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [x] S1 incidents [require a public RCA within 7 days](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/incident-review/#timeline-that-we-expect-for-reviews-to-be-completed) of the incident. If this review cannot be made public, [create a separate public RCA](https://handbook.gitlab.com/handbook/engineering/root-cause-analysis/#how-to-perform-an-rca). See https://gitlab.com/gitlab-org/gitlab/-/work_items/607116+ - [ ] Close the review before the due date - [ ] Go back to the incident channel or page and close out the remaining post-incident tasks