# [#INC-9343: The ci_runner_jobs SLI of the ci-runners service on shard `saas-linux-small-amd64` has an apdex violating SLO](https://app.incident.io/gitlab/incidents/9343) Generated by Kam Kyrala on 20 Apr 2026 18:18. All timestamps are local to Etc/UTC # Key Information | Metric | Value | | ------ | ------ | | Customers Affected | All GitLab.com users running CI jobs on the `saas-linux-small-amd64` shared runner shard during the impact window | | Requests Affected | CI jobs submitted to `saas-linux-small-amd64`. `ci_runner_jobs` apdex dropped from ~100% to ~20% at the worst point (alert fired at 43.59%); pending queue peaked around 24K jobs. | | Incident Severity | ~"Severity::2" | | Impact Start Time | Mon, 20 Apr 2026 14:30:00 UTC| | Impact End Time | Mon, 20 Apr 2026 17:25:00 UTC | | Total Duration | 2 hours, 55 minutes | | Link to Incident Issue | [INC-9343](https://app.incident.io/gitlab/incidents/9343) | | Correlated GCP Incident | [FQYFFGP](https://status.cloud.google.com/) — `us-east1-d`, multiple products (GCE, GKE, Persistent Disk, Cloud SQL, Cloud Build, Dataflow, Filestore) | # Summary **Problem**: The apdex score for ci_runner_jobs on the saas-linux-small-amd64 shard of the ci-runners service dropped to 43.59%, violating the SLO. This caused CI pipelines to be delayed and jobs to remain pending. **Impact**: Multiple users experienced CI pipeline delays and jobs stuck in pending on the saas-linux-small-amd64 shard. This resulted in a significant backlog of jobs and an increase in support ticket volume about job pickup failures. Service has now recovered, with Apdex returning to healthy levels and job queues trending down. **Causes**: A confirmed Google Cloud platform incident ([FQYFFGP](https://status.cloud.google.com/)) in `us-east1-d` — caused by an error in quota utilization calculation in an underlying storage layer preventing new Persistent Disk creation — surfaced to GitLab as `ZONE_RESOURCE_POOL_EXHAUSTED` errors on `n2d-standard-2` VM inserts. Because all six d-pinned runner managers on `saas-linux-small-amd64` could not create new VMs, CI runner capacity was exhausted on the shard and new CI jobs were not picked up. **Response strategy**: We reconfigured runner nodes to use the us-east1-c zone to relieve capacity exhaustion and reduce the job queue backlog. All nodes were updated and reloaded with new configuration. Monitoring confirms that Apdex has recovered and job queues are clearing. A follow-up has been created to expand hosted runner support to a third zone and reduce risk from future zonal issues. # What went well? 1. **Fast escalation.** Alert fired 15:06 UTC; within ~35 min the on-call EOC, Production on-call, IMOC, and Runners Platform had engaged. 2. **Rich, actionable telemetry.** Grafana (`ci-runners-main/ci-runners-overview`), runner-manager logs, and GCP audit logs made it possible to narrow the cause to `ZONE_RESOURCE_POOL_EXHAUSTED` on `n2d-standard-2` in `us-east1-d` within roughly 30 min of focused investigation. 3. **Correct upstream-provider diagnosis ahead of GCP's public acknowledgement.** Responders identified the issue as zone-localized to `us-east1-d` and began mitigation by ~17:00 UTC — approximately 3 hours before Google's public "Symptom identified" update at 2026-04-20 16:01 PDT / ~23:01 UTC on the Service Health page for incident `FQYFFGP`. Canary `gcloud compute instances create` runs and Kibana served as ground truth in the absence of a public GCP signal. 4. **Direct canary validation before config change.** Running `gcloud compute instances create` in candidate zones was used to confirm `n2d-standard-2` capacity in `us-east1-b` and `us-east1-c` before editing chef — avoiding blind config changes on production runners. 5. **Known-working mitigation pattern was available.** Moving managers away from the exhausted zone is an approach this team has used before (INC-7530, INC-9055); the response did not require inventing a mitigation from scratch. 6. **Follow-up created before close.** A structural follow-up to expand hosted runners to a third zone ([production-engineering#28738](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28738)) was opened before resolution, capturing context while it was fresh. # What was difficult? 1. **Recurring structural issue.** This is at least the 5th firing of `CiRunnersServiceCiRunnerJobsApdexSLOViolationSingleShard` on the same shard (INC-6014, INC-6283, INC-7530, INC-8376, INC-9055, and now INC-9343). The underlying two-zone concentration has been flagged in prior postmortems but not addressed. The confirmed GCP incident in `us-east1-d` (FQYFFGP) reinforces that this concentration is exactly the failure mode a third zone protects against — a provider-side zonal outage is not a capacity-planning problem we can solve by buying more VMs. 2. **No usable fallback during single-zone exhaustion.** Every `saas-linux-small-amd64` runner manager is pinned 1:1 to either `us-east1-c` or `us-east1-d` via `MachineOptionsMap.google-zone` — so when one zone exhausted, we lost half the fleet with no automatic failover. 3. **Mitigation re-exposed the same risk.** Moving all managers to `us-east1-c` briefly triggered `ZONE_RESOURCE_POOL_EXHAUSTED` in `c` as load doubled on that zone. Demand dropped in time to avoid a second SLO breach, but the mitigation was one-zone-to-one-zone, not multi-zone. 4. **No upstream provider signal during the active incident.** Google did not publicly acknowledge the `us-east1-d` incident on Service Health until well after our mitigation was underway. Responders had to infer the provider-side nature of the issue from audit logs and canary creates. Better integration of GCP Personalized Service Health into the EOC workflow could shorten this lag on future provider incidents. 5. **Early mis-read of telemetry.** Initial aggregated queries suggested both `us-east1-c` and `us-east1-d` were exhausted, which delayed a focused mitigation. Direct Kibana searches and a canary VM create in `c` later showed `c` was always healthy — only `d` was failing. Zone attribution in the log aggregations was noisier than expected. 6. **Uncertainty about `us-east1-b` viability.** Mid-incident, we could not quickly confirm whether HAProxy / CI-gateway / Cloud NAT would support ephemeral VMs in `us-east1-b`, which held back using it as a fallback. The information existed (chef-repo, Terraform, past commits, INC-4485 history) but was not consolidated. 7. **Shard zone pinning is opaque without reading chef.** The 1-manager-per-zone layout is explicit in `roles/runners-manager-saas-linux-small-amd64-*.json` but not visible in runbooks or dashboards, so responders outside Runners Platform could not quickly reason about zone impact. If Capacity Overload Performance Issues was a contributing factor of the incident: • Is there an existing Capacity Planning issue related to this incident? Yes. [Epic &1459](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/1459) tracks the VPC migration for hosted runners. Michelle Gill flagged the `n2d-standard-2` SKU on `saas-linux-small-amd64` as a capacity concern on 25 Mar 2026, noting recurring queue spikes and autoscaling past 16K machines. Capacity Planning Tracker issues [#2353](https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2353), [#2373](https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2373), [#2425](https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2425), and [#2458](https://gitlab.com/gitlab-com/gl-infra/capacity-planning-trackers/gitlab-com/-/issues/2458) reference this shard. • Could the incident reasonably have been avoided with improved Tamland coverage? No — not in this case. With the GCP Service Health correlation now confirmed (FQYFFGP), the triggering failure was a provider-side bug in PD quota-utilization calculation in `us-east1-d`, not organic demand exceeding our forecast. Tamland cannot forecast a provider software defect. However, Tamland improvements could still reduce blast radius of a future similar event — see below. • Should Capacity Planning be extended to prevent future occurrences of this, or similar, incidents? Yes. Recommended: track per-zone per-SKU create-failure rate as a leading indicator, and include "effective capacity assuming one zone is unavailable" in the shard capacity model. The follow-up to expand to a third zone ([production-engineering#28738](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28738)) is the primary structural remedy — with a confirmed upstream zonal outage now on record, this is a resilience requirement rather than a capacity optimization. # Investigation Details <details> <summary> Timeline</summary> # Incident Timeline All times UTC, 2026-04-20 unless otherwise noted. **14:30** — Impact begins. `ci_runner_jobs` apdex on `saas-linux-small-amd64` starts degrading. Root cause (identified later): `ZONE_RESOURCE_POOL_EXHAUSTED` on `n2d-standard-2` in `us-east1-d`, affecting the six d-pinned runner managers (green-2/4/6, blue-2/4/6). Now confirmed as a downstream symptom of Google Cloud incident FQYFFGP. **15:06** — Prometheus Alertmanager fires `CiRunnersServiceCiRunnerJobsApdexSLOViolationSingleShard`. Incident auto-created in Triage. **15:07 – 15:08** — Igor Wiedler (EOC) acknowledges escalation, accepts incident at **Severity 3**, status → Investigating. **15:34 – 15:35** — Escalated to GitLab.com Production on-call (Alex Hanselka ack). Kam Kyrala manually escalates tier2 DevOps Rails. **15:41** — Severity upgraded to **Severity 2**. IMOC (Donald Cook) engaged. Investigation active. **~15:45 – 16:30** — Investigation narrows the cause to runner capacity pressure. Grafana and runner-manager logs show a high sustained `no_free_executor` rate from ~14:00 UTC. GCP audit logs show `ZONE_RESOURCE_POOL_EXHAUSTED` (status.code=8) on `n2d-standard-2` VM inserts — concentrated in `us-east1-d`. **16:01 (PDT) / ~23:01 UTC** — *(post-mitigation, added retroactively)* Google publishes "Symptom identified" for incident FQYFFGP on Service Health — the first public acknowledgement of the `us-east1-d` issue. GitLab mitigation was already complete by this time. **16:32** — Public update: cause narrowed to executor shortage; team evaluating cross-zone options with awareness of network/HAProxy trade-offs. **~16:30 – 17:05** — Canary `gcloud compute instances create` tests confirm: - `us-east1-c` accepts `n2d-standard-2` creates (sanity check; existing managers use c). - `us-east1-b` accepts `n2d-standard-2` creates (potential fallback; never before used by this shard). Kibana cross-check confirms the exhaustion is isolated to `us-east1-d` — `us-east1-c` has zero `ZONE_RESOURCE_POOL_EXHAUSTED` matches in the incident window. **~17:00 – 17:15** — Mitigation begins: d-pinned managers repointed to `us-east1-c` via chef role edit + converge. First manager updated and successfully reloaded. **17:06 – 17:11** — Brief secondary spike: `us-east1-c` starts throwing `ZONE_RESOURCE_POOL_EXHAUSTED` as load doubles on it. Peak ~1,100 errors/min at 17:06. Subsides as demand drops — no SLO re-breach. **17:08** — Public update: mitigation in progress, reconfiguring to `us-east1-c`. A separate Windows runner issue (runner-manager process not running since April 15) noted as unrelated. **17:25** — **Impact ends.** Apdex recovers toward 100%. No further zone-exhaustion errors on c or d. Pending queue begins to drain sharply. **17:28** — Status → Monitoring. **17:42** — Public update: backlog drained, apdex healthy, queue clearing. Continuing to monitor. **18:14** — Follow-up [production-engineering#28738](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28738) opened to expand hosted runners to a third zone. Status → Documenting. **2026-04-20 17:30 (PDT) / 2026-04-21 00:30 UTC** — *(post-mitigation, added retroactively)* Google marks FQYFFGP as resolved for all affected users/projects (per Service Health: "resolved... as of Monday, 2026-04-20 10:30 US/Pacific"). # Investigation Notes </details> **Root cause.** A confirmed Google Cloud platform incident ([FQYFFGP](https://status.cloud.google.com/)) in zone `us-east1-d`. Per Google's preliminary analysis, the incident was caused by "inability to create new Persistent Disk (PD) resources within the us-east1-d zone due to an error in calculation of quota utilization in an underlying storage layer." This surfaced to GitLab as `ZONE_RESOURCE_POOL_EXHAUSTED` on `n2d-standard-2` VM inserts in `us-east1-d`, because VM creation depends on PD allocation. Successful single-VM canary creates during the incident were consistent with intermittent rather than total failure — matching Google's description of a quota-calculation bug rather than an exhausted physical pool. GCP's mitigation was to bypass the faulty logic in the affected zone and facilitate PD allocation in nearby zones. The same GCP incident impacted Google Compute Engine, GKE, Persistent Disk, Cloud SQL, Cloud Build, Dataflow, and Filestore in `us-east1-d` — all services we rely on, though only GCE/PD were in the runner-create path. **Why the shard was structurally exposed.** - All 12 managers for `saas-linux-small-amd64` are distributed across exactly two zones: `us-east1-c` (green-1/3/5, blue-1/3/5) and `us-east1-d` (green-2/4/6, blue-2/4/6). - Each manager is pinned to a single zone via `MachineOptionsMap.google-zone` in chef — no zonal failover in the docker-machine google driver. - This concentration is defined in `gitlab-com/gl-infra/chef-repo/roles/runners-manager-saas-linux-small-amd64-{blue,green}-{1..6}.json` and has not changed since the shard was created. - With only two zones in rotation, any single-zone provider incident (as FQYFFGP now demonstrates) removes 50% of shard capacity with no automatic recovery path. Expanding to a third zone is the direct remedy. **Why `us-east1-b` was not used during mitigation.** - No saas-linux runner manager has ever been configured for `us-east1-b` — verified in chef-repo git history. - GitLab does operate runner fleets in `us-east1-b` elsewhere (Distribution runners in the `omnibus-build-runners` project, using `n1-highcpu-32`). - For hosted runners, the CI Gateway HAProxy and Cloud NAT topology supporting `us-east1-b` was not immediately verifiable during the incident, which raised concern about moving workload there mid-incident. The follow-up issue addresses that gap. **Interaction with INC-4485.** The Oct 2025 incident was a separate failure of the zonal CI-gateway URL on private runners; the Oct mitigation was to bypass the CI-gateway and route runner traffic to `gitlab.com` directly. Shared runners already use the CI-gateway ILB (regional), rolled out in May 2022 (chef-repo commit `070a98abc`). The INC-4485 mitigation did not affect `saas-linux-small-amd64` and was not a factor in this incident. **External provider incident correlation (FQYFFGP).** - **Google Cloud Service Health Event ID:** `FQYFFGP` - **GCP-reported impacted zone:** `us-east1-d` (exact match to our d-pinned manager set) - **GCP-reported impacted products:** Cloud Build, Cloud Filestore, Cloud Dataflow, Cloud SQL, Google Compute Engine, Google Kubernetes Engine, Persistent Disk — GCE and PD are directly in the runner VM provisioning path - **GCP-reported incident window:** 2026-04-17 21:44:48 PDT (start) → 2026-04-20 10:30 PDT (resolved) / 11:42:05 PDT (public resolution notice) - **GCP "Symptom identified" timestamp:** 2026-04-20 09:01:17 PDT (~16:01 UTC) — after GitLab had already narrowed the cause to `us-east1-d` exhaustion via internal telemetry - **GCP root cause (preliminary):** quota-utilization calculation error in an underlying storage layer preventing new PD creation in `us-east1-d` - **GCP mitigation:** bypassed the problematic logic in `us-east1-d`; facilitated PD allocation in nearby zones - **Consistency with our observations:** canary `n2d-standard-2` creates in `us-east1-d` during our incident window succeeded intermittently rather than failing uniformly, consistent with a quota-calculation bug (partial allowance) rather than hard resource exhaustion. `us-east1-c` and `us-east1-b` were unaffected in both GitLab observations and in the GCP incident scope. **Adjacent noise — image pull failures.** A spike in `saas_linux_runner_image_pull_failures` occurred during the incident window, but investigation confirmed these were customer-side `pull access denied` / `manifest unknown` / Docker Hub rate-limit errors from a variety of images and registries — not a platform regression and not a primary driver of the apdex drop. **Mitigation details.** - Moved d-pinned managers (green-2/4/6) to `us-east1-c` via chef role edit and manual converge. - Success rate on `us-east1-d` was effectively zero during focused bursts at peak; last recorded d-zone spike at ~16:55–17:11 UTC then trailed off. - `us-east1-c` briefly saw `ZONE_RESOURCE_POOL_EXHAUSTED` after the move (peak ~1,100/min at 17:06) but subsided as pending demand drained; the shard stayed above SLO breach. **Follow-ups.** - [production-engineering#28738](https://gitlab.com/gitlab-com/gl-infra/production-engineering/-/work_items/28738) — Expand hosted runners to `us-east1-b` (third zone). **Priority reinforced** by confirmed GCP zonal outage on record. - Consider revisiting SKU choice (`n2d-standard-2`) as part of the third-zone work, per Michelle Gill's earlier proposal. - **New:** Evaluate integrating [GCP Personalized Service Health](https://cloud.google.com/service-health/docs) alerts into the EOC workflow so that provider-side incidents surface to responders earlier (in this incident, Google's public acknowledgement came after our mitigation was complete). # Follow-ups **Follow-up** **Owner** [Expand hosted runners to us-east1-b to reduce single-zone exhaustion risk](https://app.incident.io/gitlab/incidents/9343?tab=post-incident) Unassigned # Review Guidelines This review should be completed by the team which owns the service causing the alert. That team has the most context around what caused the problem and what information will be needed for an effective fix. The EOC or IMOC may create this issue, but unless they are also on the service owning team, they should assign someone from that team as the DRI. ### For the person opening the Incident Review - [x] Set the title to `Incident Review: (Incident issue name)` - [x] Assign a `Service::*` label (most likely matching the one on the incident issue) - [x] Set a `Severity::*` label which matches the incident - [x] In the `Key Information` section, make sure to include a link to the incident issue - [x] Find and Assign a DRI from the team which owns the service (check their slack channel or assign the team's manager) **The DRI for the incident review is the issue assignee.** ### For the assigned DRI - [x] Fill in the remaining fields in the `Key Information` section, using the incident issue as a reference. Feel free to ask the EOC or other folks involved if anything is difficult to find. - [x] If there are metrics showing `Customers Affected` or `Requests Affected`, link those metrics in those fields - [x] For all S1 and S2 incidents, begin the [Feature Change Lock (FCL) process](https://handbook.gitlab.com/handbook/engineering/#feature-change-locks) and [open an issue in the FCL project](https://gitlab.com/gitlab-com/feature-change-locks/-/issues/new?description_template=feature-change-lock) :point_right: https://gitlab.com/gitlab-com/feature-change-locks/-/work_items/95+ - [x] Create a few short sentences in the Summary section summarizing what happened (TL;DR) - [x] Link any corrective actions and describe any other actions or outcomes from the incident - [x] Consider the implications for self-managed and Dedicated instances. For example, do any bug fixes need to be backported? - [x] Once discussion wraps up in the comments, summarize any takeaways in the details section - [x] If the incident timeline does not contain any sensitive information and this review can be made public, turn off the issue's confidential mode and link this review to the incident issue. - [x] S1 incidents [require a public RCA within 7 days](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/incident-review/#timeline-that-we-expect-for-reviews-to-be-completed) of the incident. If this review cannot be made public, [create a separate public RCA](https://handbook.gitlab.com/handbook/engineering/root-cause-analysis/#how-to-perform-an-rca). - [x] Close the review before the due date - [x] Go back to the incident channel or page and close out the remaining post-incident tasks