2026-04-30: The ci_runner_jobs SLI of the ci-runners service on shard saas-linux-small-amd64 has an apdex violating SLO

The ci_runner_jobs SLI of the ci-runners service on shard saas-linux-small-amd64 has an apdex violating SLO (Severity 2)

Problem: The ci_runner_jobs SLI for the ci-runners service on the saas-linux-small-amd64 shard dropped below its SLO due to lack of available compute resources, resulting in blocked job execution and a significant backlog.

Impact: From around 13:59 UTC to 16:50 UTC, customers using hosted GitLab runners on the saas-linux-small-amd64 shard were unable to start new CI jobs. The job queue peaked at about 9,070 pending jobs, blocking job execution. Only this shard was affected. As of 16:55 UTC, job processing has returned to normal, the backlog is cleared, and customers can start jobs as usual.

Causes: Google Cloud resource exhaustion in all three us-east1 zones prevented new compute machines from being created for the saas-linux-small-amd64 shard. Persistent stockout errors for the previous machine type (N2D) were the primary cause.

Response strategy: - Increased T2D machine type quotas to 3,000 CPUs per project across all nine affected Google Cloud projects

• Migrated runner managers to T2D machine type, focusing on the most impacted zones

• Monitored queue and error rates, confirming job processing is now normal and the pending queue is reduced to 20 jobs

• Team will continue to monitor for recurrence, as underlying Google Cloud capacity issues persist but without current customer impact

This ticket was created to track INC-9698, by incident.io 🔥