2025-07-31: Elevated queue durations for saas-linux-small runner shards
Elevated queue durations for saas-linux-small runner shards (Severity 3)
Problem: There was an elevated queue duration for saas-linux-small runner shards due to CI/CD runner VM creation failures.
Impact: The majority of jobs experienced up to 15-minute wait times, affecting user experience with delayed CI/CD Runners' queue handling.
Causes: The creation failures were primarily due to Google Cloud regional capacity exhaustion in us-east1-c and us-east1-d.
Response strategy: The issue was escalated to Google, who confirmed they are investigating. Follow-up actions have been created to improve alerting for future runner creation failures.
This ticket was created to track INC-3034, by incident.io