2026-08-18: Loadbalancer error rate violating SLO

Loadbalancer error rate violating SLO (Severity 2)

Problem: The web service loadbalancer in the main stage reported elevated 5xx errors, causing an error rate of 0.1514% over the last hour and violating its SLO.

Impact: Customers saw elevated 500 errors when loading or fetching commits on GitLab.com. This affected web, API, Git operations, and CI/CD user flows.

Causes: We have not confirmed a single root cause. dw_read_blobs_notifications lined up with the start of the incident, but we have not established that it caused the issue. We also confirmed Gitaly FindCommit ResourceExhausted errors across multiple storage hosts, with increased node_schedstat_waiting and traffic concentrated on large repository and fork networks.

Response strategy: We disabled the suspected feature flag, ruled out a canary-only issue, and paged the Gitaly team. We also reviewed Gitaly and scheduler signals and confirmed the service indicators were returning toward normal, but we have not confirmed a final fix yet.