2025-09-15: API service loadbalancer and nginx_ingress error rate SLO violation

API service loadbalancer and nginx_ingress error rate SLO violation (Severity 2)

Problem: API / web / gits services and CI runners returned high rates of 5xx errors, causing widespread service degradation and slow or failed operations across GitLab.com.

Impact: Users across GitLab.com experienced slow or failing operations.

Causes: Underperforming patroni node caused long-running database transactions leading to timeouts on several services: API, Web, Gitaly, Ci runner. Root cause is still TBD.

Response strategy: We published a status page update to inform users. The team is investigating the long-running transactions. System metrics and error rates are now returning to normal.

This ticket was created to track INC-3966, by incident.io 🔥