2025-09-10: Apdex SLO violation for workhorse web requests in cny stage

Apdex SLO violation for workhorse web requests in cny stage (Severity 4 (Low))

Problem: Web requests in the cny stage had degraded performance, causing the Apdex score to fall below the SLO threshold over the last 6 hours.

Impact: Web requests through the workhorse component in the cny stage experienced degraded performance, with the Apdex score dropping as low as 96.6% between 00:50 and 01:30 UTC. Service quality fell below SLO thresholds for a short period, affecting user experience.

Causes: The incident was caused by a significant increase in requests per second and CPU saturation for Gitaly, primarily due to demanding gRPC calls such as PackObjectsHookWithSidechannel and PostUploadPackWithSidechannel. The slowness was traced to Gitaly, not the database.

Response strategy: No active response was required as the impact subsided on its own. There was no evidence of system abuse.

---

This ticket was created to track INC-3819, by incident.io 🔥