2026-03-09: Gitaly goserver apdex SLO violation in cny stage

Gitaly goserver apdex SLO violation in cny stage (Severity 2)

Problem: The gitaly goserver component in the cny stage experienced degraded performance, with apdex dropping below SLO thresholds and elevated error rates.

Impact: Significant slowness for users with projects hosted on Gitaly Canary 1. Most affected users are internal GitLab staff, but external users who fork projects landing on this node may also experience degraded performance.

Causes: A small number of users, mostly GitLab employees, generated large volumes of SSH upload-pack requests, causing resource exhaustion and degraded performance on Gitaly Canary 1. Much of the traffic was unauthenticated and difficult to attribute, with some activity likely automated by AI-driven tooling.

Response strategy: We banned the problematic user and killed large git pack-object processes, reducing memory pressure and process counts. System health is improving and metrics are returning toward normal levels.

This ticket was created to track INC-8175, by incident.io 🔥