2026-06-30: Gitaly goserver apdex SLO violation in cny

Gitaly goserver apdex SLO violation in cny (Severity 2)

Problem: The Gitaly goserver in the cny stage suffered severe degraded performance, with apdex dropping to 65.6% around 14:37–14:40 UTC and continuing to fluctuate due to high concurrent git clone and fetch activity, mainly from unauthenticated sources.

Impact: Users in the cny stage experienced widespread 500 errors, failed CI job checkouts, and slow or failed page loads, especially when accessing gitlab-org/gitlab and related forks. This disrupted internal and external users' workflows and blocked deployments and feature flags.

Causes: A surge of git upload-pack and pack-objects requests, mostly from unauthenticated wide-scale clone activity targeting gitlab-org/gitlab and its forks, led to high memory usage and host load. This overwhelmed the Gitaly service in the cny stage, resulting in severe performance issues. The problematic traffic was initially traced to a single suspicious IP, but further investigation confirmed significant distributed fork activity also contributed to the overload.

Response strategy: 1. We deployed a custom rule to block a suspicious IP generating heavy traffic. 2. On affected hosts, we terminated large pack-related processes and restarted the Gitaly service. 3. We lowered the Cloudflare rate limit for unauthenticated fork requests from 60/second to 30/second and reduced unauthenticated git-pack-upload concurrency from 10 to 5. 4. We continue to observe improved concurrency and error rates after these mitigations, with ongoing monitoring and adjustments as needed.

This ticket was created to track INC-11609, by incident.io 🔥