2026-06-17: Broken master
Broken master (Severity 2)
Problem: Automated abuse of the repository archive endpoint saturated the Gitaly 'GetArchive' queue, causing timeouts for archive requests and blocking new package builds and deployments.
Impact: New package builds and deployments using the 'e2e:test-on-cng' pipeline were blocked in the cny environment due to archive request timeouts. Already-built package deployments were unaffected. Legitimate package builds using curl were not impacted. As of 20:53 UTC, service has recovered and new builds and deployments can proceed.
Causes: A large volume of automated archive download requests using the wget client, coming from many IPs, saturated the Gitaly 'GetArchive' queue and caused resource exhaustion. Blocking the main abusive IP from Hong Kong reduced some traffic, but distributed abuse from other IPs continued. No recent Gitaly code changes were found to be responsible. See GitLab issue: #22331 (closed).
Response strategy: What we did: Deployed a Cloudflare custom block rule (hard block/403) targeting GET requests to /api/v4/projects/278964/repository/archive.zip from the Wget/1.21. client on gitlab.com. This was considered safe because long-term historical traffic showed no legitimate use of this signature. The rule is live, and a back-port to Terraform is prepared for ongoing management.
Current state:
• Gitaly GetArchive queue on the cny node has recovered.
• e2e:test-on-cng / build-cng is unblocked; new package builds and deployments can proceed.
• The rule is active in Cloudflare and managed as code.
Known limitations: The block is narrow (by user agent and path), so attackers may evade it by changing user agent or archive format. We are monitoring for recurrence and ready to extend the rule.
Follow-ups:
• Implement internal Gitaly throttling of unauthenticated GetArchive requests (rate limiting).
• Complete standard governance for the mitigation.
Recommendation: We recommend monitoring continue for another eight hours (until 07:53 UTC on 18 June 2026).
This ticket was created to track INC-11183, by incident.io