Implement circuit breaker in Rails Gitaly client for ResourceExhausted errors
What does this MR do and why?
Implements a circuit breaker pattern in Gitlab::GitalyClient to handle ResourceExhausted errors from Gitaly. When Gitaly returns capacity limit errors, the circuit breaker opens to reduce load and returns errors early without making additional calls.
This helps protect GitLab from cascading failures when Gitaly is under heavy load.
Key features
- Circuit breaker per service/rpc endpoint - Isolates failures so one endpoint's issues don't affect others
- Bypasses circuit breaker for authenticated requests - Only unauthenticated requests are throttled to avoid blocking logged-in users
- Handles both direct and stream responses - Works correctly for all Gitaly RPC patterns
-
Released behind
gitaly_circuit_breakerfeature flag - Safe rollout with ability to disable if issues arise
Implementation details
-
New
Gitlab::GitalyClient::CircuitBreakerclass - Wraps calls with circuit breaker logic -
Integration into
Gitlab::GitalyClient::Call- Applies circuit breaker to all Gitaly RPC calls -
Updated
Gitlab::CircuitBreakerutility - Addedexception:parameter for exception-based error handling
Closes https://gitlab.com/gitlab-org/gitlab/-/issues/582826
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Follow-up items
- Add Prometheus metrics (https://gitlab.com/gitlab-org/gitlab/-/work_items/587646+)
- Configure thresholds based on analysis from #582823 (separate MR)
- Create feature flag rollout issue
Edited by Vasilii Iakliushin