Implement circuitbreaker/roundTripper using gobreaker
What does this MR do and why?
Implement circuitbreaker/roundTripper using gobreaker
The first of a series of MRs to add a circuitbreaker to workhorse. This circuitbreaker aims to reduce load on rails when the user has been rate limited. If the user receives a set number of consecutive 429 responses from rails, workhorse will immediately respond with a 429 instead of forwarding the request to rails.
Uses workhorse's existing Redis connection to persist the circuitbreaker state.
This currently only works for ssh. Https uses basic authentication, and we won't have the key_id to use as a unique user identifier. I'll prepare a followup MR that uses the user's gitlab username as a unique key to make this also work for https.
References
Related to https://gitlab.com/gitlab-org/gitlab/-/issues/525377
Screenshots or screen recordings
| Before | After |
|---|---|
How to set up and validate locally
To try this out locally, I've created a test branch (freinink/525377-workhorse-gobreaker-test-branch) that wraps the internal api proxy with this circuit breaker and adds a loggingMiddleware.
- Checkout
freinink/525377-workhorse-gobreaker-test-branch - Via ssh, successfully do some git operations (clone, push, pull)
- Comment out this line so calls to
allowedwill always return Too Many Requests gdk tail workhorse- Via ssh, do the same git operations. They should all fail
- Inspect workhorse logs, you should see some gobreaker logs (
gobreaker: state change) to indicate the circuitbreaker was tripped - Subsequent requests don't reach
/allowed
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.