Implement circuitbreaker/roundTripper using gobreaker

What does this MR do and why?

Implement circuitbreaker/roundTripper using gobreaker

The first of a series of MRs to add a circuitbreaker to workhorse. This circuitbreaker aims to reduce load on rails when the user has been rate limited. If the user receives a set number of consecutive 429 responses from rails, workhorse will immediately respond with a 429 instead of forwarding the request to rails.

Uses workhorse's existing Redis connection to persist the circuitbreaker state.

This currently only works for ssh. Https uses basic authentication, and we won't have the key_id to use as a unique user identifier. I'll prepare a followup MR that uses the user's gitlab username as a unique key to make this also work for https.

References

Related to https://gitlab.com/gitlab-org/gitlab/-/issues/525377

Screenshots or screen recordings

Before After

How to set up and validate locally

To try this out locally, I've created a test branch (freinink/525377-workhorse-gobreaker-test-branch) that wraps the internal api proxy with this circuit breaker and adds a loggingMiddleware.

  1. Checkout freinink/525377-workhorse-gobreaker-test-branch
  2. Via ssh, successfully do some git operations (clone, push, pull)
  3. Comment out this line so calls to allowed will always return Too Many Requests
  4. gdk tail workhorse
  5. Via ssh, do the same git operations. They should all fail
  6. Inspect workhorse logs, you should see some gobreaker logs (gobreaker: state change) to indicate the circuitbreaker was tripped
  7. Subsequent requests don't reach /allowed

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Fred Reinink

Merge request reports

Loading