Draft: Replace circuit breaker with rate limited user cache

Fred Reininkrequested to merge
freinink/579767-replace-circuitbreaker-with-rate-limit-cache into master

What does this MR do and why?

Replace circuit breaker with simplified rate limit cache

Reduces calls to redis by removing the need to manage a state machine with mutexes. Instead, cache the value of the retry-after header returned by rails when receiving a 429. Store this value in Redis, but also cache locally to reduce overall calls to Redis.

This is a pretty significant refactoring with a very large diff, but I've basically kept the same structure as in the previous implementation, and just removed the interface with gobreaker.

I intend to follow-up to remove the circuit breaker config values, and potentially rename circuitbreaker to something like ratelimitcache or blockcache.

References

Related to https://gitlab.com/gitlab-org/gitlab/-/work_items/579767

Screenshots or screen recordings

Before After

How to set up and validate locally

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Fred Reinink

Merge request reports