Implement redis store for circuitbox
What does this MR do and why?
By default, the circuit breaker is using a memory store to track the state of the circuit. This is okay for a single node environment since threads can share the store.
On production though, we have multiple nodes (puma and sidekiq) so a need for a persisted store is required.
This adds a custom store that will utilize redis as the storage. It uses the redis-ratelimiting DB for this since it's kind of related with rate limiting.
The Gitlab::CircuitBreaker::Store
is based on Circuitbox's memory
store (https://github.com/yammer/circuitbox/blob/main/lib/circuitbox/memory_store.rb)
but modified to use redis.
This is behind the circuit_breaker
feature flag.
How to set up and validate locally
- Update the
Gitlab::Llm::Concerns::CircuitBreaker
module with the following values:
@circuit ||= Circuitbox.circuit(service_name, {
exceptions: [InternalServerError],
error_threshold: 1,
volume_threshold: 1,
time_window: 60,
sleep_window: 300
})
- Open the rails console and run the following command:
3.times.map { Gitlab::Llm::OpenAi::Client.new(User.first).chat(content: "TEST"*3000000) }
- Open another rails console and run the same command. Requests should not be sent.
- After 5 minutes, execute the following command on both console:
Gitlab::Llm::OpenAi::Client.new(User.first).chat(content: "Hello")
. It should be sent and response is returned.
MR acceptance checklist
This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.
-
I have evaluated the MR acceptance checklist for this MR.
Related to #408208 (closed)