Skip to content

Implement redis store for circuitbox

Patrick Bajao requested to merge 408208-circuit-breaker-redis-store into master

What does this MR do and why?

By default, the circuit breaker is using a memory store to track the state of the circuit. This is okay for a single node environment since threads can share the store.

On production though, we have multiple nodes (puma and sidekiq) so a need for a persisted store is required.

This adds a custom store that will utilize redis as the storage. It uses the redis-ratelimiting DB for this since it's kind of related with rate limiting.

The Gitlab::CircuitBreaker::Store is based on Circuitbox's memory store (https://github.com/yammer/circuitbox/blob/main/lib/circuitbox/memory_store.rb) but modified to use redis.

This is behind the circuit_breaker feature flag.

How to set up and validate locally

  1. Update the Gitlab::Llm::Concerns::CircuitBreaker module with the following values:
@circuit ||= Circuitbox.circuit(service_name, {
  exceptions: [InternalServerError],
  error_threshold: 1,
  volume_threshold: 1,
  time_window:      60,
  sleep_window:     300
})
  1. Open the rails console and run the following command: 3.times.map { Gitlab::Llm::OpenAi::Client.new(User.first).chat(content: "TEST"*3000000) }
  2. Open another rails console and run the same command. Requests should not be sent.
  3. After 5 minutes, execute the following command on both console: Gitlab::Llm::OpenAi::Client.new(User.first).chat(content: "Hello"). It should be sent and response is returned.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #408208 (closed)

Edited by Patrick Bajao

Merge request reports