Skip to content

Introduce circuit breaker for OpenAI client

Bojan Marjanovic requested to merge bmarjanovic/circuit-breaker into master

What does this MR do and why?

As a part of https://gitlab.com/gitlab-org/gitlab/-/issues/403947 we are adding a circuit breaker around OpenAI calls.

We are using the circuitbox library to implement the circuit breaker pattern.

Screenshots or screen recordings

N/A

How to set up and validate locally

  1. Change the values inside the Gitlab::Llm::Concerns::CircuitBreaker, to easily validate the functionality
    • ERROR_THRESHOLD
    • VOLUME_THRESHOLD
  2. Call the OpenAI client multiple times with a big payload, e.g. .chat(content: 'DUMMY'*300000)
  3. Validate it doesn't raise an error
  4. Validate that the circuit is open, and the requests are not being sent.

Down below is a short script to validate the functionality:

  1. Update the Gitlab::Llm::Concerns::CircuitBreaker module with the following values:
@circuit ||= Circuitbox.circuit(service_name, {
  exceptions: [InternalServerError],
  error_threshold: 1,
  volume_threshold: 5,
  time_window:      60,
  sleep_window:     300
})
  1. Open the rails console and run the following command: 6.times.map { Gitlab::Llm::OpenAi::Client.new(User.first).chat(content: "TEST"*3000000) }
  2. After one minute, repeat the command
  3. The requests are not being sent.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Peter Leitzen

Merge request reports