Skip to content
Snippets Groups Projects

Introduce circuit breaker for OpenAI client

Merged Bojan Marjanovic requested to merge bmarjanovic/circuit-breaker into master

What does this MR do and why?

As a part of https://gitlab.com/gitlab-org/gitlab/-/issues/403947 we are adding a circuit breaker around OpenAI calls.

We are using the circuitbox library to implement the circuit breaker pattern.

Screenshots or screen recordings

N/A

How to set up and validate locally

  1. Change the values inside the Gitlab::Llm::Concerns::CircuitBreaker, to easily validate the functionality
    • ERROR_THRESHOLD
    • VOLUME_THRESHOLD
  2. Call the OpenAI client multiple times with a big payload, e.g. .chat(content: 'DUMMY'*300000)
  3. Validate it doesn't raise an error
  4. Validate that the circuit is open, and the requests are not being sent.

Down below is a short script to validate the functionality:

  1. Update the Gitlab::Llm::Concerns::CircuitBreaker module with the following values:
@circuit ||= Circuitbox.circuit(service_name, {
  exceptions: [InternalServerError],
  error_threshold: 1,
  volume_threshold: 5,
  time_window:      60,
  sleep_window:     300
})
  1. Open the rails console and run the following command: 6.times.map { Gitlab::Llm::OpenAi::Client.new(User.first).chat(content: "TEST"*3000000) }
  2. After one minute, repeat the command
  3. The requests are not being sent.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Peter Leitzen

Merge request reports

Merged results pipeline #844205525 passed with warnings

Pipeline: GitLab

#844205583

    Pipeline: Ruby 3 forced pipeline

    #844210848

      Pipeline: GitLab

      #844213774

        +2

        Merged results pipeline passed with warnings for 6b3bffe2

        Test coverage 82.70% (15.16%) from 2 jobs
        Approval is optional

        Set by to be merged automatically when the pipeline succeeds

        Ready to merge by members who can write to the target branch.

        Merge details

        • 1 commit and 1 merge commit will be added to master (squashes 1 commit).
        • Source branch will be deleted.
        • Auto-merge enabled

        Activity

        Filter activity
        • Approvals
        • Assignees & reviewers
        • Comments (from bots)
        • Comments (from users)
        • Commits & branches
        • Edits
        • Labels
        • Lock status
        • Mentions
        • Merge request status
        • Tracking
        • Loading
        • Loading
        • Loading
        • Loading
        • Loading
        • Loading
        • Loading
        • Loading
        • Loading
        • Loading
        Please register or sign in to reply
        Loading