Skip to content

Add a circuit breaker to Bitbucket connection

What does this MR do and why?

Adds a circuit breaker to the Bitbucket Cloud importer to catch network-type issues. The circuit keeps track of requests that responds with 500+ status codes and if it reaches the circuit's thresholds, it stops making requests to the API for a while.

Since the request can return nil, this MR also lets nil responses be handled.

The circuit's unique identifier is Bitbucket::Connection (given by self.class.name). There will therefore be one circuit for all Bitbucket Cloud requests: if bitbucket.org responds with network errors, all bitbucket.org requests should be safely handled.

How to set up and validate locally

  1. Follow https://docs.gitlab.com/ee/integration/bitbucket.html to setup OAuth for BitBucket Cloud. You will need an account on https://bitbucket.org/ that uses the same email address as the account on the gdk instance. You may need to create a new user on the gdk for this.

  2. Make sure you add the bitbucket configuration in the development section of your config/gitlab.yml as shown at https://gitlab.com/gitlab-org/gitlab-development-kit/-/blob/main/doc/howto/gitlab-oauth2.md#set-up-gdk:

    development:
      <<: *base
      omniauth:
        providers:
        - { name: 'bitbucket',
            app_id: '...',
            app_secret: '...' }
  3. Log into the gdk with the relevant user credentials and in ...-/profile/account connect to bitbucket

  4. Create a project and repo on BitBucket.

  5. Enable the feature flags: Feature.enable(:bitbucket_parallel_importer) and Feature.enable(:bitbucket_importer_exponential_backoff)

  6. On your gdk instance, create a new project > click on Import project > Bitbucket Cloud > follow instructions to connect to https://bitbucket.org/.

  7. Import the project.

  8. In a rails console, grab the import credentials: credentials = Project.last.import_data.credentials

  9. Test that the client still works by making an API request e.g. Bitbucket::Client.new(credentials).connection.get("/repositories/<your namespace>")

  10. Update lib/gitlab/circuit_breaker/notifier.rb line 9 to print what happens to the circuit, e.g. puts "notify #{event} #{service_name}"

  11. Update lib/bitbucket/connection.rb line 29 to respond with a 500: OAuth2::Response.new(Faraday::Response.new(body: 'body', status: 501))

  12. Close and start a rails console again

  13. Run the request and observe the printed logs: 12.times { Bitbucket::Client.new(credentials).connection.get("/repositories/<your namespace>") }. Note that after a few failed requests, the circuit opens and stops sending requests.

MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #251222 (closed)

Edited by Madelein van Niekerk

Merge request reports