Create branches for pull mirrors of large repositories asynchronously

What does this MR do and why?

When the mirror for a repository is missing many of the branches from the upstream repository instead of trying to create all the branches in the same process, enqueue jobs after a certain threshold to create the branches scheduled with some delay between them and set the mirror to a failed state.

The delay between branch creation jobs should reduce the impact on gitaly.

Avoid setting import state to finished when creating branches async

Leave the import state as started to allow the async branch creation to finish before allowing the mirror to be scheduled for update again

How to set up and validate locally

Testing the happy path

  1. Enable the async_mirror_branch_creation feature flag
  2. Adjust Projects::UpdateMirrorService::ASYNC_BRANCH_THRESHOLD to be something practical to test with
  3. Create a new empty project and mirror another project with many branches (for example: gitlab-org/gitlab-test)
  4. Check that there are Repositories::CreateBranchesFinishWorker enqueued and Repositories::CreateBranchWorker being scheduled (watch log/sidekiq.log or /admin/sidekiq/scheduled)

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #387717

Edited by Jerry Seto

Merge request reports

Loading