Create branches for pull mirrors of large repositories asynchronously
What does this MR do and why?
When the mirror for a repository is missing many of the branches from the upstream repository instead of trying to create all the branches in the same process, enqueue jobs after a certain threshold to create the branches scheduled with some delay between them and set the mirror to a failed state.
The delay between branch creation jobs should reduce the impact on gitaly.
Avoid setting import state to finished when creating branches async
Leave the import state as started to allow the async branch creation to finish before allowing the mirror to be scheduled for update again
How to set up and validate locally
Testing the happy path
- Enable the
async_mirror_branch_creationfeature flag - Adjust
Projects::UpdateMirrorService::ASYNC_BRANCH_THRESHOLDto be something practical to test with - Create a new empty project and mirror another project with many branches (for example:
gitlab-org/gitlab-test) - Check that there are
Repositories::CreateBranchesFinishWorkerenqueued andRepositories::CreateBranchWorkerbeing scheduled (watchlog/sidekiq.logor/admin/sidekiq/scheduled)
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Related to #387717