Project import is too slow for exports from large groups
Summary
As part of our restoration efforts for a customer in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13645, we observed project imports taking upwards of 3hrs, even for very small (e.g. <500KB) gitlab project exports.
It's our belief that user mapping and authorization refresh activity was consuming the bulk of the import time, since the projects contained zero (or single digit numbers) issues, and very little metadata.
The project imports we were performing in that context had no git bundles (i.e. no repo data) included either).
Steps to reproduce
- Export a project from a gitlab.com namespace with thousands of users and thousands of subgroups.
- Attempt a gitlab project import.
Example Project
Many available in https://gitlab.com/gitlab-com/gl-infra/infrastructure/-/issues/13645
What is the current bug behavior?
Project import takes hours.
What is the expected correct behavior?
Project import should take only seconds.
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)
Proposed solution
- Do not run
refresh_member_authorized_projects
callback ifimporting?
(for both Group and Project Import) - Enqueue
ProjectRecalculateWorker
at the end of project import in theProject#after_import