Consolidate PR import concurrency limits into shared setting
What does this MR do and why?
Previously, pull request import concurrency limits were managed separately across Bitbucket Server, Bitbucket Cloud, and GitHub importers using inconsistent, hardcoded mechanisms:
- Bitbucket Cloud (
PullRequestsImporter): hardcoded to100 - Bitbucket Server (
PullRequestsImporter): hardcoded to50(intentionally reduced to avoidGitlab::Git::ResourceExhaustedErrorfrom large refmaps) - GitHub (
PullRequestsImporter): hardcoded batch size of200
This meant that admins configuring the per-importer application settings (e.g. concurrent_bitbucket_server_import_jobs_limit) had no actual control over PR import concurrency — those settings only affected other object importers (issues, notes, milestones, etc.).
This MR introduces a new shared application setting concurrent_pull_request_import_jobs_limit (default: 200) that unifies PR import concurrency control across all three importers under a single, admin-configurable knob. This simplifies Gitaly load protection management and makes the Admin API settings actually effective for PR imports.
Bitbucket Server ref-batching: The Bitbucket Server PR importer issues explicit fetch_remote calls for merged/closed pull requests. Originally, the refmap size was implicitly bounded to 50 because concurrent_import_jobs_limit fed into its construction. A later unrelated change (!197510 (merged)) broke that connection, causing fetch_remote to use PER_PAGE (100) instead. MAX_REFS_PER_FETCH = 50 restores the limit by batching fetch_remote calls into chunks of 50 refs each, independently of concurrent_pull_request_import_jobs_limit.
Behavior changes:
- GitHub PR importer: no effective change (was already
200) - Bitbucket Server PR importer: concurrency limit raised from
50→200(the new default), with ref-batching added to keep individualfetch_remotecalls safe - Bitbucket Cloud PR importer: concurrency limit raised from
100→200(the new default)
References
Closes #597793
Screenshots or screen recordings
No UI changes — this is a backend-only change to application settings and importer logic.
How to set up and validate locally
1. Verify the new setting exists and has the correct default
# In a Rails console (bin/rails c)
ApplicationSetting.current.concurrent_pull_request_import_jobs_limit
# => 2002. Verify the setting is readable and writable via the API
# Read current value (requires admin token)
curl --header "PRIVATE-TOKEN: <admin_token>" \
"http://gdk.test:3000/api/v4/application/settings" | \
jq '.concurrent_pull_request_import_jobs_limit'
# => 200
# Update the setting
curl --request PUT \
--header "PRIVATE-TOKEN: <admin_token>" \
--header "Content-Type: application/json" \
--data '{"concurrent_pull_request_import_jobs_limit": 25}' \
"http://gdk.test:3000/api/v4/application/settings" | \
jq '.concurrent_pull_request_import_jobs_limit'
# => 253. E2E - Test the importers (optional)
Enable the importers in the Admin area
Before importing, make sure the relevant importers are enabled:
- Go to Admin area → Settings → General → Import and export sources
- Enable the importers you want to test: GitHub, Bitbucket Cloud, and/or Bitbucket Server
- Save changes
Import from GitHub
- Go to New project → Import project → GitHub
- Authenticate with a GitHub personal access token (requires
reposcope) - Select a repository that has pull requests and start the import
- Monitor the Sidekiq queues or check the Rails log to confirm
PullRequestsImporterjobs are enqueued and processed - After the import completes, verify that pull requests appear in the imported project
Import from Bitbucket Cloud
- Go to New project → Import project → Bitbucket Cloud
- Authenticate with your Bitbucket Cloud credentials
- Select a repository that has pull requests and start the import
- Verify that pull requests appear in the imported project after completion
Import from Bitbucket Server
Bitbucket Server requires a running Bitbucket Server instance, which is more involved to set up locally. The Group Import team maintains access to a shared Bitbucket Server instance — reach out to them if you need access.
- Go to New project → Import project → Bitbucket Server
- Enter the Bitbucket Server URL, username, and personal access token
- Select a repository that has pull requests (including merged/closed ones to exercise the ref-batching path) and start the import
- Verify that pull requests appear in the imported project after completion
- Confirm no
Gitlab::Git::ResourceExhaustedErrorerrors appear in the logs during the import
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.