Review GitHub Import spread job limit
Add rate limiter for PullRequest importer (!81026 - merged) implemented a mechanism to ensure that the GitHub Import importer jobs don't pressure other services. With this new mechanism, GitHub Import is restricted to importing 200 pull requests per minute, while for other resources, the limit is 1000 per minute. This means that even if the instance has enough resources, the import process won't be able to use them due to these limits.
Importing a project that has a large number of merge requests, comments, diff notes, events, etc can take a long time due to these limits, as estimated by the Migration Estimate spreadsheet
For example, importing a project with 100000 MRs, 5 comments per MR, 5 merge request reviews, 5 merge requests, and 5 events would at least take 44 hours to finish as GitHub Import wouldn't be able to execute importer workers faster than that.
(100,000 / 200 + 100000 * ( 5 + 5 + 5 + 5 ) / 1000) / 60 = 44 hours
Proposed solutions
We should analyze if we can increase the 1000 limit since the number was arbitrarily chosen.
For example, we could customize the limit via application settings so administrators could customize it based on their instance resources.
Another option would be to use the defer worker feature on GitHub Import workers, which would delay the execution of the workers in case the database is unhealthy.
Chosen solution
After analyzing the logs, we found out that the hardcoded limit of 1000 is not appropriate for all instances. For instance, when we migrated Rails and Kubernetes in our 3K instance, the importer workers started accumulating in the scheduled
queue, even though there were sufficient resources available for processing them.
However, it is quite challenging to determine a number that would fit all cases. There are numerous variables that would impact the limit.
Allowing the limit to be configured through application settings is the best short-term solution. This will enable Self-Managed and Dedicated instances to adjust the limit as per their use case.
So, for now, we will move the hardcoded setting to an application setting in which the default value is 1000; in other words, the limit would continue the same on .com and other existing instances.