Add rate limit handling to imports from GHES

What does this MR do and why?

This MR updates the GitHub import client to wait for rate limits to expire when importing from GitHub Enterprise Server (GHES). Previously, our importer assumed that rate limiting was never enabled on GHES instances, when rate limiting is only disabled by default on GHES instances. So when importing from rate-limited instances, nothing stopped executing import workers even though API requests to fetch resources would raise errors, and eventually the workers would exhaust their retries and fail to import all expected project resources. The consequences were more pronounced with lower rate limits.

References

Screenshots or screen recordings

This MR does not make any changes to the UI, but its effects are visible on the project import history page:

Before After
image image
image image

How to set up and validate locally

  • Set up the GHES test instance and request access to a user on the instance according to https://gitlab.com/gitlab-org/foundations/import-and-integrate/team/-/blob/main/importers/github_enterprise_server_test_instance.md
  • Using the ghe_root_site_admin credentials located in the shared engineering 1Password vault, log into the GHES management console, enable rate limiting, and set the limit to something low.
    • Note: This feature makes it so that importing may take a long time. With core limits to 50 requests per hour, the vuejs-core project took about 3 hours to import into my GDK. I didn't see a way to change the time window for rate limits, so each time a limit is reached means an additional hour to import.
  • Import one of the projects on the instance using the GitLab API
  • Verify that the import completed as expected with all records imported. Verify that rate limits were encountered by checking importer.log for entries containing "severity": "INFO" and "exception_class":"Gitlab::GithubImport::RateLimitError"
  • Verify imports from https://github.com are unaffected

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #621582 (closed)

Edited by Sam Word

Merge request reports

Loading
Loading