Gitlab::GithubImport::RateLimitError when importing a Github project with a large number of PRs
Summary
We're working on importing a large github project into GitLab. Project is 15GB in size and has just short of 80k PRs. We're hitting a Gitlab::GithubImport::RateLimitError when trying to import.
Relevant logs and/or screenshots
root@sandbox2:/var/log/gitlab/gitlab-rails# grep -i critical importer.log
root@sandbox2:/var/log/gitlab/gitlab-rails# grep -i error importer.log
{"severity":"ERROR","time":"2021-04-29T00:15:32.260Z","correlation_id":"78f8e3d7a5686b881564b73f355d3926","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T01:14:33.626Z","correlation_id":"e4a2bf913353ed7509420b9836e47511","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T02:12:10.620Z","correlation_id":"782261026beaf86fd761ec94a80ee592","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T03:13:04.836Z","correlation_id":"5560b03020bef4d7fafe05d579061882","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T04:14:20.310Z","correlation_id":"e7c4e130abcba45a4cbb0ae3a889e296","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T05:14:50.676Z","correlation_id":"827a7e78917ae58d413c3fd41d7b38be","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T06:16:45.481Z","correlation_id":"dcce895b17323fab789070c5e731ec4a","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T07:15:30.455Z","correlation_id":"26c1d47cdb85d33cccd3c38370bdddab","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T08:18:11.776Z","correlation_id":"a7ebadf55dab37a3eb1bb7ce7e3e4833","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T09:18:02.244Z","correlation_id":"2fc227fc13651a2d1e3ad7c6ee7811dd","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T10:18:26.461Z","correlation_id":"50a4ec3f64dd2451dc3355f4f7b66501","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T11:19:04.783Z","correlation_id":"f9088082885ca18b999b81eece744eb5","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-29T12:19:17.287Z","correlation_id":"95089aaf2241a897d39f89da2dba0f81","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
root@sandbox2:/var/log/gitlab/gitlab-rails# date
Thu Apr 29 14:07:24 UTC 2021
root@sandbox2:/var/log/gitlab/gitlab-rails# zgrep -i error importer.log.1.gz
{"severity":"ERROR","time":"2021-04-28T19:56:02.388Z","correlation_id":"01F4CMGMKBNJJKW00953RDEGG0","message":"importer failed","error.message":"4:Deadline Exceeded. debug_error_string:{\"created\":\"@1619639762.000731710\",\"description\":\"Deadline Exceeded\",\"file\":\"src/core/ext/filters/deadline/deadline_filter.cc\",\"file_line\":69,\"grpc_status\":4}","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestImporter"}
{"severity":"ERROR","time":"2021-04-28T19:57:14.094Z","correlation_id":"3dbf06f7902f805cd8886eb6715b0e21","message":"importer failed","error.message":"4:Deadline Exceeded. debug_error_string:{\"created\":\"@1619639834.001818980\",\"description\":\"Deadline Exceeded\",\"file\":\"src/core/ext/filters/deadline/deadline_filter.cc\",\"file_line\":69,\"grpc_status\":4}","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestImporter"}
{"severity":"ERROR","time":"2021-04-28T19:58:13.148Z","correlation_id":"e6f69e711b990a5f26a49252618207a8","message":"importer failed","error.message":"4:Deadline Exceeded. debug_error_string:{\"created\":\"@1619639893.000515997\",\"description\":\"Deadline Exceeded\",\"file\":\"src/core/ext/filters/deadline/deadline_filter.cc\",\"file_line\":69,\"grpc_status\":4}","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestImporter"}
{"severity":"ERROR","time":"2021-04-28T20:00:41.114Z","correlation_id":"b9c03d98cf8a15141cd73e337f46b80f","message":"importer failed","error.message":"4:Deadline Exceeded. debug_error_string:{\"created\":\"@1619640041.000655564\",\"description\":\"Deadline Exceeded\",\"file\":\"src/core/ext/filters/deadline/deadline_filter.cc\",\"file_line\":69,\"grpc_status\":4}","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestImporter"}
{"severity":"ERROR","time":"2021-04-28T20:03:39.186Z","correlation_id":"f39092d77a04c7dfba046664a1b29f18","message":"importer failed","error.message":"4:Deadline Exceeded. debug_error_string:{\"created\":\"@1619640219.000789676\",\"description\":\"Deadline Exceeded\",\"file\":\"src/core/ext/filters/deadline/deadline_filter.cc\",\"file_line\":69,\"grpc_status\":4}","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestImporter"}
{"severity":"ERROR","time":"2021-04-28T20:09:36.092Z","correlation_id":"2f6cbd3425e7008e3f5169a906b25b61","message":"importer failed","error.message":"4:Deadline Exceeded. debug_error_string:{\"created\":\"@1619640576.000831228\",\"description\":\"Deadline Exceeded\",\"file\":\"src/core/ext/filters/deadline/deadline_filter.cc\",\"file_line\":69,\"grpc_status\":4}","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestImporter"}
{"severity":"ERROR","time":"2021-04-28T22:06:36.908Z","correlation_id":"24e65b6210914f6ed215bc058c5e9969","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
{"severity":"ERROR","time":"2021-04-28T23:19:35.116Z","correlation_id":"8c7ca138c5b9fc7907ecd87396caebb1","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError","import_source":"github","project_id":6,"importer":"Gitlab::GithubImport::Importer::PullRequestMergedByImporter","parallel":true}
Possible fixes
Discussing this with @kassio on slack, it appears the error is stemming out of this loop: https://gitlab.com/gitlab-org/gitlab/blob/master/lib/gitlab/github_import/importer/pull_requests_merged_by_importer.rb#L31. It appears the large number of PRs would cause this loop to fail, and upon a new attempt the cache would be cold again, which would cause the failure to repeat itself and this would repeat ad aeternum.