Github api rate limit not handled correctly when importing project with single_endpoint_notes_import option set to true
Summary
After updating our tests to use newly added api option single_endpoint_notes_import
, it seems like we don't handle rate limit correctly when importing notes (comments) from github.
When running import of rspec/rspec-core
projects, we are consistently missing around 2000 mr comments. The amount of comments is not exact, but very similar from run to run which is similar to previous issues we had when hitting api rate limit.
Steps to reproduce
Import https://github.com/rspec/rspec-core with single_endpoint_notes_import
to true
What is the current bug behavior?
Missing mr comments that are otherwise present when single_endpoint_notes_import
is not enabled.
What is the expected correct behavior?
All comments should be imported
Relevant logs and/or screenshots
Only relevant error I see in the logs for these failing runs is diffsnoteimport
running in to rate limit.
{"feature_category":"importers","import_type":"github","severity":"ERROR","time":"2022-10-24T03:14:49.130Z","correlation_id":"79b5cd7b7d38555d460c469aeaab9453","project_id":2,"source":"Gitlab::GithubImport::Importer::SingleEndpointDiffNotesImporter","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError"}
On the run that has this option disabled, there is a rate limit error as well for notes import, but all of the comments are there.
{"feature_category":"importers","import_type":"github","severity":"ERROR","time":"2022-10-24T07:53:57.750Z","correlation_id":"adbf67ab528c806be309cddf0caf3e1f","project_id":2,"source":"Gitlab::GithubImport::Importer::SingleEndpointIssueEventsImporter","message":"importer failed","error.message":"Gitlab::GithubImport::RateLimitError"}
data.json <- diff with missing stuff. Missing labeled/unlabeled events is unrelated, it's a different problem