You need to sign in or sign up before continuing.
Optimize GitHub importer for speed and memory
What does this MR do?
It applies some improvements to GitHub importer. Importing of a private GH repo (~1500 issues, ~180 PRs, ~3600 commits), was taking over 30 minutes and memory usage reached 950 MB, after these changes it took 8 minutes and memory was at 700 MB max (for both cases, Sidekiq starts at ~400 MB).
Are there points in the code the reviewer needs to double check?
N/A
Why was this MR needed?
People are asking for it, see issues below.
Screenshots (if relevant)
N/A
Does this MR meet the acceptance criteria?
-
CHANGELOG entry added -
Documentation created/updated -
API support added - Tests
-
Added for this feature/bug -
All builds are passing
-
-
Conform by the merge request performance guides -
Conform by the style guides -
Branch has no merge conflicts with master
(if you do - rebase it please) -
Squashed related commits together
What are the relevant issue numbers?
https://gitlab.com/gitlab-com/support-forum/issues/996 and https://gitlab.com/gitlab-org/gitlab-ce/issues/19124#note_13588843