Skip to content

Optimize GitHub importer for speed and memory

What does this MR do?

It applies some improvements to GitHub importer. Importing of a private GH repo (~1500 issues, ~180 PRs, ~3600 commits), was taking over 30 minutes and memory usage reached 950 MB, after these changes it took 8 minutes and memory was at 700 MB max (for both cases, Sidekiq starts at ~400 MB).

Are there points in the code the reviewer needs to double check?

N/A

Why was this MR needed?

People are asking for it, see issues below.

Screenshots (if relevant)

N/A

Does this MR meet the acceptance criteria?

What are the relevant issue numbers?

https://gitlab.com/gitlab-com/support-forum/issues/996 and https://gitlab.com/gitlab-org/gitlab-ce/issues/19124#note_13588843

Merge request reports