Implement progressive elasticsearch indexing for project mirrors
What does this MR do?
Currently, every time a project imports commits from another repo via the
pull mirror functionality, the elasticsearch repository index is completely regenerated. This is quite expensive - a complete reindex of the v4.4 Linux kernel, for instance, takes 40 (ruby) or 10 (experimental) minutes, depending on which indexer you use.
We store the last indexed commit in the database so it's simple to use this to make the indexing progressive, as it is for
git push operations.
Are there points in the code the reviewer needs to double check?
Why was this MR needed?
Screenshots (if relevant)
Does this MR meet the acceptance criteria?
- Changelog entry added, if necessary
- Documentation created/updated
- API support added
- Added for this feature/bug
- All builds are passing
- Conform by the merge request performance guides
- Conform by the style guides
Branch has no merge conflicts with
master(if it does - rebase it please)
- Squashed related commits together
What are the relevant issue numbers?
Closes #2881 (closed)