Zero Downtime Re-indexing
This Epic was created a while back which was intended to capture the issues that would be related to zero downtime reindexing. We have been evaluating approaches of doing this. Most recently, we are rolling out Elasticsearch to more customers on GitLab.com, which has also given us operational experience on indexing/reindexing on production environment. One approach stands out and appears to us as a viable solution for the short term by utilizing Elasticsearch index alias and re-index API. @DylanGriffith has captured the related work in a proposal, which is also copied here
[From Slack](https://gitlab.slack.com/archives/C3TMLK465/p1586302510125700)
I propose that the next 3 steps for supporting reindexing with zeroish downtime should be:
1. Implement a switch to pause all of the write requests to elasticsearch => https://gitlab.com/gitlab-org/gitlab/-/issues/204826
2. GitLab Elasticsearch integration should only ever read/write from alias => https://gitlab.com/gitlab-org/gitlab/-/issues/213628
3. GitLab Elasticsearch integration support for in-cluster re-indexing => https://gitlab.com/gitlab-org/gitlab/-/issues/213629
Yesterday we were able to reindex our cluster in 3 hours and I'd argue that this delay in indexing is probably pretty reasonable and there may even be further ways of speeding this up. Also there would be further incremental steps to make it really zero downtime by using the aliases to search both indices at the same time. If we do this I believe we can hold off on all the other admin redesign multiple index stuff since this will be capable of solving all the reindexing cases we have planned and further I feel there will likely be creative additions we can add to this to support future migrations that may be different to the ones we've just done.
These approaches are based now on some more operational experience that leads me to believe this architecture will be able to perform reindexes much faster with much less strain on other systems. It's also informed by the fact that this approach has worked and appears to work for all of the upcoming cases we have.
Also importantly this approach is iterative in that every one of these changes is actually a very small feature to implement without wide reaching consequences and each one of them alone would have helped us out with the recent migrations we've been doing.
We can have a meeting to discuss in more detail or do async (whichever works) but these steps would require some small frontend changes that would certainly benefit from some time from UX and FE though based on what I can see would probably be less FE work than getting the other multi index MR merged.
This Epic will include issues that are related to this approach
epic