Sync deleted records to elasticsearch if index gap occurs

Problem to solve

When elasticsearch has been disabled for a period of time then enabled again, the record deleted during this time period has to be removed from the index.

Intended users

GitLab Starter users

Proposal

As elasticsearch is sharded by routing, which is the project id, looping through by projects would be the most performant (if possible). This means:

1. going through projects table
   1. if project exists, loop related nested models, see if there are gaps from consecutive iid.
   2. if there is a gap in project id, remove all things related to this project id.

For Note and Snippet, there is no iid so the above would not apply. We can only loop through these tables directly and ask Elasticsearch to delete without giving routing.

Documentation

What does success look like, and how can we measure that?

Deleted documents during elasticsearch being disabled will not show up in search results once elasticsearch is enabled again.

Links / references

A sub issue of #5299 (closed).