Sync deleted records to elasticsearch if index gap occurs
Problem to solve
When elasticsearch has been disabled for a period of time then enabled again, the record deleted during this time period has to be removed from the index.
Intended users
GitLab Starter users
Proposal
As elasticsearch is sharded by routing
, which is the project id, looping through by projects would be the most performant (if possible). This means:
- going through projects table
- if project exists, loop related nested models, see if there are gaps from consecutive
iid
. - if there is a gap in project id, remove all things related to this project id.
- if project exists, loop related nested models, see if there are gaps from consecutive
For Note
and Snippet
, there is no iid
so the above would not apply. We can only loop through these tables directly and ask Elasticsearch to delete without giving routing
.
Documentation
What does success look like, and how can we measure that?
Deleted documents during elasticsearch being disabled will not show up in search results once elasticsearch is enabled again.
Links / references
A sub issue of #5299 (closed).
Edited by Mark Chao