Geo: Add gap handling in Log Cursor
From gitlab-com/migration#295 (comment 88041617), we see that the Log Cursor has gaps on occasion. This leads to events not being skipped on occasion.
I added gitlab-org/gitlab-ee!6513 to give us more logging as to how often this occurs. It looks like this affects repository updates events the most, which is not too surprising because that dominates the event log.
I think we should modify the Log Cursor to do something like:
- Detect any gaps and save the event IDs in Redis or in the database with the timestamp
- Periodically check whether the gaps appeared in the event log. If the timestamp is older than some time (e.g. 10 minutes) and replication lag is under control, drop the gap.
- Otherwise, process the gap.
Edited by Stan Hu