Active-active geo log cursors
Description
Currently, we don't have a way to scale the Geo log cursor horizontally. This is potentially a major limititation for Geo at GitLab.com scale
We can have multiple Geo log cursors in a secondary node. One is active, all the others are passive. This provides HA, but not horizontal scaling. We can only increase the number of events processed per second by getting that active log cursor to process more.
However, we don't have a strict requirement to execute log events in-order. Most event processing actually happens asynchronously, in sidekiq. This means that we don't really have a strict requirement for events to be processed sequentially. This means we can process them in parallel.
So, as long as we introduce a mechanism by which log cursor processes don't process the same event in parallel, we can scale the geo log cursor horizontally.
Proposal
Currently, we control which cursor process is active by having them all compete for a single lease in redis. We could introduce 2 leases ("odd" and "even", and have processes to compete for them.
Each log cursor then only processes events for its half of the workload.
This is trivially generalizable to N leases instead of 2.