Collapse duplicate replication jobs

Problem to Solve

Because replication jobs are greedy (they replicate all changes to the repo), if replication jobs are queued faster than they can be processed for the same repository, excess replication jobs may need to be processed creating extra read loads and blocking features like read distribution. Importantly, it can lead to the system incorrectly thinking a replica is out of date when it simply has a backlog of no-op fetches that need to be processed.

Further details

We have been working on the assumption that each mutator RPC will schedule a discrete replication job. Part of the problem with this strategy is that each replication is a greedy procedure, meaning that each replication will attempt to replicate everything that has changed, rather than simply replicating the changes resulting from the originating mutator RPC. This means that some replication jobs are redundant and unnecessary, and running unneeded replication jobs consumes resources.

Side discussion from https://gitlab.com/gitlab-org/gitaly/issues/2436

Many Jobs to Replication Process

One potential solution to redundant replication jobs is to mark all "ready" replication jobs as being in-progress when starting the replication process. For example, if there are 5 "ready" replication jobs scheduled for replicating changes from Repo-X-on-Storage-A to Repo-X-on-Storage-B, Praefect would mark all of those jobs as "in-progress" at the start of the replication process. When that replication process fails/succeeds, Praefect would mark them all accordingly.

In taking this approach, we might find the need to add a new table representing an "active" replication process. This way we know which scheduled jobs belong to that process, and if the process fails, we know which jobs to schedule for another attempt.

Flatten Jobs to Single Column

Another potential solution is to adopt a different data model for the replication jobs. Rather than create new discrete replication job per mutator RPC, we could update a single row in a table representing the state of a repo replica. For example, the Repo-replica table might look like this:

Field	Example
Repo	@hashed/abcd/1234
Storage	gitaly-1
Checksum	deadbeef
LastMutation	98adb22d7cb7de4b44fb2e6a78f84fb8434698ad
LastReplication	6bbebcdf3d9c9625a71a5f23737211bbd89371d9

Note:

• Primary key is a combination of the repo identifier and the storage location
• Replication is needed when LastMutation and LastReplication don't match
  • LastMutation corresponds to the checksum of the primary-replica a mutation completes. This is reset on mutation start and updated on mutation complete.
  • LastReplication is the checksum of the replica after replication completes. This is reset on replication start and updated on replication complete.