Fix deadlocking on writes in Praefect

One key problem Praefect suffers from, which leads to replicas falling behind the primary is deadlocks that occur during concurrent write operations.

In this example, we have two clients that send two requests to Praefect concurrently.

  • update refs/heads/branch-a to deadbeef
  • delete refs/heads/branch-a

Let's say that it ends up on the Primary in the order:

  1. update refs/heads/branch-a to deadbeef
  2. delete refs/heads/branch-a

However, when these get relayed to the secondaries, they end up landing in a swapped order:

  1. delete refs/heads/branch-a
  2. update refs/heads/branch-a to deadbeef

This means that the update operation would fail, and the vote would fail, leading to a replication event.

flowchart LR
    subgraph Clients
        C2[Client 2<br/>update refs/heads/branch-a → deadbeef]
        C1[Client 1<br/>delete refs/heads/branch-a]
    end

    subgraph Gitaly Cluster
        Primary[Primary<br/>1. update <br/>2. delete ✓]
        S1[Secondary 1<br/>1. delete <br/>2. update ✗]
        S2[Secondary 2<br/>1. delete <br/>2. update ✗]
    end

    C1 --> Primary
    C2 --> Primary
    Primary --> S1
    Primary --> S2

    style S2 fill:#f96,stroke:#333

This affects large monorepositories, where a replication event can take a long time due to a quickly moving repository. Effectively, this means that read distribution will not work for the bulk of the workday, causing all reads to go to the primary.

We can fix this problem in praefect

Edited by Eric Ju