Fix deadlocking on writes in Praefect
One key problem Praefect suffers from, which leads to replicas falling behind the primary is deadlocks that occur during concurrent write operations.
In this example, we have two clients that send two requests to Praefect concurrently.
- update refs/heads/branch-a to deadbeef
- delete refs/heads/branch-a
Let's say that it ends up on the Primary in the order:
- update refs/heads/branch-a to deadbeef
- delete refs/heads/branch-a
However, when these get relayed to the secondaries, they end up landing in a swapped order:
- delete refs/heads/branch-a
- update refs/heads/branch-a to deadbeef
This means that the update operation would fail, and the vote would fail, leading to a replication event.
flowchart LR
subgraph Clients
C2[Client 2<br/>update refs/heads/branch-a → deadbeef]
C1[Client 1<br/>delete refs/heads/branch-a]
end
subgraph Gitaly Cluster
Primary[Primary<br/>1. update <br/>2. delete ✓]
S1[Secondary 1<br/>1. delete <br/>2. update ✗]
S2[Secondary 2<br/>1. delete <br/>2. update ✗]
end
C1 --> Primary
C2 --> Primary
Primary --> S1
Primary --> S2
style S2 fill:#f96,stroke:#333This affects large monorepositories, where a replication event can take a long time due to a quickly moving repository. Effectively, this means that read distribution will not work for the bulk of the workday, causing all reads to go to the primary.
We can fix this problem in praefect