Implement the repository worker
As part of &8911, we're introducing a write-ahead logging in Gitaly for repositories. Every write going into a repository is first written into the log and then applied from the log to repository. The log needs a dedicated worker to manage it as log application can happen outside of RPC context for example when recovering from crashes or the client that queued the write timed out. Each repository has a single worker that processes all writes into the repository, the other goroutines queue work for it to process.
For each incoming write, the worker:
- Verifies the references apply
- Append the write to the write-ahead log
- Apply the write into the repository
As part of &8911, there will be further issues to integrate with the rest of Gitaly to ensure all writes to a repository flow through the worker. This issue tracks the minimal work to implement the worker so we can further iterate on it. We should:
- Deliver a minimal implementation of the worker.
- For now, we can only focus on reference changes and ignore other logged writes.
- Write-ahead log should be stored in key-value store, Badger looks like a good option.
- Optimizations are left for later iterations.
- Have tests that ensure correctness of the worker.
- Have benchmarks in place so we can start optimizing.
With these in place, we can optimize the worker further and ensure it is perfromant enough. We should get to the following point:
Source: https://excalidraw.com/#json=HT1O9ASmZiKP6sVB9Zry5,1DxpXFPj9eFXYF3ZA33aaA