Praefect to support multiple primaries
We've been having discussions around Praefect's ability to handle all gitaly nodes due to the increased cost of having a praefect node per rails storage.
In Praefect's current state, it can only handle writing to one primary, enforced by the top level toml attribute virtual_storage_name
. However, we can change it so praefect can support multiple [[virtual_storage]]
instead of just one
# praefect.config.toml
listen_addr = "tcp://praefect.internal"
[[virtual_storage]]
name = "nfs-file21"
[[node]]
storage = "praefect-git-0"
address = "tcp://praefect-git-0.internal"
primary = true
token = 'token1'
[[node]]
storage = "praefect-git-1"
address = "tcp://praefect-git-1.internal"
token = 'token2'
[[node]]
storage = "praefect-git-2"
address = "tcp://praefect-git-2.internal"
token = 'token3'
[[virtual_storage]]
name = "nfs-file22"
[[node]]
storage = "praefect-git-1"
address = "tcp://praefect-git-0.internal"
primary = true
token = 'token1'
[[node]]
storage = "praefect-git-2"
address = "tcp://praefect-git-1.internal"
token = 'token2'
[[node]]
storage = "praefect-git-0"
address = "tcp://praefect-git-2.internal"
token = 'token3'
To rails, praefect still looks like a single gitaly node that can serve multiple storages:
git_data_dirs({
"nfs-file21" => { "gitaly_address" => "tcp://praefect.internal" },
"nfs-file22" => { "gitaly_address" => "tcp://praefect.internal" }
})
The changes in praefect would not be too complex. For each RPC request we would:
- check the incoming storage_name
- find the
virtual_storage
that matches that name - rewrite the storage_name to the primary node that is part of that
virtual_storage
step 2 is the one change we would need.
Edited by James Ramsay (ex-GitLab)