Skip to content

Praefect to support multiple primaries

We've been having discussions around Praefect's ability to handle all gitaly nodes due to the increased cost of having a praefect node per rails storage.

In Praefect's current state, it can only handle writing to one primary, enforced by the top level toml attribute virtual_storage_name. However, we can change it so praefect can support multiple [[virtual_storage]] instead of just one

# praefect.config.toml
listen_addr = "tcp://praefect.internal"

[[virtual_storage]]

name = "nfs-file21"

  [[node]]
  storage = "praefect-git-0"
  address = "tcp://praefect-git-0.internal"
  primary = true
  token = 'token1'

  [[node]]
  storage = "praefect-git-1"
  address = "tcp://praefect-git-1.internal"
  token = 'token2'

  [[node]]
  storage = "praefect-git-2"
  address = "tcp://praefect-git-2.internal"
  token = 'token3'

[[virtual_storage]]

name = "nfs-file22"

  [[node]]
  storage = "praefect-git-1"
  address = "tcp://praefect-git-0.internal"
  primary = true
  token = 'token1'

  [[node]]
  storage = "praefect-git-2"
  address = "tcp://praefect-git-1.internal"
  token = 'token2'

  [[node]]
  storage = "praefect-git-0"
  address = "tcp://praefect-git-2.internal"
  token = 'token3'

To rails, praefect still looks like a single gitaly node that can serve multiple storages:

git_data_dirs({
  "nfs-file21" => { "gitaly_address" => "tcp://praefect.internal" },
  "nfs-file22" => { "gitaly_address" => "tcp://praefect.internal" }
})

The changes in praefect would not be too complex. For each RPC request we would:

  1. check the incoming storage_name
  2. find the virtual_storage that matches that name
  3. rewrite the storage_name to the primary node that is part of that virtual_storage

step 2 is the one change we would need.

Edited by James Ramsay (ex-GitLab)
To upload designs, you'll need to enable LFS and have an admin enable hashed storage. More information