Running in HA mode (Praefect) generates a bundle per node
While testing cloning with bundles locally, I noticed that when running in HA mode (Praefect + 3 Gitaly nodes), one bundle is generate per node, because of the replication logic that calls the same RPC (PostUploadPackWithSidechannel
) on each node. This is very inefective.
There are some solutions, but they each have drawbacks.
- We could only set a
go_cloud_url
(thus enabling the generation of bundles) config on the configuration of the primary node only. That would in effect make sure only the primary node generate bundles.- However, I am still not well versed into how failover works, but if we have automatic failover for Gitaly nodes, this solution is bad because the new primary node won't have the
go_cloud_url
config set. - The bundle path has the storage name in it (see here). So currently, a Gitaly node is unable to know if a bundle created by another Gitaly node exists for a given repository. We would need to make it so that a bundle path is common across all Gitaly nodes, regardless of the storage name.
- However, I am still not well versed into how failover works, but if we have automatic failover for Gitaly nodes, this solution is bad because the new primary node won't have the
- Move the bundle-generation into Praefect
- Since Praefect coordonnates the different Gitaly nodes, it would have all the knowledge it needs to make sure to create a bundle one once per
git clone
orgit fefch
. - If I understood correctly, I think the goal of the Raft initiative is to move away from Praefect. If that is the case, this solution is probably a no-go then.
- Since Praefect coordonnates the different Gitaly nodes, it would have all the knowledge it needs to make sure to create a bundle one once per
Those are the solutions I have so far. Any feedback or suggestions welcome!
Edited by Olivier Campeau