Support for running multiple gitaly servers
This is a continuation of the gitaly sharing research issue: https://gitlab.com/charts/gitlab/issues/399
We put the groundwork in place in https://gitlab.com/charts/gitlab/merge_requests/271, that changed the storage related settings to be compatible with additional replicas in a statefulset, but we left the replicas hardcoded to 1, because we need some additional config work to support connecting to multiple.
From the groundwork MR:
I think we can provide the multi-gitaly in a followup by allowing the user to provide the the storage-key to gitaly address mapping manually, with added support for using the ordinal/index in the mapping instead of an address, and saving that information into its own entry in a configmap that can be mounted by gitaly and unicorn/sidekiq. The gitaly pods would reference the file to determine what their storage key name was, and unicorn and sidekiq would use the file to configure their storage.