Stop supporting duplicate storages in Gitaly's configuration

In our setup guide for Praefect, we say:

Instead of configuring gitaly['configuration'][:storage] uniquely for each Gitaly node, it is often easier to have include the configuration for all Gitaly nodes on every Gitaly node. You can do this because the Praefect virtual_storage configuration maps each storage name (such as gitaly-1) to a specific node, and requests are routed accordingly. This means every Gitaly node in your fleet can share the same configuration.

We're recommending naming multiple storages in Gitaly with different names and pointing them to the same storage. While this generally works, it's not really a correct configuration. Each storage should point to a unique directory. It's surprising that multiple storages may point to the same directory.

As things are, this would cause issues with Gitalys upcoming storage changes. For example, Gitaly would attempt to open the embedded database on the storage multiple times.

Gitaly with Raft will use all configured storages. Each storage will be allocated a unique ID, and will be advertised to the other nodes in the cluster. Having multiple storages pointing to the same directory would cause conflicts.

This also seems to cause issues in Gitaly already as things are. For example, our background repository maintenance routine doesn't seemingly differentiate between the duplicate storages, and schedule maintenance multiple times on the same data due to that.

It could be possible to workaround this by only taking the distinct set of storage paths and working on those internally, and routing the calls for the duplicated storages to the same storage. It does seem a bit unnecessary though, and we shouldn't be recommending such configurations to begin with.

Let's remove the recommendation from the documentation, and deprecate configuring the same storage directory with multiple names. This will require customers to update their configuration. As this is a breaking change, we can probably only enforce this in %17.0, and if transactions are opted into prior to that.

/cc @andrashorvath @mjwood I think this is something we should ideally do soonish. We're currently recommending configuring Gitaly in a manner which is not entirely correct. Adding this to &10625 (closed) given this is affecting negatively the work there.