Remove storage shards from deploy nodes' Praefect configuration
On https://gitlab.com/gitlab-com/gl-infra/production/-/issues/2294 we had an issue where the deploy-cny-01-sv-gprd
was attempting to process Praefect replication jobs. Because we don't configure the secrets for the deploy node, this was resulting in authentication errors. We have Praefect installed in the deploy node because that allows us to run Praefect SQL migrations from it, but there's no need for the process itself to be running, since migrations are executed from a direct invocation of the praefect sql-migrate
command.
We stopped the Praefect process on our deploy nodes, but we can also go further and remove the storage shards configuration from those nodes completely, to ensure that the Praefect process there can't communicate with the Gitaly shards. This also has the implication that if the Praefect process attempts to start it'll end up in a startup loop when it finds an invalid shard configuration (admittedly it's not pretty to have the process in an error loop, but we don't log, monitor or alert on the deploy node praefect, and a startup loop shouldn't consume enough resources to be a problem, so it should not cause us any noise).