Gitaly on file-store-04 is down
There may be other shards down too:
stanhu@web-01-sv-gprd.c.gitlab-production.internal:/etc/gitlab$ telnet file-04-stor-gprd.c.gitlab-production.internal 9999
Trying 10.221.2.104...
telnet: Unable to connect to remote host: Connection refused
I noticed this by getting Gitlab::Git::CommandError (14:Connect Failed):
errors.
We need better alerting and monitoring for this.