Graceful degradation when single shard is unavailable for Gitaly
Gitaly on GitLab.com is a SPOF, while there is a [plan to rollout Gitaly cluster](https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/363) this won't address free plans and the need to occasionally reboot servers or [when servers are rebooted due to cloud provider maintenance](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/4699). This issue is to explore how we can gracefully degrade a Gitaly shard when it is unavailable. Some specific scenarios like: - If the corresponding Gitaly server is down, through the web UI - Project page - Branches - Tags - Git SSH/HTTPS commandline For specific failures and the current user experience see https://gitlab.com/gitlab-org/gitlab/-/issues/332339#note_661505055
epic