GET Monitoring: add better monitoring for GitLab Shell
In !4428 (merged) and #82 (closed), we added rudimentary SLO monitoring for the GitLab Shell service.
As present, this relies on GRPC messages to Gitaly/Praefect as a proxy for service health. This is suboptimal.
In gitlab-org/gitlab-shell!496 (merged), gitlab-org/gitlab-shell#121 (closed), @igor.drozdov added metrics to GitLab Shell, but at present, this is not being scraped on GitLab.com. See https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/13834 for more details.
Once we have better metrics from GitLab-Shell, we should use those to improve out monitoring.
Note: There are TODO: statements in the runbooks project referencing this issue.
Edited by Andrew Newdigate