Add extra alerts plus fix existing alerts for Dedicated Hosted Runners (DHR) plus use saturation alert generator for hosted-runners

Zoe Braddockrequested to merge
zbraddock/alerting-fixes-for-dhr into master

https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/10355

added HostedRunnersServiceCapacityUtilizationHighSingleShard as a non-paging alert.

fixed HostedRunnersServicePendingBuildsApdexSLOViolationSingleShard and HostedRunnersServiceQueuingQueriesDurationApdexSLOViolation by changing queuing queries duration to monitor gitlab_runner_job_queue_duration_seconds_* (which works and returns data) instead of gitlab_ci_queue_retrieval_duration_seconds_* (which is empty)

Carried over from !9717 (closed) because I wasn't willing to deal with the fallout of being 150+ commits behind master :D

Edited by Zoe Braddock

Merge request reports