Draft: Add extra alerts plus fix existing alerts for Dedicated Hosted Runners (DHR)

Zoe Braddockrequested to merge
alerting-fixes-for-dhr into master

https://gitlab.com/gitlab-com/gl-infra/gitlab-dedicated/team/-/issues/10355

added HostedRunnersServiceCapacityUtilizationHighSingleShard as a non-paging alert.

fixed HostedRunnersServicePendingBuildsApdexSLOViolationSingleShard and HostedRunnersServiceQueuingQueriesDurationApdexSLOViolation by changing queuing queries duration to monitor gitlab_runner_job_queue_duration_seconds_* (which works and returns data) instead of gitlab_ci_queue_retrieval_duration_seconds_* (which is empty)

Carried over from !9680 (closed), where I hopelessly borked the pipelines and cache so I made a fresh branch and tried again.

Moved to !9793 (merged) because I wasn't willing to deal with the fallout of being 150+ commits behind master :D

Edited by Zoe Braddock

Merge request reports