Add Runner wait times to self-monitoring
Problem to solve
Runner wait times for a shared fleet are an important metric to know if you're trying to figure out if you have enough runners provisioned to meet the demands of your users in your instance. There are some different ways you can get at this information today, but none are easy or integrated with self-monitoring.
https://gitlab.com/gitlab-org/gitlab-ce/issues/67096 captures many more metrics and is related to this topic, but we don't require all of those to get started.
Other issues related to figuring out Runner shared fleet performance on gitlab.com may provide some context for the kind of information needed:
- gitlab-com/www-gitlab-com#5341 (closed)
Add wait time as a metric to Prometheus and ship a Grafana chart. If the the Instance Administration project (https://gitlab.com/groups/gitlab-org/-/epics/23670) is open by the time we implement this, we could create a customized "CI Dashboard" in that project.
Permissions and Security
This should only be available to instance administrators.