Shared Runner Managers are possibly underprovisioned
Shared runner managers need to be scaled up. On one host, for example, tasks spend 1.2s waiting for a CPU for every second that they run.
CPU is pinned near 100% for much of the day.
Load average 15 is around 2 per core.
# Details
`node_schedstat_waiting_seconds_total` on `shared-runners-manager-3.gitlab.com` up at 125%.

https://prometheus.gprd.gitlab.net/graph?g0.expr=max%20by%20(fqdn)%20(rate(node_schedstat_waiting_seconds_total%7Bfqdn%3D%22shared-runners-manager-3.gitlab.com%22%2C%20type%3D%22ci-runners%22%7D%5B1h%5D)*100)%0A%0A&g0.tab=0&g0.stacked=0&g0.range_input=2w
https://dashboards.gitlab.net/d/ci-runners-main/ci-runners-overview?orgId=1
cc @dawsmith @tmaczukin
issue