Start tracking Jobs queueing time
We should start tracking jobs queueing time to figure out our SLA for jobs running on Shared Runners.
Given that we have fair scheduling we should consider to take that into account, where we create a histogram based on:
- how many jobs is given runner running for given project,
- how many jobs is given runner running for given namespace,
- what type of runner is processing that job (shared or specific),
If we graph that on Grafana, it should answer us:
- how quickly we can pick a job from the project that has none of other jobs running (this should be preferred ones),
- how long it takes us to process 10 jobs concurrently for the project,
- monitor the picking time to create an alerting about runner slowdown.
We should implement that in GitLab CE, as a Gitlab::Metrics.histogram
with labels.