Track CPU, DB and Gitaly time spent at Sidekiq jobs and present at Grafana
Currently we track CPU time on Prometheus (as histogram) in the server middleware. Though it seems that we do not have a proper Grafana dashboard for it at https://gitlab.com/gitlab-com/runbooks (no jsonnet definition).
On the other hand, DB time is currently only tracked in our logs via Gitlab::SidekiqLogging::StructuredLogger
through db_duration
and db_duration_s
, which are exposed to Kibana.
Gitaly timings are also only set for logs (lograge) through Gitlab::InstrumentationHelper.add_instrumentation_data
.
I imagine we could take advantage of having dashboards (and even SLO's) based on CPU, DB and Gitaly time spent for each queue. We could make use of the current labels (which include latency_sensitive
and boundary
information) to have a better insight on them, but in order to accomplish that we need to make sure we track all these at Prometheus.