Exporter: Implement scrape of PostgreSQL physical-level query metrics (pg_stat_kcache: real disk IO, CPU usr, sys, etc.)
pg_stat_kcache is installed currently, but we don't collect metrics to Prometheus yet – it is worth doing it, to include it to query analysis routinely.
This can especially be important when we aim to reduce CPU or disk IO load, because pg_stat_kcache gives understanding which queries contribute to resource utilization most at physical level (pg_stat_statements doesn't do it, it works at "higher" level, not dealing with physical resource metrics).
Old issue explaining general ideas of query analysis improvements: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/12197
We aded pg_wait_sampling metrics recently: https://gitlab.com/gitlab-com/gl-infra/reliability/-/issues/24607, this is a similar task.
Once we have them all in Prometheus, we will be able to build a new powerful dashboard for query analysis.