Praefect metrics queries should not hit the database
Praefect's prometheus metrics endpoint produces some metrics that are derived from database queries. These queries can be expensive, especially when interlaced from many praefect servers being scraped in parallel, hitting the same single database primary in parallel. This was a contributing factor to the gitlab.com incident gitlab-com/gl-infra/production#2918 (closed).
This is a prometheus anti-pattern. As a GitLab operator scales praefect out, which as a stateless server process they may well do, increasing pressure is put on the database as every praefect node is scraped by prometheus.
A possible solution is to ensure that all metrics scraped by prometheus don't require DB access. Separate out those that do into a separate process, that can run as a singleton prometheus exporter. Alternatively, we could use the SQL exporter to yield metrics from given database queries.
We do something similar with gitlab-rails. DB-derived metrics are scraped by the gitlab exporter, which runs as a singleton against a replica. If every application server instance hit the DB on every prometheus scrape, we would grind to a halt.