Move gitlab_database_* metrics out of gitlab-exporter
We're looking to sunset gitlab-exporter
in #288043 (closed).
NOTE: This epic used to suggest that we drop gitlab-exporter entirely. We have come to understand that this might not be the best solution or only option. The epic has been updated to reflect that.
There are a number of Postgres related metrics exported from it that we either need to retire or move:
gitlab_database_bloat_*
gitlab_database_rows
gitlab_database_stat_table_*
The goal of this issue is to:
- Identify which of the mentioned metrics are still in use. If they are still in use, they should be documented. If not, they should be removed.
- If the metrics are in use, move them out of the exporter. This could happen in various ways, some suggestions below.
Use of the metrics
The proxies I have used for "metric is in use" so far were them being accessible in Thanos or referenced in the runbooks repo.
- Thanos: all of these appear to be exported to Thanos.
- Runbooks: only
gitlab_database_rows
seems to be used in alerts or dashboards: https://gitlab.com/search?utf8=%E2%9C%93&snippets=false&scope=&repository_ref=master&search=gitlab_database_rows&group_id=6543&project_id=1148549
Are there other uses of these metrics not covered above?
Migration of metrics
We already have a database_sampler
running as part of the main Rails app. It currently exports the gitlab_database_connection_pool_*
family of metrics. Should the above metrics move here?
Another suggestion was to consider extracting all of these into a Sidekiq job. The problem is that we need to be careful not to create additional database pressure by querying for these metrics too often. The Prometheus scraper interval was suggested to be too frequent for this. Moreover, it would be redundant and costly for every Puma instance to re-sample the database row counts, for instance. It would be better to collect these metrics at longer intervals, and hold onto it until the next scrape happens rather than querying them on-demand.
To actually export these metrics, at least two options come to mind:
- export them via the existing
/-/metrics
endpoint in Rails - add a new endpoint