Move ci_*_builds metrics out of gitlab-exporter
We're looking to sunset gitlab-exporter
in #288043 (closed).
NOTE: This epic used to suggest that we drop gitlab-exporter entirely. We have come to understand that this might not be the best solution or only option. The epic has been updated to reflect that.
There are a number of CI related metrics exported from it that we either need to retire or move:
ci_created_builds
ci_pending_builds
ci_runner_builds
ci_running_builds
ci_repeated_commands_builds
ci_stale_builds
The goal of this issue is to:
- Identify which of the mentioned metrics are still in use. If they are still in use, they should be documented. If not, they should be removed.
- If the metrics are in use, move them out of the exporter. This could happen in various ways, some suggestions below.
Use of the metrics
The proxies I have used for "metric is in use" so far were them being accessible in Thanos or referenced in the runbooks repo.
- Thanos: all of these appear to be exported to Thanos.
- Runbooks: they seem to be used in alerts or dashboards, e.g.: https://gitlab.com/search?utf8=%E2%9C%93&snippets=false&scope=&repository_ref=master&search=ci_pending_builds&group_id=6543&project_id=1148549
Are there other uses of these metrics not covered above?
Migration of metrics
Our current approach to metrics collection is to have applications export their own metrics if possible. Could these be exported by https://gitlab.com/gitlab-org/gitlab-runner instead?
If not, could these be exported by the main Rails app instead, similar to the suggestions in #290747 (closed)?