Feature Request - CI/CD pipelines exporter without needing third-party software

Proposal

As of today, exporting CI/CD pipeline metrics for observability is mostly performed by using external tools such as gitlab-ci-pipelines-exporter.

While this works, relying on a third-party software to simply pull data from GitLab's APIs is not ideal for the following reasons:

  • Observability should fully built in GitLab, especially since data is already there and we already have a bundled Prometheus instance;
  • Third-party tools need a way to access APIs, which is usually done via a PAT, and this poses some more issues:
    • PATs must have an expiration date, non-expiring tokens have been deprecated with GitLab's recent releases and this means that instance admins would have to manually refresh that token from time to time;
    • Since PATs are personal, using someone's PAT to export instance-wide CI/CD metrics is a bad practice (e.g. the user could leave the company, a token leak can cause issues to his personal projects and so on). To mitigate this risk a dedicated user can be created (like a service account), but this will consume a license seat which basically means paying for the feature;
  • Needing a third-party tool, which in turn requires some manual configuration (e.g. generating and setting the token), defeats the purpose of GitLab automated install methods such as GitLab Environment Toolkit

I understand that exporting CI/CD pipeline metrics can be cpu intensive, so my idea would be for this feature to be enabled at either project level, group level or instance wide with a specific toggle that's disabled by default. This could also leverage the dedicated monitor machine that's been created by GitLab Environment Toolkit in order to reduce the load on the rails ones (which is exactly how I would configure gitlab-ci-pipelines-exporter).

So, to summarize, what I'm proposing is to essentially "adopt" gitlab-ci-pipelines-exporter and to include it directly in GitLab, without requiring any kind of access token to configure it, thus allowing instance administrators to have a fully fledged pipeline monitoring environment out of the box.