Investigate better integration with gitlab-ci-pipelines-exporter

Proposal

We've identified that some expensive usage of the API comes from the use of a 3rd party product, gitlab-ci-pipelines-exporter.

See https://gitlab.com/gitlab-com/support/internal-requests/-/issues/5609 (internal).

We actually encourage the use of this in our documentation (here) and a number of customers rely on it to ingest metrics on their pipelines into Prometheus.

By default it creates nearly 1 million requests per day to an expensive endpoint on our API. Combined traffic with this user agent amounts to 42 million per day and nearly 1b SQL queries.

There are a number of things we can do to mitigate this:

• Lower the default limit in the project to 1 request/s (a 90% drop).
• Improve the current pipelines API endpoint so that it's faster and more performant.
• Implement an API endpoint specifically for these metrics so that provision can be optimized, cached and constant paging isn't necessary.
• Build a configurable pipeline exporter directly into GitLab.
• Other?

Customers have given interesting feedback (see linked issue) on our existing Prometheus metrics for pipelines. Some don't want to store these metrics on a VM so a Webhook isn't suitable.