Investigate better integration with gitlab-ci-pipelines-exporter
Proposal
We've identified that some expensive usage of the API comes from the use of a 3rd party product, gitlab-ci-pipelines-exporter.
See https://gitlab.com/gitlab-com/support/internal-requests/-/issues/5609 (internal).
We actually encourage the use of this in our documentation (here) and a number of customers rely on it to ingest metrics on their pipelines into Prometheus.
By default it creates nearly 1 million requests per day to an expensive endpoint on our API. Combined traffic with this user agent amounts to 42 million per day and nearly 1b SQL queries.
There are a number of things we can do to mitigate this:
- Lower the default limit in the project to 1 request/s (a 90% drop).
- Improve the current pipelines API endpoint so that it's faster and more performant.
- Implement an API endpoint specifically for these metrics so that provision can be optimized, cached and constant paging isn't necessary.
- Build a configurable pipeline exporter directly into GitLab.
- Other?
Customers have given interesting feedback (see linked issue) on our existing Prometheus metrics for pipelines. Some don't want to store these metrics on a VM so a Webhook isn't suitable.