Prometheus endpoint MVC
Problem to solve
GitLab can access multiple prometheus servers, at the environment, project, and potentially group levels. While we do expose some interface for accessing these components, it is still very much a WIP.
We should consider building a Prometheus API interface, which not only we could use, but also another service like Grafana. This could provide an easy abstraction layer to the complexity of the underlying Prometheus layout, as well as leverage features like the Kubernetes API proxying service.
This would simplify our Rails logic as well as provide turnkey integration with Grafana.
We could also push more of the charting logic onto the front end, instead of having to make backend changes for any changes to what types of queries or charts we want to support.
Focus on the first MVC of providing a Prometheus API compatible endpoint that we can use internally,
as well as a user could connect something like Grafana to (moved to #60099).
Expose a Prometheus API endpoint within GitLab, at the per-environment. We may be able to build on top of the endpoint that we use for validating valid PromQL queries.
- Note, performance with the Sidekiq caching latency is not great. We should try that as a boring solution, but we may need to consider Go for performance at some point.
At first, accept simple authentication at the project level,
which can be set in Grafana for authentication. We should also ensure whatever authentication we use for authentication between the frontend and backend also works. It would be nice if we displayed both the project URL and authentication token in the settings screen
Define the endpoint per environment, which should map to a single Prometheus server (today)
Permissions and Security
The most common use case for externally authenticating to this endpoint would be to put it in another monitoring system, like Grafana. Given that use case, supporting tokens like PAT's doesn't really make sense because it provides overly broad authentication. For example if a user got a hold of the PAT in Grafana, they could then assume the same privileges as the user who originally configured it. Having a token just for this Prometheus endpoint would reduce the blast radius, and address the most common use case we expect to see safely. Another option to strongly consider supporting is a Deploy Token. While it is generally intended to support the deploy use case, it provides read-only access to a variety of resources within the project. (Repo, containers, etc.) Re-using this would avoid requiring us to make yet another token type within GitLab.
In this issue, we decided to limit this API to signed in users. Look at #60099 for token authentication.