Potential multi-tenant / multi-user model

Description

As we continue to expand our integration with Prometheus, we will need to bring this to GitLab.com as well as support increased scaling. As one example, we are already planning to be able to stand up a Prometheus server for each project and each environment, in the future. (gitlab-ce#27521)

As part of this, one would expect to have users need to connect to the Prometheus server in certain conditions. Perhaps they need to validate the scrape targets, or perform some custom PromQL queries that GitLab (or their monitoring solution) isn't set up presently to do.

It would also be a significant benefit if we could bring this to GitLab.com, so users there could demo the power of our Prometheus integration.

Proposal

If the following requirements are true:

1. End users will need to be able to access the Prometheus console
2. We want to support Prometheus integration in GitLab.com (or larger customer deployments)
3. We will be launching multiple Prometheus servers, e.g. per project / environment

We will need some way to both route and control access to these servers. A natural spot to do this, would seem to be upstream to solve both of these challenges.

One way to do this, would be to leverage the OAuth 2 Proxy concept that we have as a baseline. This could provide two key benefits:

1. It would reverse proxy the traffic directed towards the Prometheus server, meaning that all inbound traffic to it would be restricted to a well known set of IP's.
2. It could perform access control, ensuring that the user who is logged in via OAuth has the proper access rights to a specific Prometheus server.
3. Provide an audit log of any access.