Introduce OAuth 2 Proxy to help Prometheus authenticate users, move behind nginx

Description

Presently Prometheus is running on TCP port 9090, but does not support HTTPS. When SSL is turned on for the GitLab domain, HSTS is enabled which breaks Prometheus. (See gitlab-ce#27069)

We do not expect to see much direct usage of Prometheus, as the web console is designed to run manual queries. Instead, we expect to see primary consumption of Prometheus data served through GitLab itself (see gitlab-ce#26905), or a dashboard service like Grafana. However we would still like to preserve access to the Prometheus console.

As such we need to find a way to be able to allow access to Prometheus when SSL has been enabled, as it does not support HTTPS or offer authentication.

Proposed Solution

We can use nginx to terminate HTTPS and act as a reverse proxy on the main GitLab URL, which will solve the HTTPS issue. Next, we send that traffic to an OAuth2 Proxy, which will be able to restrict access to the Prometheus console to only authenticated GitLab users.

Browser ---HTTPS---> NGINX ---HTTP---> OAuth2 Proxy ---HTTP---> Prometheus

While this flow has a few steps, it will allow us to offer access to the Prometheus console over SSL and with GitLab authentication. Since the OAuth2 Proxy would be packaged in Omnibus, we can script the OAuth2 configuration between the two components.

The URL to access Prometheus could be configured as https://gitlab.customer.com/-/prometheus to avoid colliding with any users or groups.

This solution offers a couple benefits:

  • No new DNS names or SSL certificates are required, and no further ports exposed outside of localhost.
  • Suitable to be enabled by default, as no manual steps are required.
  • Lays foundation for further improving the Prometheus server by making it more internet friendly (HTTPS/Auth)

It also has a few negatives:

  • I don't believe there is a current way to restrict OAuth2 access to specific groups of users, say Masters or Admins. This could be a problem for some use cases of GitLab with public sign up, however they can always disable Prometheus and access it locally.
  • This will take slightly longer to implement than some of the simpler direct nginx configuration changes, as there is more work involved.
    • If we do select this path, we can always document how to temporarily add a custom nginx config file to reverse proxy it on another port or DNS name.
  • It will be strange accessing the Prometheus console from a URL off the GitLab domain, as well as seeing an OAuth2 authentication dance the first time through.

Rationale for Selection

Given that we do not expect the Prometheus console to be used directly often, we should not optimize for this workflow over other higher value aspects of the combined GitLab/Prometheus solution. Namely:

  • Whether it can be enabled by default without effort any intervention
  • The ability to see the value of Prometheus in GitLab workflows.