Attackers can snoop on any project's prometheus metrics and environments

The following issue was reported through HackerOne.

Weakness: Insecure Direct Object Reference (IDOR) Link: https://hackerone.com/reports/406578

Details: The Projects::Prometheus::AlertsController#create and Projects::Prometheus::AlertsController#update allow an attacker to provide an environment_id and prometheus_metric_id that doesn't belong to their project. When an alert persists that doesn't belong to their project, they can trigger it using the Projects::Prometheus::AlertsController#notify endpoint. This will leak information about the victim's environment, project, and their Prometheus metric.

Proof of concept

For the vulnerability to be exploitable, the user requires the prometheus_alerts feature. I manually enabled this on my GitLab EE instance by monkey patching the feature_available? helper on the Project model. It can also be made available by acquiring the appropriate GitLab license. To reproduce the vulnerability, follow the steps below.

As the victim

  • sign in to any user
  • create a project and manually enable Prometheus monitoring (can point to a host that's offline) through the Integrations section
  • create a new metric for the Prometheus monitoring
  • create an environment for the project

As the attacker

  • sign in as a user that can create projects, make sure that the prometheus_alerts feature is available
  • create a new project
  • send a POST request to the /:namespace/:project/prometheus/alerts.json endpoint, as such:

Request

POST /:namespace/wow/prometheus/alerts.json HTTP/1.1
...
X-CSRF-Token: ...
X-Requested-With: XMLHttpRequest
Cookie: ...
Content-Type: application/json
Content-Length: 78

{"operator":">","threshold":1,"environment_id":1,"prometheus_metric_id":1}
  • in the request above, change the environment_id and prometheus_metric_id to the objects that were created by the victim, the server will return a response similar to the one below:

Response

HTTP/1.1 200 OK
Server: nginx
...

{"id":6,"title":"Metric title","query":"rate()","threshold":1.0,"operator":"\u003e","alert_path":"/:namespace/:project/prometheus/alerts/1.json?environment_id=1"}
  • in this response, some confidential information can be obtained: the title and query of the metric that was referenced
  • now, send a POST request to the /:namespace/:project/prometheus/alerts/notify endpoint, as such:

Request

POST /:namespace/:project/prometheus/alerts/notify HTTP/1.1
...
Content-Type: application/json
Cookie: ...

{"alerts":[{"labels":{"gitlab_metric_id":1}}]}
  • this will send an email to the masters / owners of the /:namespace/:project project with the environment name and the full path of the project that has the referenced gitlab_metric_id. Note: this information leak ONLY works when the attacker only referenced an environment_id that belongs to another (private) project. It does NOT work when a gitlab_metric_id was referenced that doesn't belong to the project. This is because the asynchronous job becomes a no-op when the gitlab_metric_id doesn't belong to the project referenced in the URL (it's like someone foresaw this issue...). Because the attacker isn't authorized to create Prometheus metrics / alerts for the project, this will never work. This endpoint does have another vulnerability though, see "Bonus vulnerability" at the bottom of the report.

What I'm currently unable to figure out is whether a (Kubernetes) cluster with Prometheus configured is provisioned with the alerts that belong to another project. If that's the case, the attacker can snoop on any metric by creating an alert that will always trigger (using lt 0, gt 0, and eq 0). This may increase the severity of the vulnerability. However, I didn't set up a Prometheus configuration on Kubernetes, so I haven't confirmed this.

Caveat: the database contains a UNIQUE constraint on the prometheus_metric_id column. This means that an attacker can only snoop on metrics that don't have an alert set up yet. However, this doesn't apply to the environment_id. The attacker can reference any environment as often as they want to.

Bonus vulnerability (CSRF)

The Projects::Prometheus::AlertsController#notify endpoint disables any CSRF protection. This means that an attacker can trigger a fake alert by letting a victim visit a web page that contains the following form (note: the attacker needs the namespace, project name, and metric ID before this can be exploited):

<form method="post" action="http://gitlab-instance/:namespace/:project/prometheus/alerts/notify">
  <input name="alerts[][labels[gitlab_metric_id]]" value="1" />
  <input type="submit" />
</form>
<script>document.forms[0].submit();</script>

Impact

Exploiting this vulnerability gives an attacker access to any (private) project name, environment name, metric query, and metric title, that has at least one unmonitored Prometheus metric. This may also result in the alerts being provisioned on a Kubernetes cluster with Prometheus monitoring enabled.

The CSRF vulnerability doesn't seem to have any immediate impact for GitLab, but I'd think it's rather annoying that a project owner / master were to receive false-positive alerts notifications from GitLab.

Timeline: 2018-09-07 20:48:48 +0000: @jritchey (comment) Hi @jobert ,

Thank you for submitting this report. We are currently investigating the issue. Due to our current workload, we will get back with you as soon as we can with an update.

Best regards, James

2018-09-07 23:55:57 +0000: @asaba (user assigned to bug [team-only])