Skip to content

Unauthenticated access to victims Grafana datasources through /grafana/proxy endpoint

HackerOne report #1566306 by joaxcar on 2022-05-11, assigned to GitLab Team:

Report | How To Reproduce

Report

Intro

I have found a way to make unauthenticated arbitrary (authenticated as victim) requests to an integrated Grafana servers /:datastore_id/proxy/:path endpoint. What this means is that an unauthenticated user can send API requests to any Grafana datasource which have proxy access enabled. Everything from Prometheus to SQL servers.

I must admit that I am not too familiar with Grafana so take that in consideration if I make mistakes in my description.

I will make an initial write-up here, first describing the root cause and a simple POC. I will then add a more detailed analysis later on as a comment as to not bloat the initial report too much.

Grafana integration

A project maintainer/owner have the option to turn on a Grafana integration to allow users to reference Prometheus graphs in GitLab Markdown (see documentation.

To do this the maintainer will have to enter a Grafana sever URL and an Admin Grafana API token. This is important, the documentation and implementation requires a token with admin access to Grafana.

When the integration is set up a user can add specially crafted "share" links in GitLab markdown and the graph will render in the rendered markdown.

All this is made possible by three calls made from GitLab backend to the Grafana server and handled by /lib/grafana/client.rb

###  [@]param uid [String] Unique identifier for a Grafana dashboard  
    def get_dashboard(uid:)  
      http_get("#{[@]api_url}/api/dashboards/uid/#{uid}")  
    end

    # [@]param name [String] Unique identifier for a Grafana datasource  
    def get_datasource(name:)  
      http_get("#{[@]api_url}/api/datasources/name/#{Addressable::URI.encode_component(name)}")  
    end

    # [@]param datasource_id [String] Grafana ID for the datasource  
    # [@]param proxy_path [String] Path to proxy - ex) 'api/v1/query_range'  
    def proxy_datasource(datasource_id:, proxy_path:, query: {})  
      http_get("#{[@]api_url}/api/datasources/proxy/#{datasource_id}/#{proxy_path}", query: query)  
    end

Generating a graph from a link will involve first making a call to fetch a dashboard, then a subsequent call to fetch a datasource and in the end making a request to the proxy endpoint. All this is restricted in multiple ways. /lib/grafana/validator.rb ensures that calls are only made to Prometheus datasources and that the graph data is only of certain kinds.

The problem

This is all great. But the proxy_datasource call is also exposed directly through the endpoint https://gitlab.com/group/project/-/grafana/proxy which is accessible by any user with access to view the project. This includes unauthenticated users on open projects. I believe that it is the /app/controllers/projects/grafana_api_controller.rb controller that is responsible for this.

Any request to this endpoint will end up running the

http_get("#{[@]api_url}/api/datasources/proxy/#{datasource_id}/#{proxy_path}", query: query)

and adding

    def request_params  
      {  
        headers: {  
          'Authorization' => "Bearer #{[@]token}",  
          'Accept' => 'application/json',  
          'Content-Type' => 'application/json'  
        },  
        follow_redirects: false  
      }  
    end

Thus allowing the user to make calls to Grafana with the configured Admin API key. Making calls this way does not enforce any restrictions on what datasources are targeted. This makes the impact of an attack warring greatly. But an important note here is that the datasource ids are just an integer starting from 1. So any Grafana instance is easily enumerated for datasources.

The proxy endpoint allows the attacker to make API calls to any datasource with proxy access. This includes everything from Prometheus, Elasticsearch and even SQL databases.

The proxy_path in the code snippets above is everything after /grafana/proxy/:id in the request. And it is the API path to be called on the datasource server. For Prometheus servers this means that we can access the API with the proxy_pathlike /api/v1/ANYTHING but the API paths will differ depending on what datasource is targeted.

Impact and CVSS

In my CVSS assesment of this issue I have put Attack complexity as low. This is probably up for discussion but I want to describe my thoughts on this. GitLabs CVSS calculator states this on attack complexity

AC:H Successful attack depends on conditions beyond the attacker's control.

  • A certain setting has to have a non-default value to make the attack possible

This can read as any settings that have been modified would rise the complexity to High. But the official CVSS specs describes the same as

If a specific configuration is required for an attack to succeed, the Base metrics should be scored assuming the vulnerable component is in that configuration. The Base Score is greatest for the least complex attacks.

I would argue that the complexity here is LOW as there is no special non default settings needed to exploit this feature: when the feature is activated. Activating the Grafana integration by following the official guide should, I believe, be considered default settings. The feature can of course not be exploited if not activated, but if activated and configured correctly it is vulnerable.

An easy way to scan for vulnerable projects is to append this to any project path

/-/grafana/proxy/1/api/v1/labels?a=a

If the response is a 500 error the integration is not active. If the response is a 204 response the integration is active.

Steps to reproduce

This should be even easier to reproduce with a docker image, but I had some problems with the SSL certificate when setting it up. I ended up using a free Grafana cloud account with no data on it just for the POC. To follow along you will need a Grafana server with access to create an Admin API key.

From the official GitLab docs

First create a PUBLIC project, then follow this guide from the docs

To set up the Grafana API in Grafana:

  1. In Grafana, generate an Admin-level API Token. https://grafana.com/docs/grafana/latest/http_api/auth/#create-api-token
  2. In your GitLab project, go to Settings > Monitor and expand the Grafana authentication section.
  3. To enable the integration, check the Active checkbox.
  4. For Grafana URL, enter the base URL of the Grafana instance.
  5. For API Token, enter the Administrator API token you just generated.
  6. Click Save Changes.
And then

After setting up the integration now make sure that there is a Prometheus datasource in the Grafana server. If not configure on following https://grafana.com/docs/grafana/latest/datasources/add-a-data-source/ and take a note of the datasource ID.

If you do not find the ID use the Grafana API in this Curl command

curl -H "Authorization: Bearer TOKEN" https://example.grafana.net/api/datasources

to list the datasources and find one to target.

Then open a new browser and make sure to be logged out. Make a request like this

https://gitlab.com/GROUPNAME/PROJECTNAME/-/grafana/proxy/DATASOURCEID/api/v1/labels?a=a

You will be redirected to where you where before the call. This is the 204 response. Now make the exact same recuest again

https://gitlab.com/GROUPNAME/PROJECTNAME/-/grafana/proxy/DATASOURCEID/api/v1/labels?query=a

And there should be a JSON response from the Prometheus server.

The labels call is just to prove that it works. I will discuss potential leaks later. The query=a search param is just a cache buster as the responses are cached on the parameters. So if the response does not alter just change the query param.

Example

If the POC is hard to pull of I have a public project that you can use to test on. It is public so you can make the requests unauthenticated.

https://gitlab.com/project_5396771_bot/grafana-integration/-/grafana/proxy/10/api/v1/labels?query=a

If you are redirected back, click the link again. The first request will fill the cache. You should see a JSON response from my Prometheus server.

What is the current bug behavior?

Any user can make calls to the /grafana/proxy endpoint and have GitLab execute arbitrary API calls to a victims Grafana server

What is the expected correct behavior?

The grafana/proxy endpoint should not allow for arbitrary, unvalidated and unauthenticated requests

Output of checks

This bug happens on GitLab.com

Impact

Unauthenticated access to all datasources of a victims Grafana server. This could potentially include ALL Prometheus data, ALL elastic data, full access to connected databases and more.

How To Reproduce

Please add reproducibility information to this section: