CSRF via k8s cluster-integration

⚠ Please read the process on how to fix security issues before starting to work on the issue. Vulnerabilities must be fixed in a security mirror.

HackerOne report #2286823 by imrerad on 2023-12-14, assigned to GitLab Team:

Report | Attachments | How To Reproduce

Report

Summary

Gitlab supports Kubernetes integration. It relies on the agentk agent that should be running in the cluster to be managed. source code
Simply put, the agent is a bidirectional inverting proxy, it connects out to the corresponding server component called kas. kas is responsible for proxying connections to the right agentk backend.
A request to be proxied looks like this:

Client to kas:
VERB https://kas.gitlab.com/k8s-proxy/subpath

this is translated from kas to the K8s API server:
VERB https://apiserver/subpath

To access the Kubernetes cluster, one may use

• the standard CLI tool kubectl; a kubeconfig is injected into the CI/CD pipelines automatically.
  Authentication relies on the standard Authorization header (and a CI job token).

• a web browser. This feature was added to support the Kubernetes dashboard.
  Authentication relies on cookies.

The authentication/authorization code of the kas side can be found here:
internal\module\kubernetes_api\server\proxy.go

Sensitive headers are removed by kas before forwarding them to agentk:

	// remove GitLab authorization headers (job token, session cookie etc)  
	delete(r.Header, httpz.AuthorizationHeader)  
	delete(r.Header, httpz.CookieHeader)  
	delete(r.Header, httpz.GitlabAgentIdHeader)  
	delete(r.Header, httpz.CsrfTokenHeader)  

The attack I'm reporting is started from a malicious Kubernetes cluster. It aims to target these sensitive headers and is based on a simple trick - HTTP redirects.
I found KAS relays the HTTP response to the caller blindly, including HTTP redirects as well. K8s had a relevant security issue in the past (cve-2022-3172), but there are some applications that have legitimate business needs wrt redirects (e.g. clusternet).

Golang's net/http client (so kubectl as well) strips sensitive headers when it encounters redirects:
https://github.com/golang/go/blob/master/src/net/http/client.go#L985

Browsers feature a long list of related security measures, like:

• cors, preflight requests, access-control-* headers
• content security policy
• samesite cookies (which is nowadays the default where it is not configured explicitly)
• mixed content
  And probably many more.

On the server side,

• the Rails application leverages CSRF protection based on the synchronizer pattern. The per-controller CSRF token feature is not enabled.
• kas as well, and next to that it accepts requests from allowed-listed origins only

Despite all of these, I managed to find a way to steal the X-CSRF-Token header and invoke arbitrary Rails methods from an external website on behalf of the victim user - in other words, Gitlab is vulnerable to CSRF.

This attack works in Firefox, but not in Chrome (maybe it works in Opera/Safari as well, I didn't test those).
The attack could also be launched against Gitlab.com, but in that case only by employees who can access the third party sites in Gitlab's CSP set. (As of today, it is : connect-src 'self' https://gitlab.com wss://gitlab.com https://sentry.gitlab.net https://new-sentry.gitlab.net https://customers.gitlab.com https://snowplow.trx.gitlab.net https://sourcegraph.com https://collector.prd-278964.gl-product-analytics.com snowplow.trx.gitlab.net wss://kas.gitlab.com/k8s-proxy/ https://kas.gitlab.com/k8s-proxy/; ).
On customer-managed Gitlab Enterprise installations CSP is not configured by default, most deployments are affected.

Steps to reproduce

Preparation. You will need:

• a self-managed Gitlab EE installation (gitlab.example.com in my setup below - I don't own this domain, but I got an entry in the hosts file of the OS)
• a linux host with docker installed (or just ensure agentk is able to connect to k8s-csrf via 127.0.0.1)
• and optionally one more host with
  • an external IP address with port 80 open
  • a DNS record that points to the external IP of your host (gcpexp.duckdns.org in this example)
• two Gitlab accounts (one for the attacker - imre.attacker in this example, one for the victim - imre.victim in this example)
• Firefox browser for the victim

As the attacker:

1 - create a new repo (imre.attacker/csrf-poc in this example)

2 - create a new file as .gitlab/agents/csrf-agent/config.yaml with the following content (adjust the project id):

user_access:  
  access_as:  
    agent: {}  
  projects:  
    - id: imre.attacker/csrf-poc  

3 - At Operate, Kubernetes clusters, connect a new one and save the token to the file glab-agentk-token-local

4 - Run ngrok and note the https URl for the next step:

ngok http 9999  

5 - Run the k8s-csrf script. This script responds as a kube cluster when the host header is 127.0.0.1 and serves the public facing payload otherwise.

 ./k8s_csrf.py --listen-port 9999 --external-url https://your-ngrok-url  

6 - Run the agent:

docker run --network host --rm -it -v /path/to/glab-agentk-token-local:/etc/agentk/secrets/token -e POD_NAMESPACE=agentk-nsname -e POD_NAME=agent-podname registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:v16.7.0-rc3 --kas-address=ws://gitlab.example.com/-/kubernetes-agent/ --token-file=/etc/agentk/secrets/token -s 127.0.0.1:9999  

7 - Verify the agent is healthy at Operate, Kubernetes clusters

8 - Grant developer rights on this project to the victim user

9 (optional) - on the other attacker host with a DNS record, run a reverse proxy:

docker run --rm -it -v ~/.mitmproxy:/home/mitmproxy/.mitmproxy -p 80:8080 mitmproxy/mitmproxy mitmproxy --mode  reverse:https://c0d0-78-131-112-151.ngrok-free.app --set block_global=false  

Note: this step is needed to mute a Firefox prompt that warns about sending a request from a https location to an insecure http location (http://gitlab.example.com). If TLS is configured for Gitlab EE, this can be skipped.

10 - Create a new environment at Operate / Environments, select a namespace and ignore flux, save. For the external URL

• use the ngrok url if you didn't complete step #9 (closed)
• use the public DNS if you did complete step #9 (closed)

I understand this is complicated. I'm attaching a video and feel free to ask questions if something is unclear.

As the victim, using Firefox:

0 - take a look at your SSH keys (to notice the difference at the end)

1 - visit the imre.attacker/csrf-poc repo

2 - at Operate / Environments, open the dashboard of the Kube cluster

this is enough to leak your CSRF token - it should show up in the k8s-csrf screen of the attacker

3 - open the external url of this environment

this is enough to get the attacker's ssh key added to the victim's account

4 - take a look at your SSH keys

Impact

See at the dedicated textbox.

What is the current bug behavior?

Cross site requests are accepted after the CSRF token was leaked.

What is the expected correct behavior?

The CSRF token should not be leaked. Kas should check the response and govern redirects, e.g. based on a new allowlist config option about the targets it should accept and relay.

Relevant logs and/or screenshots

See the videos and screenshots.

Output of checks

This bug affects GitLab.com as well, but in that case only certain partners can exploit it.

Results of GitLab environment info

System information  
System:  
Proxy:          no  
Current User:   git  
Using RVM:      no  
Ruby Version:   3.0.6p216  
Gem Version:    3.4.21  
Bundler Version:2.4.21  
Rake Version:   13.0.6  
Redis Version:  7.0.14  
Sidekiq Version:6.5.12  
Go Version:     unknown

GitLab information  
Version:        16.6.1-ee  
Revision:       9aa991a5ee9  
Directory:      /opt/gitlab/embedded/service/gitlab-rails  
DB Adapter:     PostgreSQL  
DB Version:     13.12  
URL:            http://gitlab.example.com  
HTTP Clone URL: http://gitlab.example.com/some-group/some-project.git  
SSH Clone URL:  git@gitlab.example.com:some-group/some-project.git  
Elasticsearch:  no  
Geo:            no  
Using LDAP:     no  
Using Omniauth: yes  
Omniauth Providers:

GitLab Shell  
Version:        14.30.0  
Repository storages:  
- default:      unix:/var/opt/gitlab/gitaly/gitaly.socket  
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

Gitaly  
- default Address:      unix:/var/opt/gitlab/gitaly/gitaly.socket  
- default Version:      16.6.1  
- default Git Version:  2.42.0  

Impact

The attacker could invoke arbitrary Rails methods of the victim, so the attack effectively allows taking over the Gitlab account of the victim.
The attack is not limited to POST requests only as Rails supports the _method parameter to override the verb. (HTTP Verb tunneling)
The attack does not work for Chrome users.

Attachments

Warning: Attachments received through HackerOne, please exercise caution!

• setup.webm
• victim-poc.webm
• csrf-leak-prevented-by-csp.png
• k8s_csrf.py

How To Reproduce

Please add reproducibility information to this section: