Kubernetes runner missing capability net_admin (and many others) and no cap_add or cap_drop functionality

Summary

Kubernetes runner missing capability net_admin (and many others) compared to shared runners on gitlab.com without any cap_add or cap_drop functionality.

Steps to reproduce

Follow gitlab instructions to deploy kubernetes gitlab runner using helm into your own kubernetes cluster. We used the latest helm chart (0.19.1).

.gitlab-ci.yml 
before_script:
  - openvpn --config /tmp/$OVPN --daemon --status $OVPN_LOG

Actual behavior

ifconfig: tun0: error fetching interface information: Device not found

Expected behavior

tun0 vpn network device is displayed as part of our script:

tun0      Link encap:UNSPEC  HWaddr 00-00-00-00-00-00-00-00-00-00-00-00-00-00-00-00  
          inet addr:18.0.0.162  P-t-P:18.0.0.162  Mask:255.255.255.224
          UP POINTOPOINT RUNNING NOARP MULTICAST  MTU:1500  Metric:1
          RX packets:0 errors:0 dropped:0 overruns:0 frame:0
          TX packets:0 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:100 
          RX bytes:0 (0.0 B)  TX bytes:0 (0.0 B)

Relevant logs and/or screenshots

I have used capsh in the shared runner and the kubernetes runner to compare capabilities:

shared runner (all capabilities?):

$ capsh --print
Current: = cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read+eip
Bounding set =cap_chown,cap_dac_override,cap_dac_read_search,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_linux_immutable,cap_net_bind_service,cap_net_broadcast,cap_net_admin,cap_net_raw,cap_ipc_lock,cap_ipc_owner,cap_sys_module,cap_sys_rawio,cap_sys_chroot,cap_sys_ptrace,cap_sys_pacct,cap_sys_admin,cap_sys_boot,cap_sys_nice,cap_sys_resource,cap_sys_time,cap_sys_tty_config,cap_mknod,cap_lease,cap_audit_write,cap_audit_control,cap_setfcap,cap_mac_override,cap_mac_admin,cap_syslog,cap_wake_alarm,cap_block_suspend,cap_audit_read
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=0(root)
gid=0(root)
groups=

kubernetes runner (note cap_net_admin missing along with a load more):

$ capsh --print
Current: = cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap+eip
Bounding set =cap_chown,cap_dac_override,cap_fowner,cap_fsetid,cap_kill,cap_setgid,cap_setuid,cap_setpcap,cap_net_bind_service,cap_net_raw,cap_sys_chroot,cap_mknod,cap_audit_write,cap_setfcap
Securebits: 00/0x0/1'b0
 secure-noroot: no (unlocked)
 secure-no-suid-fixup: no (unlocked)
 secure-keep-caps: no (unlocked)
uid=0(root)
gid=0(root)
groups=

Environment description

Using gitlab.com.

kubenetes runner running within our own kubernetes cluster.

Used GitLab Runner version

Shared runner:

Running with gitlab-runner 13.2.2 (a998cacd)
  on docker-auto-scale 0277ea0f
Preparing the "docker+machine" executor
00:36
Using Docker executor with image ruby:2.5 ...
Pulling docker image ruby:2.5 ...
Using docker image sha256:896afe8ad24a941448575390111af68ddcf195105b8d1f23e60461c0b8c4ed3c for ruby:2.5 ...
Running with gitlab-ci-multi-runner 1.4.2 (bcc1794)
Using Docker executor with image golang:1.8 ...

kubernetes runner:

Running with gitlab-runner 13.2.1 (efa30e33)
  on gitlab-runner-gitlab-runner-74bb4ccccc-ltlmd NiNu2SPJ
Preparing the "kubernetes" executor
00:00
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image ubuntu:16.04 ...

Possible fixes

There is no capabilities settings in the kubernetes runner; only these for security context. You might compare to the docker runner that has cap_add and cap_drop as documented here. The kubernetes runner is clearly missing these and I am not sure why (less used/developed?).

Suggested fixes:

  • Add capability net_admin which would be a quick fix and make us really happy!
  • Add all the capabilities that are available by default in the shared runner. This would be a quicker fix but may be deemed insecure (however you could argue that case with the shared runner).
  • Add cap_add and cap_drop features similar to the docker runner (might be easier to just lift these changes from the docker runner and adapt for kubernetes?).
Edited by Alastair Munro