Skip to content

Set fsGroup for workspace volume

Vishal Tak requested to merge workspaces_hotfix into master

What does this MR do and why?

Set fsGroup for workspace volume

Problem

➜  gitlab-ui git:(add_devfile) kubectl logs -f smoke-test-workspace-74fbcf48bb-p9phh -c gl-cloner-injector-gl-cloner-injector-command-1
fatal: could not create work tree dir '/projects/example-nodejs-express-app': Permission denied
➜  gitlab-ui git:(add_devfile) kubectl edit deployment  smoke-test-workspace
deployment.apps/smoke-test-workspace edited
➜  gitlab-ui git:(add_devfile) kubectl get po -w
NAME                                    READY   STATUS              RESTARTS   AGE
smoke-test-workspace-7679b5894d-n6znm   0/1     ContainerCreating   0          8s
smoke-test-workspace-7679b5894d-n6znm   1/1     Running             0          34s
^C%
➜  gitlab-ui git:(add_devfile) kubectl exec -it smoke-test-workspace-6484574448-ckzgg -- /bin/sh
$ cd /projects
$ id
uid=5001 gid=0(root) groups=0(root)
$ git clone git@gitlab.com:gitlab-org/remote-development/gitlab-remote-development-docs.git
fatal: could not create work tree dir 'gitlab-remote-development-docs': Permission denied
$ touch test.md
touch: cannot touch 'test.md': Permission denied
$ ls -lart .
total 24
drwx------ 2 root root 16384 May  9 17:11 lost+found
drwxr-xr-x 3 root root  4096 May  9 17:11 .
drwxr-xr-x 1 root root  4096 May  9 17:19 ..
$ pwd
/projects
$ ls -lart /
total 104
drwxr-xr-x   2 root root 4096 Dec  9 19:15 boot
drwxr-xr-x   1 root root 4096 Apr 11 00:00 var
drwxr-xr-x   2 root root 4096 Apr 11 00:00 srv
drwxr-xr-x   2 root root 4096 Apr 11 00:00 opt
drwxr-xr-x   2 root root 4096 Apr 11 00:00 mnt
drwxr-xr-x   2 root root 4096 Apr 11 00:00 media
drwxrwxrwx   2 root root 4096 May  2 21:36 scripts
drwxrwxrwx   5 root root 4096 May  2 21:36 patches
drwxrwxrwx   3 root root 4096 May  2 21:36 assets
-rw-rw-rw-   1 root root   16 May  2 21:36 .gitignore
-rw-rw-rw-   1 root root   40 May  2 21:36 .gitattributes
-rw-rw-rw-   1 root root  101 May  2 21:36 .dockerignore
drwxr-xr-x   1 root root 4096 May  2 21:36 lib64
drwxr-xr-x   1 root root 4096 May  2 21:36 sbin
drwxr-xr-x   1 root root 4096 May  2 21:37 usr
drwxr-xr-x   2 root root 4096 May  2 21:37 lib32
drwxr-xr-x   1 root root 4096 May  2 21:37 bin
drwxr-xr-x   1 root root 4096 May  2 21:37 lib
drwx------   1 root root 4096 May  2 21:52 root
drwxrwxrwt   1 root root 4096 May  2 21:52 tmp
drwxrwxr-x   1 root root 4096 May  9 04:56 home
drwxr-xr-x   3 root root 4096 May  9 17:11 projects
dr-xr-xr-x  13 root root    0 May  9 17:19 sys
dr-xr-xr-x 228 root root    0 May  9 17:19 proc
drwxr-xr-x   1 root root 4096 May  9 17:19 etc
drwxr-xr-x   1 root root 4096 May  9 17:19 ..
drwxr-xr-x   1 root root 4096 May  9 17:19 .
drwxr-xr-x   1 root root 4096 May  9 17:19 run
drwxr-xr-x   5 root root  360 May  9 17:19 dev

Notice that the /projects folder where the volume is mounted does not have linux group write permissions.

Solution

As per https://kubernetes.io/blog/2020/12/14/kubernetes-release-1.20-fsgroupchangepolicy-fsgrouppolicy/

Traditionally if your pod is running as a non-root user (which you should), you must specify a fsGroup inside the pod’s security context so that the volume can be readable and writable by the Pod.

This is what this MR does. After I exec into a Pod which has fsGroup set, the output is

$ ls -lart /
total 108
drwxr-xr-x   2 root root 4096 Dec  9 19:15 boot
drwxr-xr-x   1 root root 4096 Apr 11 00:00 var
drwxr-xr-x   2 root root 4096 Apr 11 00:00 srv
drwxr-xr-x   2 root root 4096 Apr 11 00:00 opt
drwxr-xr-x   2 root root 4096 Apr 11 00:00 mnt
drwxr-xr-x   2 root root 4096 Apr 11 00:00 media
drwxrwxrwx   2 root root 4096 May  2 21:36 scripts
drwxrwxrwx   5 root root 4096 May  2 21:36 patches
drwxrwxrwx   3 root root 4096 May  2 21:36 assets
-rw-rw-rw-   1 root root   16 May  2 21:36 .gitignore
-rw-rw-rw-   1 root root   40 May  2 21:36 .gitattributes
-rw-rw-rw-   1 root root  101 May  2 21:36 .dockerignore
drwxr-xr-x   1 root root 4096 May  2 21:36 lib64
drwxr-xr-x   1 root root 4096 May  2 21:36 sbin
drwxr-xr-x   1 root root 4096 May  2 21:37 usr
drwxr-xr-x   2 root root 4096 May  2 21:37 lib32
drwxr-xr-x   1 root root 4096 May  2 21:37 bin
drwxr-xr-x   1 root root 4096 May  2 21:37 lib
drwx------   1 root root 4096 May  2 21:52 root
drwxrwxrwt   1 root root 4096 May  2 21:52 tmp
drwxrwxr-x   1 root root 4096 May  9 04:56 home
dr-xr-xr-x  13 root root    0 May  9 18:18 sys
drwxrwsr-x   5 root root 4096 May  9 18:18 projects
dr-xr-xr-x 236 root root    0 May  9 18:18 proc
drwxr-xr-x   1 root root 4096 May  9 18:18 etc
drwxr-xr-x   1 root root 4096 May  9 18:18 ..
drwxr-xr-x   1 root root 4096 May  9 18:18 .
drwxr-xr-x   1 root root 4096 May  9 18:18 run
drwxr-xr-x   5 root root  360 May  9 18:18 dev
$ ls -lart /projects
total 32
drwxrws--- 2 root root 16384 May  9 18:18 lost+found
drwxrwsr-x 5 root root  4096 May  9 18:18 .
drwxr-sr-x 3 5001 root  4096 May  9 18:18 .gl-editor
drwxr-xr-x 1 root root  4096 May  9 18:18 ..
drwxr-sr-x 4 5001 root  4096 May  9 18:19 example-nodejs-express-app

Notice how the /projects folder has linux group write permissions.

This is behind a feature flag whose rollout issue can be tracked here

Edited by Vishal Tak

Merge request reports