Isolating Kubernetes executor jobs when using shared volume for build_dir
Description
Our organisation wants to make use of PVCs for mounting build_dir. We suffer from disk io issues even when using NVMe host drives, so want to offload IO to rook ceph. tmpfs emptyDir is complex to administer given the range of repo sizes + jobs we have, a shared PVC is preferable.
Following issue #3847 (closed) and related issues/MRs, I have not been able to find any mention of properly isolating job build directories on shared storage (other than #27835).
In other words, with emptyDir the build_dir is isolated, whereas a shared PVC build_dir allows for unintentional "attacks" between jobs, e.g. a careless pipeline author might 'rm -rf' one or two directories too high and break other jobs.
I have one proposal but would be interested in hearing alternatives.
I did find issue #27835 but felt that was a more complex option, otoh I probably have missed something obvious about my proposed solution.
Proposal
The easiest way I could think to get an isolated build_dirs in a PVC is to allow dynamic values for the mountPath + subPath volume mounting options.
Having explored securityContext + initContainers and falling short, the idea comes from @jansmets who commented on the aforementioned issue: #3847 (comment 269924207)
The code change was straightforward so I've already submitted an MR (!3111 (closed)).