ephemeral and private /builds directory for kubernetes executor
Summary
I tried the PVC-based /builds directory for ephemeral storage introduced with !2862 (merged) (using kubernetes runner version 14.5.0). Result: Its not ephemeral at all. There are 2 issues with this approach:
- privacy (aka "need to know"). The /builds directory is shared with all kubernetes executors started by this runner, which means that I have to provide dedicated kubernetes runners for each and every project to keep them separate, instead of using a common pool of runners for the whole group.
- missing garbage collection. Unlike other runners the /builds directory is not cleaned up after the pipeline has been completed, AFAICT. The runner runs into "No space left on device" in almost no time.
Steps to reproduce
values.yaml
:
runners:
# runner configuration, where the multi line strings is evaluated as
# template so you can specify helm values inside of it.
#
# tpl: https://helm.sh/docs/howto/charts_tips_and_tricks/#using-the-tpl-function
# runner configuration: https://docs.gitlab.com/runner/configuration/advanced-configuration.html
config: |
[[runners]]
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
image = "ubuntu:20.04"
privileged = true
pull_policy = "always"
cpu_request = "500m"
cpu_limit = "2"
memory_request = "2Gi"
memory_limit = "8Gi"
[runners.kubernetes.pod_labels]
"environment" = "runnerpod"
[[runners.kubernetes.volumes.pvc]]
name = "runner-pvc-support"
mount_path = "/builds"
[runners.cache]
Type = "s3"
Path = "runners.cache"
Shared = true
[runners.cache.s3]
ServerAddress = "minio005.example.com:9000"
BucketName = "runners"
Insecure = false
:
runner-pvc-support.pvc.yaml:
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
name: runner-pvc-support
namespace: gitlab-runner
spec:
storageClassName: sc-repl1-pool2
accessModes:
- ReadWriteOnce
resources:
requests:
storage: 64Gi
.gitlab-ci.yml
sleeping beauty:
image: debian:buster
tags:
- gitlab-runner
script:
- cat /etc/os-release
- date
- echo "these are the resources of the regular container"
- cat /sys/fs/cgroup/cpu/cpu.shares
- cat /sys/fs/cgroup/cpu/cpu.cfs_period_us
- cat /sys/fs/cgroup/cpu/cpu.cfs_quota_us
- cat /sys/fs/cgroup/memory/memory.usage_in_bytes
- cat /sys/fs/cgroup/memory/memory.limit_in_bytes
- sleep 600
- date
Actual behavior
Run the pipeline twice (in parallel). Run a bash session in the second kubernetes executor to access the checked out code of the first, even after it has terminated. This is bad.
Expected behavior
The runner should mount a private build directory on /builds for each executor, inaccessible for other executors. This can be a subdirectory in the common PV. Please clean up and recover disk space immediately after the pipeline has been completed.
Edited by Harald Dunkel