gitlab runner upgrade 15.110 -> 16.0.2 results in buildah build stops working with kubernetes executor
Summary
gitlab runner upgrade 15.110 -> 16.0.2 results in buildah stops working with kubernetes executor
Steps to reproduce
Run buildah build on gitlab runner 16.0.2 or above (tested up to 16.3.0)
.gitlab-ci.yml
docker_build:
image: quay.io/buildah/stable:v1.31.0
before_script:
- buildah login -u $CI_REGISTRY_USER -p $CI_REGISTRY_PASSWORD $CI_REGISTRY
script:
- time buildah build -t $IMAGE_TAG -f $CI_PROJECT_DIR/$DOCKERFILE --build-context pnpm-store=.pnpm-store --layers --cache-to $CI_REGISTRY_IMAGE/cache-buildah --cache-from $CI_REGISTRY_IMAGE/cache-buildah --registries-conf $BUILDAH_REGISTRIES_CONF
- echo $CI_REGISTRY_IMAGE/$IMAGE:$IMAGE_TAG
- time buildah push $IMAGE_TAG docker://$CI_REGISTRY_IMAGE/$IMAGE:$IMAGE_TAG
Actual behavior
Running with gitlab-runner 16.0.2
image: quay.io/buildah/stable:v1.31.0
buildah build ...
Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied
time="2023-09-01T12:04:23Z" level=warning msg="failed to shutdown storage: \"mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied\""
Expected behavior
Running with gitlab-runner 15.11.0
image: quay.io/buildah/stable:v1.31.0
buildah build ...
OK!
Relevant logs and/or screenshots
job log
$ time buildah build -t $IMAGE_TAG -f $CI_PROJECT_DIR/$DOCKERFILE --build-context pnpm-store=.pnpm-store --layers --cache-to $CI_REGISTRY_IMAGE/cache-buildah --cache-from $CI_REGISTRY_IMAGE/cache-buildah --registries-conf $BUILDAH_REGISTRIES_CONF
Error: mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied
time="2023-09-01T12:09:08Z" level=warning msg="failed to shutdown storage: \"mount /var/lib/containers/storage/overlay:/var/lib/containers/storage/overlay, flags: 0x1000: permission denied\""
Environment description
Helm chart installation
helm chart values
values:
gitlabUrl: https://gitlab.foo.com/
checkInterval: 3
concurrent: 10
rbac:
create: true
runners:
tags: "k8s,linux"
secret: xxx
runUntagged: true
privileged: true
cache:
secretName: xxx
config: |
[[runners]]
[runners.kubernetes]
cpu_request = "100m"
memory_request = "100Mi"
cpu_limit = "2000m"
memory_limit = "8Gi"
helper_cpu_request = "100m"
helper_memory_request = "100Mi"
helper_cpu_limit = "2000m"
helper_memory_limit = "4Gi"
[runners.cache]
Type = "s3"
Path = "runner"
Shared = true
[runners.cache.s3]
ServerAddress = "xxxx:9000"
BucketName = "gitlab-runners-cache"
Insecure = true
AuthenticationType = "access-key"
resources:
requests:
memory: 128Mi
cpu: 100m
limits:
memory: 8Gi
cpu: 2000m
Used GitLab Runner version
Described above
Possible fixes
Have not found any workarounds yet except to downgrade gitlab runner to 15.11.0.
Edited by Jonas Kello