Job errors with failed to create cache after upgrade to Runner 16.7. Appears to be specific to FIPS helper image.
Summary
When using the FIPS helper image jobs end up failing with the below error.
ERROR: Job failed: creating cache volume: set volume permissions: running permission container.
Last version that was tested to work is 16.6.2 and following tested version where it broke was 16.7.0.
Workaround tested
Copy the prebuilt-ubi-fips-x86_64.tar.xz
image from a working version such as 16.6.2 into the runners /usr/lib/gitlab-runner/helper-images
directory and the job executes successfully.
Install the FIPS runner version 16.7.0 or higher and run any job that uses the FIPS helper image.
Actual behavior
ERROR: Job failed: creating cache volume: set volume permissions: running permission container.
Expected behavior
No errors
Relevant logs and/or screenshots
job log
Running with gitlab-runner 16.7.0 (102c81ba)
2 on mbzex30891 dhxTbSo3, system ID: s_7d8efcfc3bb9
3 feature flags: FF_NETWORK_PER_BUILD:true, FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR:true
4Resolving secrets00:00
5Preparing the "docker" executor00:06
6Using Docker executor with image gitlab-instance/dso-builder-images/builder-megalinter-documentation:v7 ...
7ERROR: Job failed: creating cache volume: set volume permissions: running permission container "346f0f310289cd9981cae12337cc06567ebd42b68566ae04931ae3557875b3d8" for volume "runner-dhxtbso3-project-14-concurrent-0-bd6de6b5758d1031-cache-c33bcaa1fd2c77edfc3893b41966cea8": waiting for permission container to finish: exit code 1
Environment description
config.toml contents
check_interval = 0
concurrent = 14
log_format = "json"
[[runners]]
environment = ["FF_NETWORK_PER_BUILD=1","FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR=1"]
executor = "docker"
name = "xyz"
request_concurrency = 14
token = "abc"
url = https://gitlab-server.example
[runners.docker]
allowed_images = ["gitlab-server.example/**/*:*"]
disable_cache = true
disable_entrypoint_overwrite = false
helper_image_flavor = "ubi-fips"
image = "gitlab-server.example/dso-builder-images/builder-kaniko:latest"
oom_kill_disable = false
privileged = false
shm_size = 0
docker info
Client:
Context: default
Debug Mode: false
Plugins:
app: Docker App (Docker Inc., v0.9.1-beta3)
buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
scan: Docker Scan (Docker Inc., v0.21.0)
Server:
Containers: 0
Running: 0
Paused: 0
Stopped: 0
Images: 0
Server Version: 20.10.21
Storage Driver: overlay2
Backing Filesystem: xfs
Supports d_type: true
Native Overlay Diff: true
userxattr: false
Logging Driver: json-file
Cgroup Driver: cgroupfs
Cgroup Version: 1
Plugins:
Volume: local
Network: bridge host ipvlan macvlan null overlay
Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
Swarm: inactive
Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
Default Runtime: runc
Init Binary: docker-init
containerd version: 78f51771157abb6c9ed224c22013cdf09962315d
runc version: v1.1.4-0-g5fd4c4d
init version: de40ad0
Security Options:
seccomp
Profile: default
selinux
Kernel Version: 4.18.0-513.5.1.el8_9.x86_64
Operating System: Red Hat Enterprise Linux 8.9 (Ootpa)
OSType: linux
Architecture: x86_64
CPUs: 16
Total Memory: 62.35GiB
Name: gitlab-server.example
ID: 3JVX:RPSR:RU24:HFRC:73CS:VMOD:XU7Q:D3Q6:TKH2:BS2Z:YDVP:A4EY
Docker Root Dir: /var/lib/docker
Debug Mode: false
Registry: https://index.docker.io/v1/
Labels:
Experimental: false
Insecure Registries:
127.0.0.0/8
Live Restore Enabled: false
Used GitLab Runner version
runners are set up on RHEL 8 hosts with fips mode enabled and the gitlab-runner-fips packages
Using Docker executor
Possible fixes
-
TBD as this bug has not yet been fully triaged.