Skip to content

Job errors with failed to create cache after upgrade to Runner 16.7. Appears to be specific to FIPS helper image.

Summary

When using the FIPS helper image jobs end up failing with the below error.

ERROR: Job failed: creating cache volume: set volume permissions: running permission container.

Last version that was tested to work is 16.6.2 and following tested version where it broke was 16.7.0.

Workaround tested

Copy the prebuilt-ubi-fips-x86_64.tar.xz image from a working version such as 16.6.2 into the runners /usr/lib/gitlab-runner/helper-images directory and the job executes successfully.

Install the FIPS runner version 16.7.0 or higher and run any job that uses the FIPS helper image.

Actual behavior

ERROR: Job failed: creating cache volume: set volume permissions: running permission container.

Expected behavior

No errors

Relevant logs and/or screenshots

job log 
Running with gitlab-runner 16.7.0 (102c81ba)
2 on mbzex30891 dhxTbSo3, system ID: s_7d8efcfc3bb9
3 feature flags: FF_NETWORK_PER_BUILD:true, FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR:true
4Resolving secrets00:00
5Preparing the "docker" executor00:06
6Using Docker executor with image gitlab-instance/dso-builder-images/builder-megalinter-documentation:v7 ...
7ERROR: Job failed: creating cache volume: set volume permissions: running permission container "346f0f310289cd9981cae12337cc06567ebd42b68566ae04931ae3557875b3d8" for volume "runner-dhxtbso3-project-14-concurrent-0-bd6de6b5758d1031-cache-c33bcaa1fd2c77edfc3893b41966cea8": waiting for permission container to finish: exit code 1

Environment description

config.toml contents 
check_interval = 0
concurrent = 14
log_format = "json"

[[runners]]
  environment = ["FF_NETWORK_PER_BUILD=1","FF_DISABLE_UMASK_FOR_DOCKER_EXECUTOR=1"]
  executor = "docker"
  name = "xyz"
  request_concurrency = 14
  token = "abc"
  url = https://gitlab-server.example

[runners.docker]
  allowed_images = ["gitlab-server.example/**/*:*"]
  disable_cache = true
  disable_entrypoint_overwrite = false
  helper_image_flavor = "ubi-fips"
  image = "gitlab-server.example/dso-builder-images/builder-kaniko:latest"
  oom_kill_disable = false
  privileged = false
  shm_size = 0
docker info 
Client:
  Context: default
  Debug Mode: false

Plugins:
  app: Docker App (Docker Inc., v0.9.1-beta3)
  buildx: Docker Buildx (Docker Inc., v0.9.1-docker)
  scan: Docker Scan (Docker Inc., v0.21.0)

Server:
  Containers: 0
  Running: 0
  Paused: 0
  Stopped: 0
  Images: 0
  Server Version: 20.10.21
  Storage Driver: overlay2
  Backing Filesystem: xfs
  Supports d_type: true
  Native Overlay Diff: true
  userxattr: false
  Logging Driver: json-file
  Cgroup Driver: cgroupfs
  Cgroup Version: 1

Plugins:
  Volume: local
  Network: bridge host ipvlan macvlan null overlay
  Log: awslogs fluentd gcplogs gelf journald json-file local logentries splunk syslog
  Swarm: inactive
  Runtimes: io.containerd.runc.v2 io.containerd.runtime.v1.linux runc
  Default Runtime: runc
  Init Binary: docker-init
  containerd version: 78f51771157abb6c9ed224c22013cdf09962315d
  runc version: v1.1.4-0-g5fd4c4d
  init version: de40ad0

Security Options:
  seccomp

Profile: default
  selinux
  Kernel Version: 4.18.0-513.5.1.el8_9.x86_64
  Operating System: Red Hat Enterprise Linux 8.9 (Ootpa)
  OSType: linux
  Architecture: x86_64
  CPUs: 16
  Total Memory: 62.35GiB
  Name: gitlab-server.example
  ID: 3JVX:RPSR:RU24:HFRC:73CS:VMOD:XU7Q:D3Q6:TKH2:BS2Z:YDVP:A4EY
  Docker Root Dir: /var/lib/docker
  Debug Mode: false
  Registry: https://index.docker.io/v1/

Labels:
  Experimental: false

Insecure Registries:
  127.0.0.0/8
  Live Restore Enabled: false

Used GitLab Runner version

runners are set up on RHEL 8 hosts with fips mode enabled and the gitlab-runner-fips packages
Using Docker executor

Possible fixes

  • TBD as this bug has not yet been fully triaged.
Edited by Darren Eastman