gitlab runner 15.x won't work with podman 4.6.x and crun 1.9.x

Summary

gitlab-runner 15.9.1 won't work with crun 1.9.2

Steps to reproduce

Setup gitlab runner with podman and try to run a pipeline with a cache, for example,


variables:
  NPMCACHE: "${CI_PROJECT_DIR}/.npmcache"

image: docker.io/node:latest
  


build:
  stage: build
  cache:
    key: builddeps
    paths:
    - .npmcache/
  ...

Actual behavior

ERROR: Preparation failed: adding cache volume: set volume permissions: running permission container "**01b216dc9fbf01d532cea70aef12fc31f1ef1d40ee199ee0aac14bdfb68351b9***************" for volume "runner-uzpd-gqh-project-***-concurrent-0-cache-********": starting permission container: Error response from daemon: crun: write to `/proc/self/oom_score_adj`: Permission denied: OCI permission denied (linux_set.go:105:0s)

Expected behavior

It can work with podman/crun

Relevant logs and/or screenshots

ERROR: Preparation failed: adding cache volume: set volume permissions: running permission container "**01b216dc9fbf01d532cea70aef12fc31f1ef1d40ee199ee0aac14bdfb68351b9***************" for volume "runner-uzpd-gqh-project-***-concurrent-0-cache-********": starting permission container: Error response from daemon: crun: write to `/proc/self/oom_score_adj`: Permission denied: OCI permission denied (linux_set.go:105:0s)

Environment description

Installation: gitlab-oss (15.4.3), and own gitlab-runner (15.9.1) setup with podman on CentOS Stream 9.

Used GitLab Runner version

Version: 15.9.1

concurrent = 1
check_interval = 0
shutdown_timeout = 0

[session_server]
  session_timeout = 1800

[[runners]]
  name = "****"
  url = "*****"
  id = ***
  token = "*****"
  token_obtained_at = 2023-06-05T17:50:23Z
  token_expires_at = 0001-01-01T00:00:00Z
  executor = "docker"
  environment = ["FF_NETWORK_PER_BUILD=1"]
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    host = "unix:///run/user/***/podman/podman.sock"
    tls_verify = false
    image = "quay.io/podman/stable"
    privileged = true
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    volumes = ["/cache"]
    shm_size = 0

Possible fixes

Downgrade crun to 1.8.x

Additional info

This does not seems to be a problem with crun, but appears to be caused by the docker library gitlab-runner uses to submit requests, and the way it is initializing the oom score adjustment which won't work for unprivileged users.

See:

Edited by Carlos Rodriguez-Fernandez