"failed liveness probe, will be restarted" loop when using securityContext.runAsNonRoot = false

Summary

See title.

Steps to reproduce

# values.yaml
image:
  registry: registry.gitlab.com
  image: gitlab-org/gitlab-runner
  tag: ubuntu
securityContext:
  allowPrivilegeEscalation: true
  readOnlyRootFilesystem: false
  runAsNonRoot: false
  privileged: true
capabilities:
  drop: ["ALL"]
podSecurityContext:
  runAsUser: 0
  fsGroup: 0

# config.toml
runners:
  config: |
    [[runners]]
      [runners.kubernetes]
      namespace = "{{.Release.Namespace}}"
      image = "ubuntu:latest"
      privileged = true

Actual behavior

12m (x2 over 14m)       Normal    Started             Pod/gitlab-runner-…    Started container gitlab-runner
12m (x3 over 13m)       Warning   Unhealthy           Pod/gitlab-runner-…    Liveness probe failed:
12m                     Normal    Killing             Pod/gitlab-runner-…    Container gitlab-runner failed liveness probe, will be restarted
12m                     Warning   Unhealthy           Pod/gitlab-runner-…    Readiness probe errored:

# ME: rpc error: code = Unknown desc = container not running (a9…a9)
# CUSTOMER: rpc error: code = Unknown desc = failed to exec in container: container is in CONTAINER_EXITED state

Expected behavior

No readiness probe error.

Relevant logs and/or screenshots

See event list above.

Environment description

Customer is using an EKS cluster with custom CA. I've reproduced this with the k8s cluster built into Docker Desktop. However, it is not clear at this point whether the custom CA is involved or not. We believe the problem is not related to the CA, though, because that aspect was taken care of with the documented "Default - Read the system certificate" config.

Used GitLab Runner version: 16.11.0

Possible fixes

?

Edited by Katrin Leinweber