"failed liveness probe, will be restarted" loop when using securityContext.runAsNonRoot = false
Summary
See title.
Steps to reproduce
# values.yaml
image:
registry: registry.gitlab.com
image: gitlab-org/gitlab-runner
tag: ubuntu
securityContext:
allowPrivilegeEscalation: true
readOnlyRootFilesystem: false
runAsNonRoot: false
privileged: true
capabilities:
drop: ["ALL"]
podSecurityContext:
runAsUser: 0
fsGroup: 0
# config.toml
runners:
config: |
[[runners]]
[runners.kubernetes]
namespace = "{{.Release.Namespace}}"
image = "ubuntu:latest"
privileged = true
Actual behavior
12m (x2 over 14m) Normal Started Pod/gitlab-runner-… Started container gitlab-runner
12m (x3 over 13m) Warning Unhealthy Pod/gitlab-runner-… Liveness probe failed:
12m Normal Killing Pod/gitlab-runner-… Container gitlab-runner failed liveness probe, will be restarted
12m Warning Unhealthy Pod/gitlab-runner-… Readiness probe errored:
# ME: rpc error: code = Unknown desc = container not running (a9…a9)
# CUSTOMER: rpc error: code = Unknown desc = failed to exec in container: container is in CONTAINER_EXITED state
Expected behavior
No readiness probe error.
Relevant logs and/or screenshots
See event list above.
Environment description
Customer is using an EKS cluster with custom CA. I've reproduced this with the k8s cluster built into Docker Desktop. However, it is not clear at this point whether the custom CA is involved or not. We believe the problem is not related to the CA, though, because that aspect was taken care of with the documented "Default - Read the system certificate" config.
Used GitLab Runner version: 16.11.0
Possible fixes
?
Edited by Katrin Leinweber