Runner deployment rollout stuck: "unable to validate against any security context constraint"
I've got an operator-managed Runner deployment that is trying to roll out a new replicaset, which is failing. Here's the deployment:
apiVersion: apps/v1
kind: Deployment
metadata:
annotations:
deployment.kubernetes.io/revision: "7"
creationTimestamp: "2022-11-07T15:36:37Z"
generation: 7
labels:
app.kubernetes.io/component: runner
app.kubernetes.io/instance: gitlab-runner-runner
app.kubernetes.io/managed-by: gitlab-runner-operator
app.kubernetes.io/name: gitlab-runner
app.kubernetes.io/part-of: runner
argocd.argoproj.io/instance: xxx
name: gitlab-runner-runner
namespace: xxx
ownerReferences:
- apiVersion: apps.gitlab.com/v1beta2
blockOwnerDeletion: true
controller: true
kind: Runner
name: gitlab-runner
uid: 38354372-a168-485d-ad5a-6ab193aca050
resourceVersion: "1435916598"
uid: b1fec4f1-e75c-4af1-874b-6a0110189649
spec:
progressDeadlineSeconds: 600
replicas: 1
revisionHistoryLimit: 10
selector:
matchLabels:
app.kubernetes.io/component: runner
app.kubernetes.io/instance: gitlab-runner-runner
app.kubernetes.io/managed-by: gitlab-runner-operator
app.kubernetes.io/name: gitlab-runner
app.kubernetes.io/part-of: runner
argocd.argoproj.io/instance: xxx
strategy:
rollingUpdate:
maxSurge: 25%
maxUnavailable: 25%
type: RollingUpdate
template:
metadata:
annotations:
gitlab-runner-runner-config: 51ccab65dbedb82cce1354608581e2c5c3a60963caae5517ff41cd854a8cef3d
creationTimestamp: null
labels:
app.kubernetes.io/component: runner
app.kubernetes.io/instance: gitlab-runner-runner
app.kubernetes.io/managed-by: gitlab-runner-operator
app.kubernetes.io/name: gitlab-runner
app.kubernetes.io/part-of: runner
argocd.argoproj.io/instance: xxx
spec:
containers:
- command:
- /bin/bash
- /scripts/entrypoint
env:
- name: CI_SERVER_URL
valueFrom:
configMapKeyRef:
key: ci_server_url
name: gitlab-runner-runner-config
- name: CI_SERVER_TOKEN
valueFrom:
secretKeyRef:
key: runner-token
name: gitlab-runner-token
- name: REGISTRATION_TOKEN
valueFrom:
secretKeyRef:
key: runner-registration-token
name: gitlab-runner-token
- name: RUNNER_REQUEST_CONCURRENCY
value: "1"
- name: RUNNER_EXECUTOR
value: kubernetes
- name: RUNNER_OUTPUT_LIMIT
value: "4096"
- name: KUBERNETES_NAMESPACE
value: xxx
- name: KUBERNETES_POLL_TIMEOUT
value: "180"
- name: CACHE_SHARED
value: "false"
- name: KUBERNETES_HELPER_IMAGE
value: registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-helper-ocp:v16.0.1
- name: REGISTER_LOCKED
value: "false"
- name: CI_SERVER_TLS_CA_FILE
value: /home/gitlab-runner/.gitlab-runner/certs/hostname.crt
image: registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-ocp:v16.0.1
imagePullPolicy: Always
lifecycle:
preStop:
exec:
command:
- gitlab-runner
- unregister
- --all-runners
livenessProbe:
exec:
command:
- /bin/bash
- /scripts/check-live
failureThreshold: 3
initialDelaySeconds: 60
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
name: runner
readinessProbe:
exec:
command:
- /bin/bash
- /scripts/check-live
failureThreshold: 3
initialDelaySeconds: 10
periodSeconds: 10
successThreshold: 1
timeoutSeconds: 1
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /secrets
name: runner-secrets
- mountPath: /scripts
name: scripts
dnsPolicy: ClusterFirst
initContainers:
- command:
- sh
- /config/configure
env:
- name: CI_SERVER_URL
valueFrom:
configMapKeyRef:
key: ci_server_url
name: gitlab-runner-runner-config
- name: CI_SERVER_TOKEN
valueFrom:
secretKeyRef:
key: runner-token
name: gitlab-runner-token
- name: REGISTRATION_TOKEN
valueFrom:
secretKeyRef:
key: runner-registration-token
name: gitlab-runner-token
- name: RUNNER_REQUEST_CONCURRENCY
value: "1"
- name: RUNNER_EXECUTOR
value: kubernetes
- name: RUNNER_OUTPUT_LIMIT
value: "4096"
- name: KUBERNETES_NAMESPACE
value: xxx
- name: KUBERNETES_POLL_TIMEOUT
value: "180"
- name: CACHE_SHARED
value: "false"
- name: KUBERNETES_HELPER_IMAGE
value: registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-helper-ocp:v16.0.1
- name: REGISTER_LOCKED
value: "false"
- name: CI_SERVER_TLS_CA_FILE
value: /home/gitlab-runner/.gitlab-runner/certs/hostname.crt
image: registry.gitlab.com/gitlab-org/ci-cd/gitlab-runner-ubi-images/gitlab-runner-ocp:v16.0.1
imagePullPolicy: Always
name: configure
resources: {}
securityContext:
allowPrivilegeEscalation: false
capabilities:
drop:
- ALL
terminationMessagePath: /dev/termination-log
terminationMessagePolicy: File
volumeMounts:
- mountPath: /secrets
name: runner-secrets
- mountPath: /config
name: scripts
readOnly: true
- mountPath: /init-secrets
name: init-runner-secrets
readOnly: true
restartPolicy: Always
schedulerName: default-scheduler
securityContext:
runAsNonRoot: true
seccompProfile:
type: RuntimeDefault
serviceAccount: gitlab-runner-sa
serviceAccountName: gitlab-runner-sa
terminationGracePeriodSeconds: 30
volumes:
- emptyDir:
medium: Memory
name: runner-secrets
- name: init-runner-secrets
projected:
defaultMode: 420
sources:
- secret:
items:
- key: runner-registration-token
path: runner-registration-token
- key: runner-token
path: runner-token
name: gitlab-runner-token
- secret:
items:
- key: tls.crt
path: hostname.crt
name: my-ca
- configMap:
defaultMode: 420
items:
- key: config.toml
path: config.toml
- key: entrypoint
path: entrypoint
- key: register-runner
path: register-runner
- key: check-live
path: check-live
- key: configure
path: configure
name: gitlab-runner-runner-config
name: scripts
status:
availableReplicas: 1
conditions:
- lastTransitionTime: "2023-07-12T14:11:44Z"
lastUpdateTime: "2023-07-12T14:11:44Z"
message: Deployment has minimum availability.
reason: MinimumReplicasAvailable
status: "True"
type: Available
- lastTransitionTime: "2023-07-15T00:04:15Z"
lastUpdateTime: "2023-07-15T00:04:15Z"
message: 'pods "gitlab-runner-runner-5ffd5c8ccc-wmtls" is forbidden: unable to
validate against any security context constraint: [pod.metadata.annotations.seccomp.security.alpha.kubernetes.io/pod:
Forbidden: seccomp may not be set pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/configure:
Forbidden: seccomp may not be set pod.metadata.annotations.container.seccomp.security.alpha.kubernetes.io/runner:
Forbidden: seccomp may not be set provider "anyuid": Forbidden: not usable by
user or serviceaccount provider "pipelines-scc": Forbidden: not usable by user
or serviceaccount provider "nonroot": Forbidden: not usable by user or serviceaccount
provider "noobaa": Forbidden: not usable by user or serviceaccount provider
"noobaa-endpoint": Forbidden: not usable by user or serviceaccount provider
"hostmount-anyuid": Forbidden: not usable by user or serviceaccount provider
"elasticsearch-scc": Forbidden: not usable by user or serviceaccount provider
"log-collector-scc": Forbidden: not usable by user or serviceaccount provider
"machine-api-termination-handler": Forbidden: not usable by user or serviceaccount
provider "hostnetwork": Forbidden: not usable by user or serviceaccount provider
"hostaccess": Forbidden: not usable by user or serviceaccount provider "rook-ceph":
Forbidden: not usable by user or serviceaccount provider "node-exporter": Forbidden:
not usable by user or serviceaccount provider "rook-ceph-csi": Forbidden: not
usable by user or serviceaccount provider "privileged": Forbidden: not usable
by user or serviceaccount]'
reason: FailedCreate
status: "True"
type: ReplicaFailure
- lastTransitionTime: "2023-07-17T13:12:34Z"
lastUpdateTime: "2023-07-17T13:12:34Z"
message: ReplicaSet "gitlab-runner-runner-6f87dd6b8c" has timed out progressing.
reason: ProgressDeadlineExceeded
status: "False"
type: Progressing
observedGeneration: 7
readyReplicas: 1
replicas: 1
unavailableReplicas: 1
It looks like the problem is caused by the addition of .spec.template.spec.seccompProfile
; none of my other runners appear to have this field. I don't know where it's come from either.
The runner spec is pretty simple:
apiVersion: apps.gitlab.com/v1beta2
kind: Runner
metadata:
annotations:
argocd.argoproj.io/tracking-id: xxx:apps.gitlab.com/Runner:xxx/gitlab-runner
kubectl.kubernetes.io/last-applied-configuration: |
{"apiVersion":"apps.gitlab.com/v1beta2","kind":"Runner","metadata":{"annotations":{"argocd.argoproj.io/tracking-id":"xxx:apps.gitlab.com/Runner:xxx/gitlab-runner"},"labels":{"argocd.argoproj.io/instance":"xxx"},"name":"gitlab-runner","namespace":"xxx"},"spec":{"ca":"phe-ca","gitlabUrl":"https://gitlab.phe.gov.uk","imagePullPolicy":"Always","token":"gitlab-runner-token"}}
creationTimestamp: "2022-11-07T15:29:46Z"
finalizers:
- finalizer.gitlab.com
generation: 1
labels:
argocd.argoproj.io/instance: xxx
name: gitlab-runner
namespace: xxx
resourceVersion: "1429050863"
uid: 38354372-a168-485d-ad5a-6ab193aca050
spec:
ca: phe-ca
gitlabUrl: https://gitlab.xxx
imagePullPolicy: Always
token: gitlab-runner-token
status:
phase: Running
registration: succeeded
... so I don't think it's caused by anything in the runner object.