Runner failed "ERROR: Job failed: prepare environment"

Good morning,

I have deployed Gitlab CE in a Kubernetes Cluster RKE2.

We was on 15.11.13 version and we update to 16.11.6, but now we have problem with the Runners.

Sometimes when a job is launched we received a Fatal Error:

Using FF_USE_POD_ACTIVE_DEADLINE_SECONDS, the Pod activeDeadlineSeconds will be set to the job timeout: 1h0m0s... Waiting for pod gitlab/runner-psfcccmz-project-101-concurrent-0-hzm6q97c to be running, status is Pending WARNING: Failed to pull image with policy "": image pull failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry-gitlab.dev.example.lan/example/thirdpartymirror/kaniko-project/executor:debug": failed to resolve reference "registry-gitlab.dev.example.lan/example/thirdpartymirror/kaniko-project/executor:debug": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed ERROR: Job failed: prepare environment: waiting for pod running: pulling image "registry-gitlab.dev.example.lan/example/thirdpartymirror/kaniko-project/executor:debug": image pull failed: rpc error: code = Unknown desc = failed to pull and unpack image "registry-gitlab.dev.example.lan/example/thirdpartymirror/kaniko-project/executor:debug": failed to resolve reference "registry-gitlab.dev.example.lan/example/thirdpartymirror/kaniko-project/executor:debug": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Sometimes happend sometimes not.

This is the config I got in the runner section inside of the values of the gitlab:

gitlab-runner: gitlabUrl: https://gitlab.dev.example.lan replicas: 4 certsSecretName: "{{ trusted_cas_secret }}" runners: # Changing this doesn't fully set the token everywhere # secret: gitlab-runner-secret config: | [[ runners ]] tls-ca-file = "{{ gitlab_runner_cert_path }}/{{ ldap_ca_filename }}.crt" builds_dir = "/tmp" environment = ["HOME=/tmp", "SSL_CERTFILE={{ gitlab_runner_cert_path }}/{{ ldap_ca_filename }}.crt"] # url = https://gitlab.dev.example.lan [runners.kubernetes] service_account = "gitlab-job-runner" privileged = false helper_image_flavor = "alpine" [runners.kubernetes.pod_security_context] #run_as_nonroot = true run_as_nonroot = false #run_as_user = 100 run_as_user = 0 run_as_group = 65533 [[runners.kubernetes.volumes.secret]] name = "{{ trusted_cas_secret }}" read_only = true mount_path = "{{ gitlab_runner_cert_path }}/" rbac: create: true serviceAccountName: gitlab-runner-sa

Do you have any idea what is happening or can you suggest me something?