Kubernetes executor: Pulling image from GitLab Container Registry fails (401 Unauthorized)

Summary

When using the Kubernetes executor to run CI/CD jobs on Kubernetes, we cannot pull images from our private GitLab Container Registry (registry.example.com). Usually the GitLab runner can authenticate against our GitLab Container Registry (registry.example.com) using the dynamic job token of the user who started the job.

Steps to reproduce

1. Set up a GitLab Omnibus installation.
2. Enable the container registry on GitLab
3. Create a K8s cluster
4. Set up a GitLab runner using the Kubernetes executor

.gitlab-ci.yml

```yml build: stage: build image: registry.example.com/group/repository:latest script: - echo "Test" ```

Actual behavior

The job cannot start as the image cannot be pulled: pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed

Running with gitlab-runner 16.2.0 (782e15da)
  on gitlab-runner-XXXXXXXXXX-XXXXX XXXXXXXXX, system ID: X_XXXXXXXXXXXX
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image registry.example.com/group/repository:latest ...
Using attach strategy to execute scripts...
Preparing environment
Waiting for pod gitlab-runner/runner-XXXXXXXX--project-1-concurrent-1-XXXXXXXX to be running, status is Pending
Waiting for pod gitlab-runner/runner-XXXXXXXX--project-1-concurrent-1-XXXXXXXX to be running, status is Pending
	ContainersNotInitialized: "containers with incomplete status: [init-permissions]"
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
Waiting for pod gitlab-runner/runner-XXXXXXXX--project-1-concurrent-1-XXXXXXXX to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper]"
	ContainersNotReady: "containers with unready status: [build helper]"
WARNING: Event retrieved from the cluster: Failed to pull image "registry.example.com/group/repository:latest": rpc error: code = Unknown desc = failed to pull and unpack image "registry.example.com/group/repository:latest": failed to resolve reference "registry.example.com/group/repository:latest": pull access denied, repository does not exist or may require authorization: server message: insufficient_scope: authorization failed
WARNING: Event retrieved from the cluster: Error: ErrImagePull
WARNING: Event retrieved from the cluster: Error: ImagePullBackOff
WARNING: Failed to pull image with policy "IfNotPresent": image pull failed: Back-off pulling image "registry.example.com/group/repository:latest"
ERROR: Job failed: prepare environment: waiting for pod running: pulling image "registry.example.com/group/repository:latest": image pull failed: Back-off pulling image "registry.example.com/group/repository:latest". Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Expected behavior

The job starts as expected & pulls the image.

Running with gitlab-runner 16.2.0 (782e15da)
  on gitlab-runner-XXXXXXXXXX-XXXXX XXXXXXXXX, system ID: X_XXXXXXXXXXXX
Preparing the "kubernetes" executor
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image registry.example.com/group/repository:latest ...
Using attach strategy to execute scripts...
Preparing environment
Waiting for pod gitlab-runner/runner-XXXXXXXX--project-1-concurrent-1-XXXXXXXX to be running, status is Pending
Running on runner-XXXXXXXX--project-1-concurrent-1-XXXXXXXX via gitlab-runner-XXXXXXXXXX-XXXXX...
Getting source from Git repository
Fetching changes with git depth set to 20...
Initialized empty Git repository in /builds/group/repository/.git/
Created fresh repository.
Checking out XXXXXXXX as detached HEAD (ref is master)...
Skipping Git submodules setup
Executing "step_script" stage of the job script

Environment description

config.toml contents

[[runners]]

  [runners.cache]
    Type = "s3"
    Shared = true
    [runners.cache.s3]
      ServerAddress = "s3.amazonaws.com"
      AccessKey = "XXX"
      SecretKey = "XXX"
      BucketName = "XXX"
      BucketLocation = "XXX"

  [runners.kubernetes]
    namespace = "gitlab-runner"
    image = "alpine:latest"
    pull_policy = "if-not-present"

Used GitLab Runner version

Running with gitlab-runner 16.2.0 (782e15da)