Gitlab-runner fails to pull image from docker registry for pipelines triggered via project access tokens

Summary

I'm using GitLab Community Edition 13.4.1 with Gitlab Runner 13.4.0 because I was waiting for #219551 (closed) to be fixed.
I want to use project access tokens for creating new branches and pushing changes on them. This results in a build pipeline that is running with the privileges of the bot user that is associated with the project access token.
Some of the jobs in the pipeline are based on Docker images from the project's container registry. Those jobs are now failing in the preparation phase of the job (i.e. in the Preparing the docker+machine" executor phase) when the executor is trying to pull the image for the job.

ERROR: Job failed (system failure): Error response from daemon: Get https://registry.example.com/v2/klaas.dellschaft/project-access-tokens/manifests/branch-859673: unauthorized: HTTP Basic: Access denied (docker.go:142:0s)

Further details:

The corresponding registry belongs to a public project, i.e. the images in it are accessible even without being logged in to the registry (I tried by docker logout registry.example.com and then docker pull registry.example.com/...).

I was also able to manually access the registry when being logged in with the project access token. (I tried by docker logout registry.example.com, docker login registry.example.com -u gitlab-ci-token, and then docker pull registry.example.com/...)

Steps to reproduce

Create an empty project
In that project, create a project access token with the following scope: api, write_repository, read_registry, write_registry
Put that project access token into the CI/CD variables under the variable name PROJECT_ACCESS_TOKEN
Add the attached .gitlab-ci.yml to that project
After adding the pipeline file, a new pipeline should be automatically started.
The job push-docker-19-to-project-registry pushes the docker:19.03.12 image into the project's container registry.
The job use-docker-19-from-project-registry then uses that image from the project's container registry for a "hello world" test
After the two automatic jobs from the pipeline have been successfully running (with the permissions of the user who pushed the .gitlab-ci.yml), please manually trigger the job create-branch-with-project-access-token. This job will use the Gitlab API to create a new branch, using PROJECT_ACCESS_TOKEN from the CI/CD variables. This will create a pipeline for the new branch, using the credentials of the bot user that is associated with the project access token.

Example Project

Not possible to demonstrate on Gitlab.com because project access tokens are not available there.

What is the current bug behavior?

After creating a new branch using a project access token, a new pipeline is created that is using the credentials of the bot user that is associated with the project access token. In this pipeline, the job use-docker-19-from-project-registry is failing when the docker+machine executor is trying to pull the image for the job from the project's container registry.

What is the expected correct behavior?

The job use-docker-19-from-project-registry for the newly created branch should be successful, i.e. it should be possible that the gitlab-runner / executor can access images from the project's container registry, even if the pipeline is running with the privileges of the bot user associated a project access token for that project.

Relevant logs and/or screenshots

Running with gitlab-runner 13.4.0 (xxx)
  on runner-spawner xxx
Preparing the "docker+machine" executor
00:18
Using Docker executor with image registry.example.com/klaas.dellschaft/project-access-tokens:branch-859673 ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.example.com/klaas.dellschaft/project-access-tokens:branch-859673 ...
ERROR: Preparation failed: Error response from daemon: Get https://registry.example.com/v2/klaas.dellschaft/project-access-tokens/manifests/branch-859673: unauthorized: HTTP Basic: Access denied (docker.go:142:0s)
Will be retried in 3s ...
Using Docker executor with image registry.example.com/klaas.dellschaft/project-access-tokens:branch-859673 ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.example.com/klaas.dellschaft/project-access-tokens:branch-859673 ...
ERROR: Preparation failed: Error response from daemon: Get https://registry.example.com/v2/klaas.dellschaft/project-access-tokens/manifests/branch-859673: unauthorized: HTTP Basic: Access denied (docker.go:142:0s)
Will be retried in 3s ...
Using Docker executor with image registry.example.com/klaas.dellschaft/project-access-tokens:branch-859673 ...
Authenticating with credentials from job payload (GitLab Registry)
Pulling docker image registry.example.com/klaas.dellschaft/project-access-tokens:branch-859673 ...
ERROR: Preparation failed: Error response from daemon: Get https://registry.example.com/v2/klaas.dellschaft/project-access-tokens/manifests/branch-859673: unauthorized: HTTP Basic: Access denied (docker.go:142:0s)
Will be retried in 3s ...
ERROR: Job failed (system failure): Error response from daemon: Get https://registry.example.com/v2/klaas.dellschaft/project-access-tokens/manifests/branch-859673: unauthorized: HTTP Basic: Access denied (docker.go:142:0s)

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info


(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Results of GitLab application Check

Expand for output related to the GitLab application check

(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true)
(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)
(we will only investigate if the tests are passing)