Failed to pull image: unexpected commit ref

Summary

We currently facing the following issues with some of our Gitlab Pipelines since we are using the Gitlab Dependency Proxy:

ERROR: Job failed: prepare environment: waiting for pod running: pulling image "gitlab.example.com:443/group/group/dependency_proxy/containers/docker:20.10.6": image pull failed: rpc error: code = FailedPrecondition desc = failed to pull and unpack image "gitlab.example.com:443/group/group/dependency_proxy/containers/docker:20.10.6": failed commit on ref "unknown-sha256:db5b05108e6c9500f77547372ab8c65fc0b2525b03768d54098b5ff863f70072": unexpected commit digest sha256:a53fc07fed1c58d0fbfe533c8ce0628a89d457ae60c2b6fab9655d1114338378, expected sha256:db5b05108e6c9500f77547372ab8c65fc0b2525b03768d54098b5ff863f70072: failed precondition. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Steps to reproduce

Run a pipeline.

.gitlab-ci.yml 
job:
  tags:
    - dind
  services:
    - name: ${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/docker:20.10.16-dind
      command: ["--mtu=1300"]
  before_script:
    # Wait until the docker certificate is created otherwise we sometimes see:
    # "unable to resolve docker endpoint: open /certs/client/ca.pem: no such file or directory"
    - sleep 3
    - docker login -u $CI_DEPENDENCY_PROXY_USER -p $CI_DEPENDENCY_PROXY_PASSWORD $CI_DEPENDENCY_PROXY_SERVER
    - if [[ -z "$DOCKER_HUB_REGISTRY_PASSWORD" || -z "$DOCKER_HUB_REGISTRY_USER" ]]; then echo "Variables for docker hub login not set. You might run into issues with the docker hub rate limit..."; else docker login -u "$DOCKER_HUB_REGISTRY_USER" -p "$DOCKER_HUB_REGISTRY_PASSWORD"; fi
  script:
    - echo "Hello World"

Actual behavior

Using the Dependency Proxy causes that most of the pipeline runs failing. Removing the Dependency Proxy works fine.

Expected behavior

Images can be pulled via Dependency Proxy to avoid rate limiting

Relevant logs and/or screenshots

job log 
unning with gitlab-runner 15.9.1 (d540b510)
  on gitlab-runner-5d966b5ff5-q2frb Dhr23P3n, system ID: r_4dcQJgH9PFB7
Resolving secrets
00:00
Preparing the "kubernetes" executor
00:00
"MemoryLimit" overwritten with "4Gi"
Using Kubernetes namespace: gitlab-runner
Using Kubernetes executor with image gitlab.example.com:443/group/group/dependency_proxy/containers/docker:20.10.6 ...
Using attach strategy to execute scripts...
Preparing environment
00:07
Waiting for pod gitlab-runner/runner-dhr23p3n-project-279-concurrent-06zpvt to be running, status is Pending
Waiting for pod gitlab-runner/runner-dhr23p3n-project-279-concurrent-06zpvt to be running, status is Pending
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
	ContainersNotReady: "containers with unready status: [build helper svc-0]"
WARNING: Failed to pull image with policy "IfNotPresent": image pull failed: rpc error: code = FailedPrecondition desc = failed to pull and unpack image "gitlab.example.com:443/group/group/dependency_proxy/containers/docker:20.10.6": failed commit on ref "unknown-sha256:db5b05108e6c9500f77547372ab8c65fc0b2525b03768d54098b5ff863f70072": unexpected commit digest sha256:a53fc07fed1c58d0fbfe533c8ce0628a89d457ae60c2b6fab9655d1114338378, expected sha256:db5b05108e6c9500f77547372ab8c65fc0b2525b03768d54098b5ff863f70072: failed precondition
ERROR: Job failed: prepare environment: waiting for pod running: pulling image "gitlab.example.com:443/group/group/dependency_proxy/containers/docker:20.10.6": image pull failed: rpc error: code = FailedPrecondition desc = failed to pull and unpack image "gitlab.example.com:443/group/group/dependency_proxy/containers/docker:20.10.6": failed commit on ref "unknown-sha256:db5b05108e6c9500f77547372ab8c65fc0b2525b03768d54098b5ff863f70072": unexpected commit digest sha256:a53fc07fed1c58d0fbfe533c8ce0628a89d457ae60c2b6fab9655d1114338378, expected sha256:db5b05108e6c9500f77547372ab8c65fc0b2525b03768d54098b5ff863f70072: failed precondition. Check https://docs.gitlab.com/runner/shells/index.html#shell-profile-loading for more information

Environment description

config.toml contents 
[[runners]]
  environment = [
    "DOCKER_HOST=tcp://localhost:2376",
    "DOCKER_TLS_CERTDIR=/certs",
    "DOCKER_TLS_VERIFY=1",
    "DOCKER_CERT_PATH=/certs/client",
  ]
 [runners.kubernetes]
   image = "ubuntu:20.04"
   pull_policy = "if-not-present"
   poll_timeout = 600
   helper_cpu_limit = "200m"
   helper_image_flavor = "ubuntu"
   helper_cpu_request = "100m"
   helper_cpu_request_overwrite_max_allowed = "2000m"
   helper_memory_limit = "256Mi"
   helper_memory_request = "128Mi"
   cpu_limit = "4000m"
   cpu_limit_overwrite_max_allowed = "2000m"
   memory_limit = "2048Mi"
   memory_limit_overwrite_max_allowed = "4Gi"
   cpu_request = "750m"
   cpu_request_overwrite_max_allowed = "2000m"
   memory_request = "1200Mi"
   memory_request_overwrite_max_allowed = "4Gi"
   resource_availability_check_max_attempts = 5
   privileged = true
   [[runners.kubernetes.volumes.empty_dir]]
     name = "docker-certs"
     mount_path = "/certs/client"
     medium = "Memory"
   [[runners.kubernetes.volumes.empty_dir]]
     name = "dind-storage"
     mount_path = "/var/lib/docker"
   [[runners.kubernetes.volumes.host_path]]
     name = "hostpath-modules"
     mount_path = "/lib/modules"
     read_only = true
     host_path = "/lib/modules"
   [[runners.kubernetes.volumes.host_path]]
     name = "hostpath-cgroup"
     mount_path = "/sys/fs/cgroup"
     host_path = "/sys/fs/cgroup"
   [runners.cache]
     Type = "s3"
     Path = "gitlab_runner"
     Shared = true
     [runners.cache.s3]
       ServerAddress = "s3.amazonaws.com"
       BucketName = "gitlab-runner-cache"
       BucketLocation = "eu-central-1"
       Insecure = false
       AuthenticationType = "access-key"

Used GitLab Runner version

Version:      15.9.1
Git revision: d540b510
Git branch:   15-9-stable
GO version:   go1.18.10
Built:        2023-02-20T21:03:05+0000
OS/Arch:      linux/amd64

Possible fixes

Edited by Dominic Cardellino