Using dependency proxy with a multi-architecture upstream repository results in an image with incorrect architecture being pulled

Summary

I've been trying to use dependency proxy to pull images from Docker Hub while building images for multiple architectures (amd64 and arm64) via docker buildx (buildkitd). It seems that when dependency proxy is used, only amd64 images get pulled for the base image regardless of the platform argument passed to docker buildx build. I have constructed a synthetic minimalistic project that reproduces this behavior.

Steps to reproduce

  1. Create a pipeline that builds an image for linux/arm64 architecture with docker buildx using an image available via dependency proxy (using CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX).
  2. See that the base image that is being used (as well as the resulting image, naturally) is actually for linux/amd64.

Example Project

See project https://gitlab.com/atweel/miscellaneous/gitlab-dependency-proxy-with-multi-architecture-images-example — it contains a single Dockerfile that calls arch command and a GitLab pipeline with two jobs that are identical with one exception: base image for the Dockerfile is pulled via dependency proxy in one job and directly from Docker Hub in the other one.

See that job https://gitlab.com/atweel/miscellaneous/gitlab-dependency-proxy-with-multi-architecture-images-example/-/jobs/1933868660 logs "aarch64" as arch command's output, while https://gitlab.com/atweel/miscellaneous/gitlab-dependency-proxy-with-multi-architecture-images-example/-/jobs/1933868659 prints "x86_64".

What is the current bug behavior?

Architecture-wise different images get pulled depending on whether dependency proxy is used.

What is the expected correct behavior?

Dependency proxy provides correct image for the specified architecture.

Relevant logs and/or screenshots

Buildkit output with dependency proxy:

$ docker buildx build --platform ${TARGET_PLATFORM} --build-arg BASE_IMAGE=${BASE_IMAGE}  .
time="2022-01-01T18:56:00Z" level=warning msg="No output specified for kubernetes driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load"
#1 [internal] booting buildkit
#1 waiting for 1 pods to be ready
#1 waiting for 1 pods to be ready 27.8s done
#1 DONE 27.8s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 81B 0.0s done
#2 DONE 0.1s
#3 [internal] load .dockerignore
#3 transferring context: 2B done
#3 DONE 0.0s
#4 [internal] load metadata for gitlab.com:443/atweel/dependency_proxy/containers/debian:buster
#4 ...
#5 [auth] atweel/dependency_proxy/containers/debian:pull token for gitlab.com:443
#5 DONE 0.0s
#4 [internal] load metadata for gitlab.com:443/atweel/dependency_proxy/containers/debian:buster
#4 DONE 3.1s
#6 [1/2] FROM gitlab.com:443/atweel/dependency_proxy/containers/debian:buster@sha256:333b6452b51d512b0fe7c64b2b6bbf06ec433c5c18b749bf495dd0aafcc79093
#6 resolve gitlab.com:443/atweel/dependency_proxy/containers/debian:buster@sha256:333b6452b51d512b0fe7c64b2b6bbf06ec433c5c18b749bf495dd0aafcc79093 done
#6 sha256:9b99af5931b39ce167150ad668cfa57ddf7664697be9996cb7e0e6aebbf05843 0B / 50.44MB 0.2s
#6 sha256:9b99af5931b39ce167150ad668cfa57ddf7664697be9996cb7e0e6aebbf05843 7.34MB / 50.44MB 0.6s
#6 sha256:9b99af5931b39ce167150ad668cfa57ddf7664697be9996cb7e0e6aebbf05843 16.78MB / 50.44MB 0.8s
#6 sha256:9b99af5931b39ce167150ad668cfa57ddf7664697be9996cb7e0e6aebbf05843 30.41MB / 50.44MB 0.9s
#6 sha256:9b99af5931b39ce167150ad668cfa57ddf7664697be9996cb7e0e6aebbf05843 39.85MB / 50.44MB 1.1s
#6 sha256:9b99af5931b39ce167150ad668cfa57ddf7664697be9996cb7e0e6aebbf05843 50.33MB / 50.44MB 1.2s
#6 sha256:9b99af5931b39ce167150ad668cfa57ddf7664697be9996cb7e0e6aebbf05843 50.44MB / 50.44MB 1.9s done
#6 extracting sha256:9b99af5931b39ce167150ad668cfa57ddf7664697be9996cb7e0e6aebbf05843
#6 extracting sha256:9b99af5931b39ce167150ad668cfa57ddf7664697be9996cb7e0e6aebbf05843 1.8s done
#6 DONE 3.7s
#7 [2/2] RUN arch
#7 0.086 x86_64
#7 DONE 0.4s

Buildkit output without dependency proxy:

$ docker buildx build --platform ${TARGET_PLATFORM} --build-arg BASE_IMAGE=${BASE_IMAGE}  .
time="2022-01-01T18:56:04Z" level=warning msg="No output specified for kubernetes driver. Build result will only remain in the build cache. To push result image into registry use --push or to load image into docker use --load"
#1 [internal] booting buildkit
#1 waiting for 1 pods to be ready
#1 waiting for 1 pods to be ready 45.8s done
#1 DONE 45.8s
#2 [internal] load build definition from Dockerfile
#2 transferring dockerfile: 81B 0.0s done
#2 DONE 0.1s
#3 [internal] load .dockerignore
#3 transferring context: 2B 0.0s done
#3 DONE 0.0s
#4 [internal] load metadata for docker.io/library/debian:buster
#4 DONE 0.4s
#5 [1/2] FROM docker.io/library/debian:buster@sha256:94ccfd1c5115a6903cbb415f043a0b04e307be3f37b768cf6d6d3edff0021da3
#5 resolve docker.io/library/debian:buster@sha256:94ccfd1c5115a6903cbb415f043a0b04e307be3f37b768cf6d6d3edff0021da3 0.0s done
#5 sha256:741eb94195433e00f9799629cc66740c97d607d6f3ed207e5738995897c52959 1.05MB / 49.22MB 0.2s
#5 sha256:741eb94195433e00f9799629cc66740c97d607d6f3ed207e5738995897c52959 25.12MB / 49.22MB 0.3s
#5 sha256:741eb94195433e00f9799629cc66740c97d607d6f3ed207e5738995897c52959 33.55MB / 49.22MB 0.5s
#5 sha256:741eb94195433e00f9799629cc66740c97d607d6f3ed207e5738995897c52959 49.22MB / 49.22MB 0.6s done
#5 extracting sha256:741eb94195433e00f9799629cc66740c97d607d6f3ed207e5738995897c52959
#5 extracting sha256:741eb94195433e00f9799629cc66740c97d607d6f3ed207e5738995897c52959 1.7s done
#5 DONE 2.4s
#6 [2/2] RUN arch
#6 0.137 aarch64
#6 DONE 0.5s

Output of checks

This bug happens on GitLab.com

Edited by 🤖 GitLab Bot 🤖