Dependency proxy is unable to retrieve container images

Summary

Starting from yesterday morning all our pipelines are failing randomly due to an errors like:

failed to solve with frontend dockerfile.v0: failed to create LLB definition: gitlab.com:443/musement/dependency_proxy/containers/php:7.3-fpm-buster: not found
failed to solve with frontend dockerfile.v0: failed to create LLB definition: gitlab.com:443/musement/dependency_proxy/containers/composer:2.0: not found

Steps to reproduce

We just build in pipeline docker images using a dockerfile like:

ARG IMAGE_PREFIX=''

FROM ${IMAGE_PREFIX}composer:2.0 AS composer_img
FROM ${IMAGE_PREFIX}php:7.3-fpm-buster

COPY --from=composer_img /usr/bin/composer /usr/local/bin/composer

and a pipeline job like:

build:
  stage: docker-build
  before_script:
    - docker login -u "${CI_DEPENDENCY_PROXY_USER}" -p "${CI_DEPENDENCY_PROXY_PASSWORD}" "${CI_DEPENDENCY_PROXY_SERVER}"
  script:
    - >
      docker build \
        --file docker/Dockerfile \
        --build-arg IMAGE_PREFIX="${CI_DEPENDENCY_PROXY_GROUP_IMAGE_PREFIX}/" \
        --tag ${PHP_DOCKER_IMAGE} \
      .
    - docker push ${PHP_DOCKER_IMAGE}

What is the current bug behavior?

After the docker build command starts it randomly fails on one of the 2 images used

$ docker build \ # collapsed multi-line command
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 6.43kB done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 96B done
#2 DONE 0.0s
#4 [internal] load metadata for gitlab.com:443/musement/dependency_proxy/co...
#4 ERROR: gitlab.com:443/musement/dependency_proxy/containers/composer:2.0: not found
#3 [internal] load metadata for gitlab.com:443/musement/dependency_proxy/co...
#3 CANCELED
------
 > [internal] load metadata for gitlab.com:443/musement/dependency_proxy/containers/composer:2.0:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: gitlab.com:443/musement/dependency_proxy/containers/composer:2.0: not found
$ docker build \ # collapsed multi-line command
#1 [internal] load build definition from Dockerfile
#1 transferring dockerfile: 6.43kB done
#1 DONE 0.0s
#2 [internal] load .dockerignore
#2 transferring context: 96B done
#2 DONE 0.0s
#4 [internal] load metadata for gitlab.com:443/musement/dependency_proxy/co...
#4 CANCELED
#3 [internal] load metadata for gitlab.com:443/musement/dependency_proxy/co...
#3 ERROR: gitlab.com:443/musement/dependency_proxy/containers/php:7.3-fpm-buster: not found
------
 > [internal] load metadata for gitlab.com:443/musement/dependency_proxy/containers/php:7.3-fpm-buster:
------
failed to solve with frontend dockerfile.v0: failed to create LLB definition: gitlab.com:443/musement/dependency_proxy/containers/php:7.3-fpm-buster: not found

What is the expected correct behavior?

Never had any kind of issues till 2 days ago

Edited by Alberto Barba