pushing manifest to registry gets error "manifest blob unknown"

Summary

Pushing a docker multi-arch manifest list errors out with

manifest blob unknown: blob unknown to registry

Steps to reproduce

1. Create several multi-arch images (amd64, arm64v8, arm32v7, etc)
   • Naming scheme $IMAGE/$ARCH:$COMMIT (ie, registry.gitlab.com/paleozogt/dockerfun/amd64:e99c4336).
   • Dockerfile:

     ARG ARCH
     FROM $ARCH/ubuntu:bionic

2. Push the images
3. Use docker manifest create to pull all images together into an image named $IMAGE:$COMMIT (ie, registry.gitlab.com/paleozogt/dockerfun:e99c4336)
4. Use docker manifest push to push the manifest list

Curiously, if we use a different naming scheme of $IMAGE:$ARCH-$COMMIT (ie, registry.gitlab.com/paleozogt/dockerfun:amd64-6b972be0) it will work.

Example Project

An example project showing the problem is here:

• Branch with multi-arch failing:
  • code: https://gitlab.com/paleozogt/dockerfun/-/tree/multiarch_bad
  • pipeline: https://gitlab.com/paleozogt/dockerfun/pipelines/123362305
• Branch with multi-arch working:
  • code: https://gitlab.com/paleozogt/dockerfun/-/tree/multiarch_good
  • pipeline: https://gitlab.com/paleozogt/dockerfun/pipelines/123361885

What is the current bug behavior?

Pushing the multi-arch manifest fails even thought the images are in the repository.

What is the expected correct behavior?

Pushing the multi-arch manifest should succeed no matter the naming scheme used, as long as the images are in the repository.

Relevant logs and/or screenshots

$ docker manifest push $CI_REGISTRY_IMAGE:$CI_COMMIT_SHORT_SHA
 Pushed ref registry.gitlab.com/paleozogt/dockerfun@sha256:05f7a02dff801b3c0e18b9588b63772f647aa068f61bca0b77d547525c01f571 with digest: sha256:1e43bde8fda64de203881785cbd7be4efe7cba4f6dbd435a21f9423762fae6f1
 Pushed ref registry.gitlab.com/paleozogt/dockerfun@sha256:0b5c73ae2be5480753ccffc7c585c4526de2c8f01f81c2276bc96e7cfcd5c3b6 with digest: sha256:34962a4a4fc7a3134cd671a4c137eb4945f4361cab4e24341685828ee5187964
 Pushed ref registry.gitlab.com/paleozogt/dockerfun@sha256:0da3d97dc96540fd02918d63157911168cd660119c02f91150621afdf85e1668 with digest: sha256:00fab59b2048a51ef7c49785effffbcfd9eff7aadaa55e08cf03f1b0cd60431a
 failed to put manifest registry.gitlab.com/paleozogt/dockerfun:e99c4336: errors:
 manifest blob unknown: blob unknown to registry
 manifest blob unknown: blob unknown to registry
 manifest blob unknown: blob unknown to registry
 ERROR: Job failed: exit code 1

Output of checks

This bug happens on GitLab.com as well as my Omnibus installation.