jib - upload of docker images to gitlab container registry gives 'blob unknown to registry'
Summary
Upload of docker images with jib to the gitlab container registry gives 'blob unknown to registry'. This started last night out of a sudden (while working fine for 2 years) after a recent update (current version GitLab Enterprise Edition v16.2.4-ee) has been installed.
Interestingly: Configuring jib to use goal "dockerBuild" (means: utilize the docker daemon as opposed to create layers and upload them) makes the build run fine. An explicit "docker push" afterwards also works fine. The goal "dockerBuild" also implies no multiarch-build. It only builds one architecture then (amd64 in our case).
Yet it does not seem related to multiarch build itself because turning it off while using the regular jib "build" goal still fails with the same message. Only switching to docker daemon does help.
That may be related to docker daemon being slower and doing work (uploads) serially (though we also made jib work serially, because of earlier issues with uploads).
EDIT: I was able to reproduce the error locally. A big workspace with 15 maven jib projects were built in parallel and pushed to gitlab container reg. Only some failed with the mentioned error message (basically 2 projects failed). Interestingly: When I did some subsequent runs for these projects they finally were successful. And after that even the whole rebuild was successful. It seems to me that pushing several layers in a short time leads to this error.
Steps to reproduce
Create a multiarch (builds for amd64+arm64) maven jib build project. On image upload to gitlab container registry it fails with the given message.
Example Project
Not easily created. Can try if neccessary for investigation.
What is the current bug behavior?
Build job fails with the error output attached below and HTTP 400 Bad Request.
What is the expected correct behavior?
Image uploads should succeed as did before.
Relevant logs and/or screenshots
[INFO] BUILD FAILURE
[INFO] ------------------------------------------------------------------------
[INFO] Total time: 04:02 min
[INFO] Finished at: 2023-08-16T17:40:12Z
[INFO] ------------------------------------------------------------------------
[ERROR] Failed to execute goal com.google.cloud.tools:jib-maven-plugin:3.3.1:build (build-container-image) on project cae-live-app: Tried to push image manifest for <our-gitlab-server>/v2/<our-project>/<image-name>:develop but failed because: other: blob unknown to registry: 400 Bad Request
[ERROR] PUT https://<our-gitlab-server>/v2/<our-project>/<image-name>/manifests/develop
[ERROR] {"errors":[{"code":"MANIFEST_BLOB_UNKNOWN","message":"blob unknown to registry","detail":"sha256:0ff79a342f8f291fdf1602eb098f8af35f356de674a6ff6a2ab337942eb05d64"}]}
[ERROR] -> [Help 1]
[ERROR]
[ERROR] To see the full stack trace of the errors, re-run Maven with the -e switch.
[ERROR] Re-run Maven using the -X switch to enable full debug logging.Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: \\\\\\\\\\\\\\\`sudo gitlab-rake gitlab:env:info\\\\\\\\\\\\\\\`) (For installations from source run and paste the output of: \\\\\\\\\\\\\\\`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production\\\\\\\\\\\\\\\`)
I don't have access to this info. I can check back with Ops if neccessary.
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of: \\\\\\\`sudo gitlab-rake gitlab:check SANITIZE=true\\\\\\\`) (For installations from source run and paste the output of: \\\\\\\`sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true\\\\\\\`) (we will only investigate if the tests are passing)
I don't have access to this info. I can check back with Ops if neccessary.