`notation sign` fails with `MANIFEST_BLOB_UNKNOWN: vnd.oci.image.manifest.v1+json`
Hi folks
This was reported by a GitLab Ultimate customer as a support ticket (internal Zendesk link)
Steps to reproduce
- Build and tag an image
- Push the image to the registry
- Sign the image using Notary:
notation sign registry.gitlab.com/path/to/project:1.1.0
Expected behavior
The pushed image is signed on the repository. Notary is able to verify the image before it's pulled.
Current behavior
# notation sign REDACTED:5050/some/project:1.1.0
Warning: Always sign the artifact using digest(@sha256:...) rather than a tag(:1.1.0) because tags are mutable and a tag reference can point to a different artifact than the one signed.
Error: failed to push signature to registry with error: failed to push manifest: PUT "https://REDACTED:5050/v2/reproduction/zd-501782/manifests/sha256:8d8e1dace56a670f2488bc071ad55e44e105a6ba518d0c91fdd6fb4f31579541": response status code 400: manifest blob unknown: blob unknown to registry: sha256:f58cf8ffab338e45ede447de80fb8a16d63809791067a8e05b82971a79a74647
Server logs
Click to expand
❯ rg "01HQ176YJJMGMC9MPN8CSDMQMA" /var/log/gitlab
/var/log/gitlab/registry/current
14225:2024-02-19_17:39:24.37096 time="2024-02-19T18:39:24.370+01:00" level=info msg="router info" config_http_addr="127.0.0.1:5000" config_http_host= config_http_net= config_http_prefix= config_http_relative_urls=false correlation_id=01HQ176YJJMGMC9MPN8CSDMQMA go_version=go1.21.6 method=PUT path="/v2/reproduction/zd-501782/manifests/sha256:8d8e1dace56a670f2488bc071ad55e44e105a6ba518d0c91fdd6fb4f31579541" root_repo=reproduction router=gorilla/mux vars_name=reproduction/zd-501782 vars_reference="sha256:8d8e1dace56a670f2488bc071ad55e44e105a6ba518d0c91fdd6fb4f31579541" version=v3.88.1-gitlab
14226:2024-02-19_17:39:24.37199 time="2024-02-19T18:39:24.371+01:00" level=info msg="authorized request" auth_project_paths="[reproduction/zd-501782]" auth_user_name=root auth_user_type=personal_access_token correlation_id=01HQ176YJJMGMC9MPN8CSDMQMA go_version=go1.21.6 root_repo=reproduction vars_name=reproduction/zd-501782 vars_reference="sha256:8d8e1dace56a670f2488bc071ad55e44e105a6ba518d0c91fdd6fb4f31579541" version=v3.88.1-gitlab
14227:2024-02-19_17:39:24.37207 time="2024-02-19T18:39:24.372+01:00" level=info msg="payload copied" action="image manifest PUT" auth_project_paths="[reproduction/zd-501782]" auth_user_name=root auth_user_type=personal_access_token content_length=738 copied=738 correlation_id=01HQ176YJJMGMC9MPN8CSDMQMA go_version=go1.21.6 root_repo=reproduction vars_name=reproduction/zd-501782 vars_reference="sha256:8d8e1dace56a670f2488bc071ad55e44e105a6ba518d0c91fdd6fb4f31579541" version=v3.88.1-gitlab
14228:2024-02-19_17:39:24.37225 time="2024-02-19T18:39:24.372+01:00" level=error msg="blob unknown to registry" auth_project_paths="[reproduction/zd-501782]" auth_user_name=root auth_user_type=personal_access_token code=MANIFEST_BLOB_UNKNOWN content_type=application/vnd.oci.image.manifest.v1+json correlation_id=01HQ176YJJMGMC9MPN8CSDMQMA detail="sha256:f58cf8ffab338e45ede447de80fb8a16d63809791067a8e05b82971a79a74647" error="manifest blob unknown: blob unknown to registry" go_version=go1.21.6 host="REDACTED:5050" method=PUT remote_addr=REDACTED root_repo=reproduction uri="/v2/reproduction/zd-501782/manifests/sha256:8d8e1dace56a670f2488bc071ad55e44e105a6ba518d0c91fdd6fb4f31579541" user_agent=notation/1.1.0 vars_name=reproduction/zd-501782 vars_reference="sha256:8d8e1dace56a670f2488bc071ad55e44e105a6ba518d0c91fdd6fb4f31579541" version=v3.88.1-gitlab
14229:2024-02-19_17:39:24.37231 {"content_type":"application/json","correlation_id":"01HQ176YJJMGMC9MPN8CSDMQMA","duration_ms":1,"host":"REDACTED:5050","level":"info","method":"PUT","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:54214","remote_ip":"REDACTED","status":400,"system":"http","time":"2024-02-19T18:39:24.372+01:00","ttfb_ms":1,"uri":"/v2/reproduction/zd-501782/manifests/sha256:8d8e1dace56a670f2488bc071ad55e44e105a6ba518d0c91fdd6fb4f31579541","user_agent":"notation/1.1.0","written_bytes":166}