Container registry push is broken after upgrade to 15.4

Hi GitLab,

Yesterday I've upgraded my personal GL instance from 15.3.2 to 15.4. Everything went fine.

In the morning, I was greeted by an email of failed pipeline of nightly build. I retried pipeline, etc. nothing worked. See the job log: https://pastebin.com/d6rdgVGy

I tried running gitlab-ctl reconfigure, it did not help either.

I also restarted only registry, yet again without any success.

I tailed the logs of registry with gitlab-ctl tail registry:

2022-09-28_09:40:30.05596 time="2022-09-28T09:40:30.055Z" level=error msg="unknown error" auth_user_name= code=UNKNOWN correlation_id=01GE1QG6Q2BRR23PVNN35KW9R4 detail="unlinkat /var/opt/gitlab-registry/storage/docker/registry/v2/repositories/mareshq/internal-static-sites/signpost.mareshq.com/_manifests/tags/main-7cde7fa8/index/sha256/43d2ea4124cd8d34903c5cef22b6f00b1e39cc1b4e3308c7fe20e43c88cd4796/link: permission denied" error="unknown: unknown error" go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.05614 {"content_type":"application/json","correlation_id":"01GE1QG6Q2BRR23PVNN35KW9R4","duration_ms":5,"host":"127.0.0.1:5000","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52176","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2022-09-28T09:40:30.056Z","ttfb_ms":5,"uri":"/v2/mareshq/internal-static-sites/signpost.mareshq.com/tags/reference/main-7cde7fa8","user_agent":"GitLab/15.4.0-ee","written_bytes":332}
2022-09-28_09:40:30.05684 time="2022-09-28T09:40:30.056Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6Q2BRR23PVNMZM0M4K8 go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com vars_reference=main-c9149f0e version=v3.57.0-gitlab
2022-09-28_09:40:30.05726 time="2022-09-28T09:40:30.057Z" level=info msg="manifest downloaded" auth_user_name= correlation_id=01GE1QG6Q2BRR23PVNMZM0M4K8 digest="sha256:85bdb38cf317b0fec42f23477b7b5e554956036e4d87bd8bc3afa720e599b2bf" go_version=go1.17.13 media_type=application/vnd.docker.distribution.manifest.v2+json reference_count=9 root_repo=mareshq size_bytes=1985 tag_name=main-c9149f0e vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com vars_reference=main-c9149f0e version=v3.57.0-gitlab
2022-09-28_09:40:30.05732 {"content_type":"application/vnd.docker.distribution.manifest.v2+json","correlation_id":"01GE1QG6Q2BRR23PVNMZM0M4K8","duration_ms":6,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52172","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2022-09-28T09:40:30.057Z","ttfb_ms":6,"uri":"/v2/mareshq/internal-static-sites/signpost.cloud.mareshq.com/manifests/main-c9149f0e","user_agent":"GitLab/15.4.0-ee","written_bytes":1985}
2022-09-28_09:40:30.06886 time="2022-09-28T09:40:30.068Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6QKBAD44807ENT9PXCT go_version=go1.17.13 root_repo=mareshq vars_digest="sha256:d150969fa03bc5ccdc239852c7426a1b5292db72be59bc7a50e61af88df58806" vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.06966 time="2022-09-28T09:40:30.069Z" level=info msg="blob downloaded" auth_user_name= correlation_id=01GE1QG6QKBAD44807ENT9PXCT digest="sha256:d150969fa03bc5ccdc239852c7426a1b5292db72be59bc7a50e61af88df58806" go_version=go1.17.13 redirect=false root_repo=mareshq size_bytes=9466 vars_digest="sha256:d150969fa03bc5ccdc239852c7426a1b5292db72be59bc7a50e61af88df58806" vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.06978 {"content_type":"application/octet-stream","correlation_id":"01GE1QG6QKBAD44807ENT9PXCT","duration_ms":1,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52180","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2022-09-28T09:40:30.069Z","ttfb_ms":1,"uri":"/v2/mareshq/internal-static-sites/signpost.cloud.mareshq.com/blobs/sha256:d150969fa03bc5ccdc239852c7426a1b5292db72be59bc7a50e61af88df58806","user_agent":"GitLab/15.4.0-ee","written_bytes":9466}
2022-09-28_09:40:30.07596 time="2022-09-28T09:40:30.075Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6QTGC2MQRTR238VVGG6 go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com vars_reference=main-d3a2fcb9 version=v3.57.0-gitlab
2022-09-28_09:40:30.07632 time="2022-09-28T09:40:30.076Z" level=info msg="manifest downloaded" auth_user_name= correlation_id=01GE1QG6QTGC2MQRTR238VVGG6 digest="sha256:27ae85000bbe1b2216f32fc45b998695143c5b5efcef696cb4b7410c3efdaa40" go_version=go1.17.13 media_type=application/vnd.docker.distribution.manifest.v2+json reference_count=9 root_repo=mareshq size_bytes=1985 tag_name=main-d3a2fcb9 vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com vars_reference=main-d3a2fcb9 version=v3.57.0-gitlab
2022-09-28_09:40:30.07639 {"content_type":"application/vnd.docker.distribution.manifest.v2+json","correlation_id":"01GE1QG6QTGC2MQRTR238VVGG6","duration_ms":1,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52186","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2022-09-28T09:40:30.076Z","ttfb_ms":1,"uri":"/v2/mareshq/internal-static-sites/signpost.cloud.mareshq.com/manifests/main-d3a2fcb9","user_agent":"GitLab/15.4.0-ee","written_bytes":1985}
2022-09-28_09:40:30.08593 time="2022-09-28T09:40:30.085Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6R4CBNCRW10J96MTMBJ go_version=go1.17.13 root_repo=mareshq vars_digest="sha256:3ba95a514cf6fff29344521d43a423df9a8ed444e447e40671ffcff41d38ad51" vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.08654 time="2022-09-28T09:40:30.086Z" level=info msg="blob downloaded" auth_user_name= correlation_id=01GE1QG6R4CBNCRW10J96MTMBJ digest="sha256:3ba95a514cf6fff29344521d43a423df9a8ed444e447e40671ffcff41d38ad51" go_version=go1.17.13 redirect=false root_repo=mareshq size_bytes=9466 vars_digest="sha256:3ba95a514cf6fff29344521d43a423df9a8ed444e447e40671ffcff41d38ad51" vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.08664 {"content_type":"application/octet-stream","correlation_id":"01GE1QG6R4CBNCRW10J96MTMBJ","duration_ms":1,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52192","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2022-09-28T09:40:30.086Z","ttfb_ms":1,"uri":"/v2/mareshq/internal-static-sites/signpost.cloud.mareshq.com/blobs/sha256:3ba95a514cf6fff29344521d43a423df9a8ed444e447e40671ffcff41d38ad51","user_agent":"GitLab/15.4.0-ee","written_bytes":9466}
2022-09-28_09:40:30.12014 time="2022-09-28T09:40:30.120Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6S5QKA2T6N6Z5KDQ8N5 go_version=go1.17.13 root_repo=name vars_name=name version=v3.57.0-gitlab
2022-09-28_09:40:30.12046 {"content_type":"","correlation_id":"01GE1QG6S5QKA2T6N6Z5KDQ8N5","duration_ms":2,"host":"127.0.0.1:5000","level":"info","method":"OPTIONS","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52206","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2022-09-28T09:40:30.120Z","ttfb_ms":2,"uri":"/v2/name/tags/reference/tag","user_agent":"GitLab/15.4.0-ee","written_bytes":0}
2022-09-28_09:40:30.12128 time="2022-09-28T09:40:30.121Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6S6XTB8T4Z00KQCWVZH go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/infra.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.12147 {"content_type":"application/json","correlation_id":"01GE1QG6S6XTB8T4Z00KQCWVZH","duration_ms":3,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52220","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2022-09-28T09:40:30.121Z","ttfb_ms":3,"uri":"/v2/mareshq/internal-static-sites/infra.mareshq.com/tags/list","user_agent":"GitLab/15.4.0-ee","written_bytes":100}
2022-09-28_09:40:30.12369 time="2022-09-28T09:40:30.123Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6SAAXM81VJSAYXXV6HR go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.12396 time="2022-09-28T09:40:30.123Z" level=error msg="unknown error" auth_user_name= code=UNKNOWN correlation_id=01GE1QG6SAAXM81VJSAYXXV6HR detail="unlinkat /var/opt/gitlab-registry/storage/docker/registry/v2/repositories/mareshq/internal-static-sites/signpost.cloud.mareshq.com/_manifests/tags/main-7cde7fa8/index/sha256/be2d6e774f15fff552f5231f026322ccd481641b4b59011988fe5e47b668da0e/link: permission denied" error="unknown: unknown error" go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.12409 {"content_type":"application/json","correlation_id":"01GE1QG6SAAXM81VJSAYXXV6HR","duration_ms":1,"host":"127.0.0.1:5000","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52230","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2022-09-28T09:40:30.124Z","ttfb_ms":1,"uri":"/v2/mareshq/internal-static-sites/signpost.cloud.mareshq.com/tags/reference/main-7cde7fa8","user_agent":"GitLab/15.4.0-ee","written_bytes":338}
2022-09-28_09:40:30.12699 time="2022-09-28T09:40:30.126Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6SD3JJKSP0W9D00X1MC go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/infra.mareshq.com vars_reference=main-f1d73a22 version=v3.57.0-gitlab
2022-09-28_09:40:30.12734 time="2022-09-28T09:40:30.127Z" level=info msg="manifest downloaded" auth_user_name= correlation_id=01GE1QG6SD3JJKSP0W9D00X1MC digest="sha256:6095a68a8fb29cb7fe5c04f3880e27ce50a0955562bb6b6fe81ccfbed20f08a0" go_version=go1.17.13 media_type=application/vnd.docker.distribution.manifest.v2+json reference_count=9 root_repo=mareshq size_bytes=1985 tag_name=main-f1d73a22 vars_name=mareshq/internal-static-sites/infra.mareshq.com vars_reference=main-f1d73a22 version=v3.57.0-gitlab
2022-09-28_09:40:30.12738 {"content_type":"application/vnd.docker.distribution.manifest.v2+json","correlation_id":"01GE1QG6SD3JJKSP0W9D00X1MC","duration_ms":1,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52236","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2022-09-28T09:40:30.127Z","ttfb_ms":1,"uri":"/v2/mareshq/internal-static-sites/infra.mareshq.com/manifests/main-f1d73a22","user_agent":"GitLab/15.4.0-ee","written_bytes":1985}
2022-09-28_09:40:30.13733 time="2022-09-28T09:40:30.137Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6SPE2PY1HQQDHKYSZPE go_version=go1.17.13 root_repo=mareshq vars_digest="sha256:8555bf1f537722da02c6c1ead67d62c0c151e27db46f27caa549c6ad4be86cfd" vars_name=mareshq/internal-static-sites/infra.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.13901 time="2022-09-28T09:40:30.137Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6SPE2PY1HQQDH9JRC2F go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.13902 time="2022-09-28T09:40:30.138Z" level=info msg="blob downloaded" auth_user_name= correlation_id=01GE1QG6SPE2PY1HQQDHKYSZPE digest="sha256:8555bf1f537722da02c6c1ead67d62c0c151e27db46f27caa549c6ad4be86cfd" go_version=go1.17.13 redirect=false root_repo=mareshq size_bytes=9466 vars_digest="sha256:8555bf1f537722da02c6c1ead67d62c0c151e27db46f27caa549c6ad4be86cfd" vars_name=mareshq/internal-static-sites/infra.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.13903 time="2022-09-28T09:40:30.138Z" level=error msg="unknown error" auth_user_name= code=UNKNOWN correlation_id=01GE1QG6SPE2PY1HQQDH9JRC2F detail="unlinkat /var/opt/gitlab-registry/storage/docker/registry/v2/repositories/mareshq/internal-static-sites/signpost.cloud.mareshq.com/_manifests/tags/main-71308134/index/sha256/37da8335e3ac7778e9335979068aee8034f2a836d7555f68677e019d414b91b7/link: permission denied" error="unknown: unknown error" go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.13904 {"content_type":"application/json","correlation_id":"01GE1QG6SPE2PY1HQQDH9JRC2F","duration_ms":3,"host":"127.0.0.1:5000","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52250","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2022-09-28T09:40:30.138Z","ttfb_ms":3,"uri":"/v2/mareshq/internal-static-sites/signpost.cloud.mareshq.com/tags/reference/main-71308134","user_agent":"GitLab/15.4.0-ee","written_bytes":338}
2022-09-28_09:40:30.13904 {"content_type":"application/octet-stream","correlation_id":"01GE1QG6SPE2PY1HQQDHKYSZPE","duration_ms":3,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52262","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2022-09-28T09:40:30.138Z","ttfb_ms":2,"uri":"/v2/mareshq/internal-static-sites/infra.mareshq.com/blobs/sha256:8555bf1f537722da02c6c1ead67d62c0c151e27db46f27caa549c6ad4be86cfd","user_agent":"GitLab/15.4.0-ee","written_bytes":9466}
2022-09-28_09:40:30.14333 time="2022-09-28T09:40:30.143Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6SY5Q7PNDDVYQJS6RTG go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.14351 time="2022-09-28T09:40:30.143Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6SY5Q7PNDDVYTMRB1X2 go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/infra.mareshq.com vars_reference=main-fa96a47b version=v3.57.0-gitlab
2022-09-28_09:40:30.14355 time="2022-09-28T09:40:30.143Z" level=error msg="unknown error" auth_user_name= code=UNKNOWN correlation_id=01GE1QG6SY5Q7PNDDVYQJS6RTG detail="unlinkat /var/opt/gitlab-registry/storage/docker/registry/v2/repositories/mareshq/internal-static-sites/signpost.cloud.mareshq.com/_manifests/tags/main-8f0ed25d/index/sha256/31b6cf33661c9c29b6f91dfe2aef254c51c45f6600930d44dc3dc6f98020cfac/link: permission denied" error="unknown: unknown error" go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.14361 {"content_type":"application/json","correlation_id":"01GE1QG6SY5Q7PNDDVYQJS6RTG","duration_ms":1,"host":"127.0.0.1:5000","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52272","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2022-09-28T09:40:30.143Z","ttfb_ms":1,"uri":"/v2/mareshq/internal-static-sites/signpost.cloud.mareshq.com/tags/reference/main-8f0ed25d","user_agent":"GitLab/15.4.0-ee","written_bytes":338}
2022-09-28_09:40:30.14390 time="2022-09-28T09:40:30.143Z" level=info msg="manifest downloaded" auth_user_name= correlation_id=01GE1QG6SY5Q7PNDDVYTMRB1X2 digest="sha256:0bf64efec9780106de3c8f8cd5ea8bcc5108bced01e5759838a0b7931d8ee74a" go_version=go1.17.13 media_type=application/vnd.docker.distribution.manifest.v2+json reference_count=9 root_repo=mareshq size_bytes=1985 tag_name=main-fa96a47b vars_name=mareshq/internal-static-sites/infra.mareshq.com vars_reference=main-fa96a47b version=v3.57.0-gitlab
2022-09-28_09:40:30.14393 {"content_type":"application/vnd.docker.distribution.manifest.v2+json","correlation_id":"01GE1QG6SY5Q7PNDDVYTMRB1X2","duration_ms":1,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52274","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2022-09-28T09:40:30.143Z","ttfb_ms":1,"uri":"/v2/mareshq/internal-static-sites/infra.mareshq.com/manifests/main-fa96a47b","user_agent":"GitLab/15.4.0-ee","written_bytes":1985}
2022-09-28_09:40:30.14702 time="2022-09-28T09:40:30.146Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6T18CCW5A0AXJ2ZNP3R go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.14726 time="2022-09-28T09:40:30.147Z" level=error msg="unknown error" auth_user_name= code=UNKNOWN correlation_id=01GE1QG6T18CCW5A0AXJ2ZNP3R detail="unlinkat /var/opt/gitlab-registry/storage/docker/registry/v2/repositories/mareshq/internal-static-sites/signpost.cloud.mareshq.com/_manifests/tags/main-3805db9d/index/sha256/b9523db2c3a1977a47b384ce03233fd1d64dfca7db2d87e284d14573c46b18c7/link: permission denied" error="unknown: unknown error" go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.14732 {"content_type":"application/json","correlation_id":"01GE1QG6T18CCW5A0AXJ2ZNP3R","duration_ms":1,"host":"127.0.0.1:5000","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52286","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2022-09-28T09:40:30.147Z","ttfb_ms":1,"uri":"/v2/mareshq/internal-static-sites/signpost.cloud.mareshq.com/tags/reference/main-3805db9d","user_agent":"GitLab/15.4.0-ee","written_bytes":338}
2022-09-28_09:40:30.14937 time="2022-09-28T09:40:30.149Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6T4MJDA4HP17B8C2SWG go_version=go1.17.13 root_repo=mareshq vars_digest="sha256:3dd138d8061ac38433a6665d3b793775cfa73c5a66666404af3f784c0a6df6b4" vars_name=mareshq/internal-static-sites/infra.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.15119 time="2022-09-28T09:40:30.151Z" level=info msg="blob downloaded" auth_user_name= correlation_id=01GE1QG6T4MJDA4HP17B8C2SWG digest="sha256:3dd138d8061ac38433a6665d3b793775cfa73c5a66666404af3f784c0a6df6b4" go_version=go1.17.13 redirect=false root_repo=mareshq size_bytes=9464 vars_digest="sha256:3dd138d8061ac38433a6665d3b793775cfa73c5a66666404af3f784c0a6df6b4" vars_name=mareshq/internal-static-sites/infra.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.15133 {"content_type":"application/octet-stream","correlation_id":"01GE1QG6T4MJDA4HP17B8C2SWG","duration_ms":3,"host":"127.0.0.1:5000","level":"info","method":"GET","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52288","remote_ip":"127.0.0.1","status":200,"system":"http","time":"2022-09-28T09:40:30.151Z","ttfb_ms":1,"uri":"/v2/mareshq/internal-static-sites/infra.mareshq.com/blobs/sha256:3dd138d8061ac38433a6665d3b793775cfa73c5a66666404af3f784c0a6df6b4","user_agent":"GitLab/15.4.0-ee","written_bytes":9464}
2022-09-28_09:40:30.15144 time="2022-09-28T09:40:30.151Z" level=info msg="authorized request" auth_user_name= correlation_id=01GE1QG6T53WV116AH2JTJA0HY go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.15165 time="2022-09-28T09:40:30.151Z" level=error msg="unknown error" auth_user_name= code=UNKNOWN correlation_id=01GE1QG6T53WV116AH2JTJA0HY detail="unlinkat /var/opt/gitlab-registry/storage/docker/registry/v2/repositories/mareshq/internal-static-sites/signpost.cloud.mareshq.com/_manifests/tags/main-d3a2fcb9/index/sha256/27ae85000bbe1b2216f32fc45b998695143c5b5efcef696cb4b7410c3efdaa40/link: permission denied" error="unknown: unknown error" go_version=go1.17.13 root_repo=mareshq vars_name=mareshq/internal-static-sites/signpost.cloud.mareshq.com version=v3.57.0-gitlab
2022-09-28_09:40:30.15172 {"content_type":"application/json","correlation_id":"01GE1QG6T53WV116AH2JTJA0HY","duration_ms":2,"host":"127.0.0.1:5000","level":"info","method":"DELETE","msg":"access","proto":"HTTP/1.1","referrer":"","remote_addr":"127.0.0.1:52294","remote_ip":"127.0.0.1","status":500,"system":"http","time":"2022-09-28T09:40:30.151Z","ttfb_ms":2,"uri":"/v2/mareshq/internal-static-sites/signpost.cloud.mareshq.com/tags/reference/main-d3a2fcb9","user_agent":"GitLab/15.4.0-ee","written_bytes":338}

A different project with repository cleanup policy is stuck deleting old tags which to me looks like GitLab is DoSing it's own registry.

It is clear that there is an authorization issue, but this should be ok since it is using $CI_REGISTRY_PASSWORD. So it is GitLab-managed.

This is the current cleanup policy:

Registry configuration from /etc/gitlab/gitlab.rb:

registry_external_url 'https://registry.example.com' # redacted domain name

gitlab_rails['registry_enabled'] = true
gitlab_rails['registry_path'] = "/var/opt/gitlab-registry/storage"

UPDATE: I fixed the issue.

How to fix

I am using custom registry storage path (it is sitting on external volume mounted to the VM), which you can see in the configuration.

After running the latest upgrade, the ownership on /var/opt/gitlab-registry/storage dir were registry:git. After changing them with chown -R registry:root /var/opt/gitlab-registry/storage, the issue was resolved.

I've checked the ownership on the files in the /var/opt/gitlab/registry dir and files in it.

Conclusion

Looks like it is an issue with Chef and upgrades.

Final note

I will leave this as an issue for now for GitLab to decide if this is a bug or not.