[container registry cleanup job] "invalid tag: missing manifest digest" with external minio storage
We in our company had an omnibus Gitlab installation in docker-compose with all files stored on a local filesystem. We migrated to helm-based installation with external minio object storage (it was not an easy migration, especially the fact that we had to make four secrets with minio credentials and connection details in different formats). Overall Gitlab works with the minio we have: artifacts, backups, packages, uploads are fine. And the container registry seems to work fine, but some issue is there with the container registry cleanup job: after every run we see many "invalid tag: missing manifest digest" messages in the container registry Gitlab UI, that red triangle next to tags. In the minio storage we see that the "directories" of tags (e.g. gitlab-registry/docker/registry/v2/repositories/group
/project
/imagename
/_manifests/tags/3.2.1) present, but with no manifest files inside. After the first cleanup run seems like calculation for the tags to be deleted goes wrong and we end up with all the tags missing manifest digests. We use these cleanup settings: keep 5 tags per image name, remove tags older than 7 days.
We looked into existing issues and found some very similar issues: #393202 and #388865 (closed) but they seem to be related to multi-arch docker images and using versioning in minio storage, with is not our case, the docker images we have are not multi-arch, and minio buckets versioning is off from the beginning. Minio storage is external, an official minio helm chart installed separately from gitlab.
Gitlab helm chart version: gitlab-7.9.1
Gitlab-CE app version: v16.9.1
Minio helm chart version: minio-5.0.15
Minio app version: RELEASE.2024-02-17T01-15-57Z