Investigate: Container registry clean up policy deletes wrong images
Summary
Setting a container registry cleanup policy to keep the most recent 5 tags per image name and letting the clean up do its job removed tags that are more recent than tags that were kept.
A similar issue has been reported (see #351538 (closed)) but that is for tags pointing to the same image. In my case, all tags pointed to different images (as judged by their digest).
Steps to reproduce
- create a project
- push a pile of container images created over a longish time span
My project had two image names with 46 and 30 tags respectively. The first image was created on 2016-09-24. You can see the lists of images before cleanup as well as which ones were kept at paddy-hack/nikola#11 (comment 1129486462) - set a container registry clean up policy
I guess the default will do as well, but in my case I used 5 tags per image name - wait for the cleanup to run
- watch, in horror
😉 , that the expected number of tags remains but more recent ones have been removed
Example Project
This happened with my paddy-hack/nikola project.
It may happen later today with paddy-hack/sphinx and paddy-hack/devuan as well. I'm cleaning out my personal namespace in preparation for the Free Tier 5GB storage limit that comes into effect on or after 2022-10-19
I have copies of all container images in a localhost
registry so I can fix the damage or even restore the registry state to what it was before the cleanup job ran. That will take a while though because I have a slow network connection.
What is the current bug behavior?
The cleanup removes tags that are more recent than those that are kept.
What is the expected correct behavior?
The cleanup policy is satisfied and the most recent tags are kept whereas older tags are removed.
Relevant logs and/or screenshots
See paddy-hack/nikola#11 (comment 1129486462) for observed results.
Output of checks
This bug happens on GitLab.com.
Possible fixes
Sorry, no clue