Retention/expiration policy for container images
Problem to solve
The GitLab Container Registry allows users to build, publish and share Docker images using the Docker client or programmatically via their pipelines. For organizations that push many images via CI, it is critical that they are able to delete these images programmatically, so that they lower the cost of storage and ensure that images are easy to find and verify.
In order to delete an image from storage, the image must first be untagged, using either the Container Registry UI or the API. Then, instance owners must run garbage collection to remove the images and manifests from storage.
We learned in a recent round of user research that what administrators really need is the ability to set and define policies at the instance level that are enforced (although that can be overwritten if necessary) at the group and project level. We also learned that developers although not against cleaning up old images, may not think about it.
- I as a systems administrator, need the ability to set, view and update retention and expiration policies for the Gitlab Container Registry, so that I can not waste storage on old, unused images and ensure that each group and project are following my organization's best practices.
- Default settings are applied at the instance level
- Groups and projects inherit instance settings
- Utilize regex for defining rules
- Groups and projects can update the regex for which images to keep, but not the expiration time for images that will be deleted.
- Admin can define when to run this background job and monitor it as needed
- For regex, we will start with the same options as in the bulk delete API
- Removes only tags matching the given name_regex
- Never remove the tag named latest
- Keep N latest matching tags (if keep_n is specified)
- Only remove tags that are older than X amount of time (if older_than is specified)
- I as a developer, need the ability to view the container registry retention policies set by my instance administrator, for my group or project, and be able to adjust the name regex, so I can ensure that my release images do not get unintentionally removed.
- Delaney (Development Team Lead)
- Sasha (Software Developer)
- Devon (DevOps Engineer)
- Sidney (Systems Administrator)
- Our self-managed customers can set (and forget) reasonable expiration policies that are enforced by default by every project in their organization.
- GitLab.com enforces expiration policies, marking the many, many unused images currently being stored ready for deletion. And (once online garbage collection is available), greatly reducing the cost of storage for the gitlab.com Container Registry.
- Allow administrators to use regex to define container registry expiration policies as part of the
registrysection of their gitlab.rb file.
- Allow developers to view these policies in their group and/or project.
- Allow developers to update the name regex for their group and/or project.
- gitlab.rb regex
registry: enabled: true ... gc: retention_period: 30d ignore_tags: - latest - production-*
Permissions and Security
- Developers and above can update the name regex for a group/project they have permission to
- Instance Owners can set regex based policies in the gitlab.rb file
- We are currently working to improve the performance of the image removal process, and since this will likely have many images to remove when first implemented, we should ensure that the performance meets our standards.
What does success look like, and how can we measure that?
- Success looks like we can help our customers and ourselves lower the cost of storage for the Container Registry.
- We can measure this for gitlab.com (once we can run garbage collection) by looking at the overall cost of storage.
- We can also measure the total number of images over time and we should see a significant decrease in the that number and then see it level off over time.