Remove Docker images from the GitLab Container Registry using a project level policy

Problem to solve

Users are building many Docker images as part of their pipelines. Many of these images are only needed for a short time. There isn't a good way for developers to delete these images, so they frequently don't. This results in either ballooning storage costs, or the Admin trying to remove images manually via the Container Registry API, which is error prone and inefficient.

User stories

I as a systems administrator, need to ensure that my development teams are expiring unused images, so that I can run garbage collection, delete them from storage and lower the cost of storage.
I as a developer, need the ability to adjust my project's and Docker repositories' expiration policy, so that I can ensure that my important images (ones required for a given release) are not accidentally removed.

Intended users

Further details

Goals

Our self-managed customers can set (and forget) a reasonable expiration policy that is enforced by default for every new project in their organization.
GitLab.com enforces expiration policies on all new projects, so that more images are eligible for garbage collection, so that we can, once online garbage collection is available, greatly reduce the cost of storage for the gitlab.com Container Registry.

Proposal

MVC

Introduce a new user interface that will allow project owners to view and update tag expiration policies for all new projects.

Project owners may enable / disable the feature at the project level in Project--> Settings --> CI/CD --> Tag Expiration Policies
Allow project owners to update the default policy at the project level for all Docker repositories in the project:
- Expiration interval
- Expiration schedule
- Retain at least n tags
- Expire tags matching the name (regex allowed)

User Interface (These designs are still in motion and are subject to change)

CI/CD Settings --> Docker Tag Expiration Policies

Docker Repository view (not included in this issue)

What we are not doing in the MVC

It's important to note that this issue does not include tag retention policies. This is the because we are building this feature using the existing GitLab bulk delete API, which does not specify which tags to keep. We will do this, but it won't be included in the MVC.
The MVC will not include policies at the Docker repository level, that will be done with: #37242

Future Plans (beyond the MVC)

Expiration Policies

Retention Policies

Protect Docker images from being deleted by setting a retention policy

Availability

Expand Docker tag expiration and retention policies to all existing projects

Open questions

Are there any performance concerns for this issue?
What defaults should we provide?
How do we handle scheduling acting on policies?

Permissions and Security

Developers and above can update the tag expiration rules at the Docker repository level
Project owners can enable/disable the feature and adjust project level rules

Documentation

Testing

We will have to test performance and functionality.

What does success look like, and how can we measure that?

We help our customers and ourselves lower the cost of storage for the Container Registry.
We create greater visibility and awareness of Container Registry storage management features.
We understand if this can be rolled out to existing projects.
We learn from the MVC and continue to iterate on this valuable feature.

Data collection / measurement

number of new projects created / number of new projects with tag expiration policies enabled
count # of new projects with tag expiration policies disabled