Skip to content

Remove Docker images from the GitLab Container Registry using a project level policy

Problem to solve

Users are building many Docker images as part of their pipelines. Many of these images are only needed for a short time. There isn't a good way for developers to delete these images, so they frequently don't. This results in either ballooning storage costs, or the Admin trying to remove images manually via the Container Registry API, which is error prone and inefficient.

User stories

  • I as a systems administrator, need to ensure that my development teams are expiring unused images, so that I can run garbage collection, delete them from storage and lower the cost of storage.
  • I as a developer, need the ability to adjust my project's and Docker repositories' expiration policy, so that I can ensure that my important images (ones required for a given release) are not accidentally removed.

Intended users

Further details

Goals

  • Our self-managed customers can set (and forget) a reasonable expiration policy that is enforced by default for every new project in their organization.
  • GitLab.com enforces expiration policies on all new projects, so that more images are eligible for garbage collection, so that we can, once online garbage collection is available, greatly reduce the cost of storage for the gitlab.com Container Registry.

Proposal

MVC

Introduce a new user interface that will allow project owners to view and update tag expiration policies for all new projects.

  • Project owners may enable / disable the feature at the project level in Project--> Settings --> CI/CD --> Tag Expiration Policies
  • Allow project owners to update the default policy at the project level for all Docker repositories in the project:
    • Expiration interval
    • Expiration schedule
    • Retain at least n tags
    • Expire tags matching the name (regex allowed)
User Interface (These designs are still in motion and are subject to change)

CI/CD Settings --> Docker Tag Expiration Policies docker_expiration_policy

Docker Repository view (not included in this issue) image

What we are not doing in the MVC

  • It's important to note that this issue does not include tag retention policies. This is the because we are building this feature using the existing GitLab bulk delete API, which does not specify which tags to keep. We will do this, but it won't be included in the MVC.
  • The MVC will not include policies at the Docker repository level, that will be done with: #37242
Future Plans (beyond the MVC)
Expiration Policies
  1. Leverage Docker tag expiration policies at repository level
  2. Use the GitLab API to define protected images to prevent them from being deleted during garbage collection
  3. Leverage the GitLab API to define and update Docker tag expiration policies
  4. Enable multiple policies for expiring Docker images from the GitLab Container Registry
Retention Policies
  1. Protect Docker images from being deleted by setting a retention policy
Availability
  1. Expand Docker tag expiration and retention policies to all existing projects

Open questions

  • Are there any performance concerns for this issue?
  • What defaults should we provide?
  • How do we handle scheduling acting on policies?

Permissions and Security

  • Developers and above can update the tag expiration rules at the Docker repository level
  • Project owners can enable/disable the feature and adjust project level rules

Documentation

Testing

  • We will have to test performance and functionality.

What does success look like, and how can we measure that?

  • We help our customers and ourselves lower the cost of storage for the Container Registry.
  • We create greater visibility and awareness of Container Registry storage management features.
  • We understand if this can be rolled out to existing projects.
  • We learn from the MVC and continue to iterate on this valuable feature.

Data collection / measurement

  • number of new projects created / number of new projects with tag expiration policies enabled
  • count # of new projects with tag expiration policies disabled

Links

Edited by Tim Rizzi