TTL Policy for the Dependency Proxy

Problem to solve

You can use the GitLab Dependency Proxy to proxy and cache container images from Docker Hub. This can help you by decreasing your reliance on external dependencies and decrease your average build times.

The problem is that if you continue to use the feature the cache will begin to fill up with a lot of stale data. You can always clear the cache using the API. But GitLab should have a standard TTL and eviction policy for clearing the cache so that we don't waste money on storage.

Proposal

Add a daily job that will remove all artifacts from the Dependency Proxy that are more than 90 days old.

Why this is important

We are actively discussing how to grow adoption of this feature, including expanding the scope to allow for pulling container images from Amazon (ECR) and Google's (GCR) container registries. This could result in many more images being added to the cache and object storage.

When a new/unknown artifact is cached at time T, set its TTL to T+90. The artifact should be removed from the cache after 90 days.

Why 90 days

I chose 90 days because that is the most common setting for the Container Registry cleanup policy, as seen in the pie chart below. Also, the majority of policies are set up to run daily.