Throttling for Cleanup policy for tags
Problem to solve
Currently, the Cleanup policy for tags feature is only allowed for projects created after 12.8.
We would like to turn on the feature for all projects, but there are performance concerns because tag deletion can be slow.
Intended users
Further details
Proposal
When we turn on the policy feature for all projects, we need a way to prevent the sidekiq queue from backing up and slowing things to a halt, especially if there are cases when there are large amounts of tags to be cleaned up for a given policy on it's first-time running. To prevent thousands of tags from being queued by one project, we will add throttling to the queue or service.
Permissions and Security
- There are no permissions changes required for this change.
Documentation
- Add a section to the Container Registry Admin docs that jobs will be throttled and how to troubleshoot when something goes wrong.
Availability & Testing
What does success look like, and how can we measure that?
We are able to enable container expiration and retention policies for all projects on GitLab.com and self-managed instances regardless of which container registry is being used.
Other concerns
- Before turning on the expiration policy feature for all projects, please open a scalability issue: https://gitlab.com/gitlab-com/gl-infra/scalability/-/issues/new?issuable_template=Review%20Request