Handle ghost container repositories on the rails side

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Work on this issue
• Close this issue

🔥 Problem

To work properly, container repositories should exist in the rails and the container registry side.

While investigating for #395749 (closed), we noticed that we can have a situation where the container repository doesn't exist on the container registry side anymore. We are not sure what could be the root cause here but one possible lead would be how container repositories are created on the rails side: they are created when a login is done to push a repository. The problem there is that clients could ask for such login and never push anything to the container registry = a 👻 repository on the rails side.

Here is a Kibana search showing the problem at hand.

🚒 Solution

Two aspects:

1. Identify those 👻 repositories
2. Remove them

For (1.), we can't possible do it in a giant loop. There are way too many objects. What would be smarter to do is: whenever we interact with the container registry and a 404 is returned, mark that container repository as suspicious.

The above will create a backlog of suspicious 👻 repositories. For (2.), apply the usual stuff: have a limited capacity worker that will loop on each element of that backlog. For each container repository, contact the container registry to confirm that the container repository doesn't exist (https://gitlab.com/gitlab-org/container-registry/-/blob/master/docs-gitlab/api.md#get-repository-details is your friend here). If confirmed, destroy the container repository.

🔮 Other aspects

Filling this under bugperformance because the container repository is very large. Cleaning it up (like this issue goal) improves the overall performance.