Use a fixed page size when getting tags of a container repository

🗄 Context

There is quite a scope to unfold here. I'll try to be as short as possible but still give the complete context here.

The Container Registry hosts container repositories and tags for users. Now, the container repositories exists in both sides: the rails backend and the Category:Container Registry. Tags, however, only exist on the Container Registry side.

This means that to get tags of a single container repository, rails has to query the Container Registry API to get the list of tags. This API supports pagination parameters.

Aside from that, there is an ongoing data migration on the container repository. In short, some metadata around container repositories and their tags will now exist in a database. The existing repositories (and their tags) on gitlab.com are being migrated to this database. This database unlocks several future features but that's another story. To keep things simply, we can have migrated and non migrated container repositories.

Now, back to the tags list API. We observed the following:

1. For non migrated repositories, all tags are returned, no matter which pagination parameters are sent (including not sending any).
   • This is a known 🐛 that exists for quite some time now.
2. For migrated repositories, things are different: the pagination parameters are properly taken into account.

Given (1.), the rails has been implemented to not send any pagination parameters. It was not necessary because all tags were returned all the times. Now that we have migrated repositories, rails is still not sending any pagination parameters. As such, the default ones are applied (first page, page size of 100) and this leads to a 💥 where only the first 100 tags are returned.

This problem is surfaced on the UI where only 100 tags are displayed and leads to a confusing UX. That is issue #359277 (closed).

It also impacts cleanup policies, where the policy will only consider the 100 first tags = the policy is way less efficient.

The impact is only for gitlab.com as the container registry migration for self-managed has not started (yet).

⚡ Quick solution

Two facts to take into account here:

• This is a highly confusing situation.
• The majority of the container repositories on gitlab.com have a quite low amount of tags.

So, the idea is that the function that gets those tags should behave as previously: return all the tags. For migrated container repositories, it should walk all the available pages. That could work but this also means more network requests for the container registry. While we think about this solution, quick and boring one can be used:

• Send 10 000 as the wanted page size to the container registry.
  • As seen here, this covers 99.99+ % of the container repositories on gitlab.com.
  • We don't get the next page, never.

Let's see what happens when we send a fixed page size:

• For non migrated container repositories, there is no impact. Pagination parameters being ignored, we can send whatever we want to the container registry.
• For migrated container repositories, the first 10 000 tags will be returned in a single request.

After discussing the above, this quick solution is good enough to already fix the problem for 99.99+ % of the cases.

🔍 What does this MR do and why?

• Update the container registry client so that /tags/list is queried with a fixed page size: 10 000.
• Update the related spec.

Given that migrated container repositories will trigger a database access, going from 100 to 10 000 might pose a risk. To increase our confidence here, we decided to use a feature flag. The rollout issue is [Feature flag] Rollout of `container_registry_t... (#374073 - closed). The feature flag will not really be used for an incremental rollout but more for a safety switch. If things go wrong on the Container Registry side, we can fall back to the current behavior.

📺 Screenshots or screen recordings

See next section

⚙ How to set up and validate locally

The main issue to test this MR is that the container registry with the metadata database is not part of GDK (for now). So we will need to pull the project and build it from scratch.

1. Have a GDK ready with the Container Registry support.
2. Follow these instructions to setup a Container Registry with the new API support.
3. Create a new project.
4. To push repositories with 100+ tags, you can use https://gitlab.com/nmezzopera/container-factory or you can manually push several tags to a given repository or use this bash script.

I pushed 105 tags to a given image.

Let's check the UI (<project url>/container_repository):

😿

That's because no pagination parameters are sent the default value (100) is used.

Let's check the rails console:

ContainerRepository.last.tags.size # => 100

Same situation. 😿

This MR to the rescue! 🚒 Let's enable the feature flag :

Feature.enable(:container_registry_tags_list_default_page_size)

Check the UI:

Check the rails console:

ContainerRepository.last.tags.size # => 105

typebug fixed! 🎉

🚦 MR acceptance checklist

This checklist encourages us to confirm any changes have been analyzed to reduce risks in quality, performance, reliability, security, and maintainability.

• I have evaluated the MR acceptance checklist for this MR.