Add keyset pagination to list tags API

What does this MR do and why?

In this MR, we add keyset pagination, in addition to the existing offset-based pagination. This newly introduced pagination uses the container registry API is getting a list of tags for the repository. The container registry API is available only for migrated repositories (i.e. Gitlab.com for now) thus this new pagination method will also only be available for migrated repositories.

We put the changes behind the feature flag: use_registry_api_to_list_tags with rollout issue gitlab-com/gl-infra/production#17527 (closed).

Documentation is also updated in this MR.

How to set up and validate locally

Prerequisites:

1. In Rails console, enable the feature:

Feature.enable(:use_registry_api_to_list_tags)

2. Set local setup as migrated?. For now, only Gitlab.com is considered migrated, we can override that by changing the following line in #migrated? in app/models/container_repository.rb to:

  def migrated?
    true || Gitlab.com_except_jh?
  end

3. Have a metadata database setup for the container registry (i.e. run it outside the gdk). You can find the setup guide here.

4. A project with multiple repository tags.

5. Private token that has permissions to your project and the registry.

Steps:

1. Try to curl the endpoint before and after turning on the feature flag (i.e. when the flag is disabled). By default, it is the offset-based pagination that is used if the pagination parameter is not set.

curl --header "PRIVATE-TOKEN: glpat-xxxx_xxxxxxx" \
     "http://gdk.test:3000/api/v4/projects/<project-id>/registry/repositories/<repository-id>/tags"

2. Now, we will try to access the keyset pagination, by providing the parameter: pagination=keyset.

curl --header "PRIVATE-TOKEN: glpat-xxxx_xxxxxxx" \
     "http://gdk.test:3000/api/v4/projects/28/registry/repositories/13/tags?pagination=keyset"

3. Now we will try the sort parameter, we should see the results sorted in descending order based on name with the curl command below. When the sort parameter is not provided, it defaults to ascending order by name.

curl --header "PRIVATE-TOKEN: glpat-xxxx_xxxxxxx" \
     "http://gdk.test:3000/api/v4/projects/28/registry/repositories/13/tags?pagination=keyset&sort=desc"

4. Now we try to test the per_page parameter and test out the pagination. When not provided, per_page defaults to 20 items per page.

curl -v --header "PRIVATE-TOKEN: glpat-xxxx_xxxxxxx" \
     "http://gdk.test:3000/api/v4/projects/28/registry/repositories/13/tags?pagination=keyset&per_page=2"

Notice the -v flag so we can see the response headers. If there is a next page to the current page, we will see a Link header in the response. Something like the following:

Link: <http://gdk.test:3000/api/v4/projects/28/registry/repositories/13/tags?id=28&last=tag10&page=1&pagination=keyset&per_page=2&repository_id=13&sort=asc>; rel="next"

To test out this link header that we got, we try to curl the given URL and we should get the next page.

curl -v --header "PRIVATE-TOKEN: glpat-xxxx_xxxxxxx" "http://gdk.test:3000/api/v4/projects/28/registry/repositories/13/tags?id=28&last=tag10&page=1&pagination=keyset&per_page=2&repository_id=13&sort=asc"

And we get the next page.

5. Feel free to try the different combinations of per_page and sort and try to follow the Link header up to the last page.

Errors:

We return a 400 (:bad_request) when the keyset-based pagination is tried to be accessed but the repository is not migrated and thus it does not support keyset-based pagination.

MR acceptance checklist

✅ Please evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Related to #432470 (closed)