Add keyset pagination to list tags API
What does this MR do and why?
In this MR, we add keyset pagination, in addition to the existing offset-based pagination. This newly introduced pagination uses the container registry API is getting a list of tags for the repository. The container registry API is available only for migrated repositories (i.e. Gitlab.com for now) thus this new pagination method will also only be available for migrated repositories.
We put the changes behind the feature flag: use_registry_api_to_list_tags
with rollout issue gitlab-com/gl-infra/production#17527 (closed).
Documentation is also updated in this MR.
How to set up and validate locally
Prerequisites:
- In Rails console, enable the feature:
Feature.enable(:use_registry_api_to_list_tags)
- Set local setup as
migrated?
. For now, only Gitlab.com is considered migrated, we can override that by changing the following line in#migrated?
inapp/models/container_repository.rb
to:
def migrated?
true || Gitlab.com_except_jh?
end
-
Have a metadata database setup for the container registry (i.e. run it outside the gdk). You can find the setup guide here.
-
A project with multiple repository tags.
-
Private token that has permissions to your project and the registry.
Steps:
- Try to
curl
the endpoint before and after turning on the feature flag (i.e. when the flag is disabled). By default, it is the offset-based pagination that is used if thepagination
parameter is not set.
curl --header "PRIVATE-TOKEN: glpat-xxxx_xxxxxxx" \
"http://gdk.test:3000/api/v4/projects/<project-id>/registry/repositories/<repository-id>/tags"
- Now, we will try to access the keyset pagination, by providing the parameter:
pagination=keyset
.
curl --header "PRIVATE-TOKEN: glpat-xxxx_xxxxxxx" \
"http://gdk.test:3000/api/v4/projects/28/registry/repositories/13/tags?pagination=keyset"
- Now we will try the
sort
parameter, we should see the results sorted in descending order based onname
with the curl command below. When thesort
parameter is not provided, it defaults to ascending order byname
.
curl --header "PRIVATE-TOKEN: glpat-xxxx_xxxxxxx" \
"http://gdk.test:3000/api/v4/projects/28/registry/repositories/13/tags?pagination=keyset&sort=desc"
- Now we try to test the
per_page
parameter and test out the pagination. When not provided,per_page
defaults to 20 items per page.
curl -v --header "PRIVATE-TOKEN: glpat-xxxx_xxxxxxx" \
"http://gdk.test:3000/api/v4/projects/28/registry/repositories/13/tags?pagination=keyset&per_page=2"
Notice the -v
flag so we can see the response headers. If there is a next page to the current page, we will see a Link
header in the response. Something like the following:
Link: <http://gdk.test:3000/api/v4/projects/28/registry/repositories/13/tags?id=28&last=tag10&page=1&pagination=keyset&per_page=2&repository_id=13&sort=asc>; rel="next"
To test out this link header that we got, we try to curl the given URL and we should get the next page.
curl -v --header "PRIVATE-TOKEN: glpat-xxxx_xxxxxxx" "http://gdk.test:3000/api/v4/projects/28/registry/repositories/13/tags?id=28&last=tag10&page=1&pagination=keyset&per_page=2&repository_id=13&sort=asc"
And we get the next page.
- Feel free to try the different combinations of
per_page
andsort
and try to follow theLink
header up to the last page.
Errors:
We return a 400 (:bad_request
) when the keyset-based pagination is tried to be accessed but the repository is not migrated and thus it does not support keyset-based pagination.
MR acceptance checklist
Related to #432470 (closed)