Skip to content

Keyset pagination for `Project` based on `id` column

Andreas Brandl requested to merge ab/keyset-pagination into master

What does this MR do?

This MR prepares support for keyset pagination for the Projects API endpoint based on the id column.

The idea is to:

  • Add filters id_before, id_after as a regular filter to the endpoint -> extracted into !19949 (merged)
  • We already support order_by=id and sort=asc|desc
  • That's all we need for keyset pagination!

The code for the pagination now only needs to do this:

  • Based on the order/sort information, figure out how to describe the next page
  • Add a link to the next page to the response header

I'm separating two concerns:

  1. Translate the HTTP request information and handling the response (e.g. setting headers)
  2. Working out the pagination information for the next page

Issue: #32186 (closed)

Database queries

This is a database query example for this request:

GET /api/v4/projects?pagination=keyset&id_after=22345098&order_by=id&per_page=20
SELECT
    "projects".*
FROM
    "projects"
WHERE
    "projects"."visibility_level" = 10
    AND (id > 22345098)
    AND "projects"."pending_delete" = FALSE
ORDER BY
    "projects"."id" DESC
LIMIT 20

Plan: https://explain.depesz.com/s/n7A

Outlook

Going forward, the idea is to add similar filtering support for other columns. We will basically have to match the order_by columns and allow filtering with respective %_after, %_before filters in order to support keyset pagination on those columns. If we wanted to keep supporting all order directions, we will have to add filters and corresponding indexes for all order-by directions available (#36157 (closed)).

Once this is proven to work and an accepted direction we want to go at, we'll add this to all available endpoints.

Does this MR meet the acceptance criteria?

Conformity

Availability and Testing

Security

If this MR contains changes to processing or storing of credentials or tokens, authorization and authentication methods and other items described in the security review guidelines:

  • Label as security and @ mention @gitlab-com/gl-security/appsec
  • The MR includes necessary changes to maintain consistency between UI, API, email, or other methods
  • Security reports checked/validated by a reviewer from the AppSec team
Edited by 🤖 GitLab Bot 🤖

Merge request reports