graphQL pagination broken on package -> pipelines
🔥 Bug
Packages can be linked to pipelines through a build info object:
flowchart LR
Packages::Package-- 1:n ---BuildInfos-- 1:1 ---CI::Pipeline
We have a has_many through:
statement in place to provide a "shortcut" on package objects.
🖥 GraphQL query
In graphQL, we can query for the details of a package including the pipelines. This pipelines
field is automatically paginated.
It seems that we have an issue with the paginator on this query.
On staging, we have a package linked to 5 pipelines.
Query 1 (no pagination parameters)
query {
package(id: "gid://gitlab/Packages::Package/49726") {
id
pipelines {
nodes {
id
}
pageInfo {
endCursor
hasNextPage
hasPreviousPage
startCursor
}
}
}
}
All pipelines are returned
Query 2 (first: 2)
query {
package(id: "gid://gitlab/Packages::Package/49726") {
id
pipelines(first: 2) {
nodes {
id
}
pageInfo {
endCursor
hasNextPage
hasPreviousPage
startCursor
}
}
}
}
Two first pipelines returned
Query 3 (first: 2, after: X)
query {
package(id: "gid://gitlab/Packages::Package/49726") {
id
pipelines(first: 2, after: "eyJpZCI6IjEzMzE4MTk4In0") {
nodes {
id
}
pageInfo {
endCursor
hasNextPage
hasPreviousPage
startCursor
}
}
}
}
The last 2 pipelines are returned
🔍 Investigation
Let's create locally a package linked to 5 pipelines: (in a rails console)
# creates a package
def fixture_file_upload(*args, **kwargs)
Rack::Test::UploadedFile.new(*args, **kwargs)
end
# note the package id here
pkg = FactoryBot.create(:npm_package, project: Project.first)
# creates 5 pipelines
Packages::BuildInfo.create!(package_id: pkg.id, pipeline_id: FactoryBot.create(:ci_pipeline).id)
Packages::BuildInfo.create!(package_id: pkg.id, pipeline_id: FactoryBot.create(:ci_pipeline).id)
Packages::BuildInfo.create!(package_id: pkg.id, pipeline_id: FactoryBot.create(:ci_pipeline).id)
Packages::BuildInfo.create!(package_id: pkg.id, pipeline_id: FactoryBot.create(:ci_pipeline).id)
Packages::BuildInfo.create!(package_id: pkg.id, pipeline_id: FactoryBot.create(:ci_pipeline).id)
Query 1 (no pagination params)
query {
package(id: "gid://gitlab/Packages::Package/745") {
id
pipelines {
nodes {
id
}
pageInfo {
endCursor
hasNextPage
hasPreviousPage
startCursor
}
}
}
}
We can see this SQL query:
SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" IN (346, 347, 348, 349, 350) ORDER BY "ci_pipelines"."id" DESC LIMIT 100
Note: the IN condition comes from the fact that we’re getting the pipelines using a has_many :pipelines, through: x, disable_joins: true . So, we will always have the 5 pipeline ids in that IN condition, no matter what the pagination params are.
Notice the order used: id DESC
We get a list of 5 ids
Query 2 (first: 2)
query {
package(id: "gid://gitlab/Packages::Package/745") {
id
pipelines(first: 2) {
nodes {
id
}
pageInfo {
endCursor
hasNextPage
hasPreviousPage
startCursor
}
}
}
}
SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" IN (346, 347, 348, 349, 350) ORDER BY "ci_pipelines"."id" DESC LIMIT 2
Same order and the 2 first pipelines are returned
Note the end cursor for the next graphql query.
Query 3 (first: 2, after: X)
query {
package(id: "gid://gitlab/Packages::Package/745") {
id
pipelines(first: 2, after: "eyJpZCI6IjM0OSJ9") {
nodes {
id
}
pageInfo {
endCursor
hasNextPage
hasPreviousPage
startCursor
}
}
}
}
triggers this SQL query:
SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" IN (346, 347, 348, 349, 350) AND ("ci_pipelines"."id" < 349) LIMIT 2
Wrong ids returned (it's the two last ones from Query 1).
No ORDER BY
clause id ASC
= pagination broken
🔮 Possible workaround
Disabling new_graphql_keyset_pagination
fixes the issue.
Here is the SQL from Query 3 with that feature flag disabled:
SELECT "ci_pipelines".* FROM "ci_pipelines" WHERE "ci_pipelines"."id" IN (346, 347, 348, 349, 350) AND (("ci_pipelines"."id" < 349)
) AND "ci_pipelines"."id" != 349 ORDER BY "ci_pipelines"."id" DESC LIMIT 2
the ORDER BY
clause is there = the correct pipelines are returned