Optimize and reenable pageInfo.hasNextPage for Vulnerabilities
Summary
In !39261 (merged) we disabled loading hasNextPage as this it large performance issues especially for large group with hundreds projects without vulnerabilities. To quickly solve that issue (~P1/~S1) we changed the query to not load hasNextPage and we were relying on endCursor
value. Now we want to reenable loading hasNextPage from GraphQL API, but to do so we need to first find why calculating value of hasNextValue
is timing out on database and then after extensive tests we can reenable.
Steps to reproduce
- Great a group with many (> 100) subgroups and many (> 100) projects in it. In one project run pipeline security jobs to get Vulnerabilities.
- In GraphQL explorer send request:
query {
group(fullPath: "largegroup") {
vulnerabilities(first: 20) {
pageInfo {
...PageInfo
__typename
}
__typename
}
__typename
}
}
fragment PageInfo on PageInfo {
startCursor
endCursor
hasNextPage
__typename
}
- You see
Internal Server Error
in the response.
Example Group
https://gitlab.com/groups/parentgroup2
What is the current bug behavior?
Calculating an returning hasNextPage causes timeouts on database.
What is the expected correct behavior?
Calculating an returning hasNextPage does not cause timeouts on database.
Implementation plan
While calculating the hasPreviousPage
and hasNextPage
attribute of the pageInfo
we are currently firing a count query to check if there are more records. Instead of firing that count query what we can do is basically fetching one more record from database and then check if we had the extra record in the resultset to determine hasNextPage
& hasPreviousPage
.
See !33632 (closed)