Missing pagination headers for projects with over 10,000 jobs in self-managed instances

Problem Statement

It was discovered (with the help of GitLab's great support team!) that pagination headers are no longer being produced for projects with over 10,000 jobs in any GitLab instance, not just GitLab.com (documentation updates were missed on this, and there was no mention of this in release notes; I'm concerned about how changes like this get into the GitLab baseline without any heads up to users).

Why is this an issue?

We have been using our own internally-coded Python utilities (making use of requests) to manage some aspects of our GitLab instance, and some rely on the jobs API. We ran this script for several months without issue, and it was always able to iterate over every job for a project in our instance. We can't customize the amount of jobs per page beyond 100, so our only method of obtaining the number of pages was through the header X-Total-Pages. This disappeared after we upgraded to version 14.9.3 and is still missing in 14.10.0. We would love the ability to get this header back in self-managed instances, knowing that there could be a performance impact (although we've never seen one in projects with over 115,000 jobs).

This bug would ideally address two needs:

1. the ability to get all pagination headers back for any size project in self-managed instances
2. the ability to set the pagination to be over 100 jobs per page

Without the X-Total-Pages header, all we can do in our code is "guess" how many pages exist, and break out of a loop once we hit an empty page; this is far from ideal, especially as our projects may continue to grow past any arbitrary limit we think of.

Associated Information

Thanks to the support team for helping track this down!

• Cleanup api_kaminari_count_with_limit (!81084 - merged) - this MR removed a related feature flag
• Default enabled api_kaminari_count_with_limit (!82003 - merged) - this MRchanged the default status of the related feature flag
• Add pagination headers limit info (!86493 - merged) - this MR is to cover the documentation gap that occurred

Proposal

Replace the constant MAX_COUNT_LIMIT with something that is configurable for self-managed instances.

Option 1: Set in gitlab.yml

1. Replace MAX_COUNT_LIMIT with a gitlab.yml setting with default value 10_000, and add a setting to re-enable pagination for the jobs API.
2. Make it possible to change this value in Omnibus and the Helm chart.

Option 2: Configure in admin panel

1. Replace MAX_COUNT_LIMIT with an application setting with default value 10_000, and add a setting to re-enable pagination for the jobs API.
2. Expose the setting in the GraphQL, REST API, and UI

Discussion

Option 1 is more performant but less dynamic. It is probably flexible enough as this is not a limit that users will need to change very frequently. It would also make it technically possible to have a different limit on different nodes, which means that users could spin up internal-only nodes for specific scripts, while keeping the a lower limit set their "public" nodes.

Also note that offset pagination is deprecated for the jobs API and out of scope.