provide a bypass for the 10,000 API kaminari limit - self managed customers
About
In %11.8 a limit of 10_000
was implemented in the Kaminari module.
The jobs API doesn't [return pagination headers on purpose [..]
The x-total-pages
header is now omitted. Both we and our customers consider this to be a breaking change.
From [API] Omit `X-Total` and `X-Total-Pages` header... (gitlab-foss!23931 - merged):
Note: This is a breaking API change in the sense that API clients cannot expect the
X-Total
andX-Total-Pages
headers to always be present anymore.
This arose from a change we made, and so I don't believe it's accurate to call it a bug. Also, bug issues auto close - which will not be satisfactory for the affected customers. So, raising this as typefeature.
The nature of this request is:
Although, we could make this endpoint take in an additional parameter only for self-managed to return the count at the risk of slowing down their system?
Until %14.10 SM customers were able to use a feature flag to by-pass this change, but in 14.10 and later the FF was no longer present, so this is when it became a breaking change.
Some discussion about this change can be found in #361362 (closed).
Brett Walker
We have an issue, Discussion: Querying large sets of data in para... (#362957), to discuss possible solutions to this problem.
Proposal
- Add a new boolean application setting to control whether an instance omits
X-Total
andX-Total-Pages
headers. When enabled,#exceeds_count?
would always returnfalse
and the instance would never omit the headers. - The setting must be
false
by default. - The setting must be enabled (be
true
) on GitLab.com so it continues to omit headers when the limit is reached. If it can be enabled just forGitLab.com?
through a data migration that would be simple. If not, we may need to add the setting first without it being functional, make a change request to have someone enable it for production and staging, and then merge a change to have the setting apply once it has been enabled.