provide a bypass for the 10,000 API kaminari limit - self managed customers

Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.

• Work on this issue
• Close this issue

About

In %11.8 a limit of 10_000 was implemented in the Kaminari module.

source

The jobs API doesn't [return pagination headers on purpose [..]

The x-total-pages header is now omitted. Both we and our customers consider this to be a breaking change.

From [API] Omit `X-Total` and `X-Total-Pages` header... (gitlab-foss!23931 - merged):

Note: This is a breaking API change in the sense that API clients cannot expect the X-Total and X-Total-Pages headers to always be present anymore.

This arose from a change we made, and so I don't believe it's accurate to call it a bug. Also, bug issues auto close - which will not be satisfactory for the affected customers. So, raising this as typefeature.

The nature of this request is:

Although, we could make this endpoint take in an additional parameter only for self-managed to return the count at the risk of slowing down their system?

Until %14.10 SM customers were able to use a feature flag to by-pass this change, but in 14.10 and later the FF was no longer present, so this is when it became a breaking change.

Some discussion about this change can be found in #361362 (closed).

See also:

Brett Walker

We have an issue, Discussion: Querying large sets of data in para... (#362957), to discuss possible solutions to this problem.

Proposal

• Add a new boolean application setting to control whether an instance omits X-Total and X-Total-Pages headers. When enabled, #exceeds_count? would always return false and the instance would never omit the headers.
• The setting must be false by default.
• The setting must be enabled (be true) on GitLab.com so it continues to omit headers when the limit is reached. If it can be enabled just for GitLab.com? through a data migration that would be simple. If not, we may need to add the setting first without it being functional, make a change request to have someone enable it for production and staging, and then merge a change to have the setting apply once it has been enabled.