[API] Omit `X-Total` and `X-Total-Pages` headers when items count is more than 10,000
What does this MR do?
This is inspired by what we do in
- We issue a
COUNTlimited to 10,001
- If the total is more than 10,000, we then call
without_counton the paginated data (thus the
- This feature is behind the
api_kaminari_count_with_limitfeature flag to ensure that if there is a regression, or if this is breaking users' workflow in a significant way, we can rollback easily.
- The plan is to remove the flag after at least 1 release of it being enabled on GitLab.com.
Note: This is a breaking API change in the sense that API clients cannot expect the
X-Total-Pages headers to always be present anymore.
10,000 seems like a good number, it takes 2 seconds to count jobs for
gitlab-ce on GitLab.com with a 10,000 limit, and timeouts with a 100,000 limit:
SELECT COUNT(count_column) FROM (SELECT 1 AS count_column FROM “ci_builds” WHERE “ci_builds”.“type” IN (‘Ci::Build’) AND “ci_builds”.“project_id” = 13083 LIMIT 10000) subquery_for_count Aggregate (cost=12258.53..12258.54 rows=1 width=8) (actual time=2070.293..2070.293 rows=1 loops=1) Buffers: shared hit=6079 read=3942 I/O Timings: read=2002.545 -> Limit (cost=0.57..12133.53 rows=10000 width=4) (actual time=0.083..2067.605 rows=10000 loops=1) Buffers: shared hit=6079 read=3942 I/O Timings: read=2002.545 -> Index Scan using index_ci_builds_on_project_id_and_id on ci_builds (cost=0.57..9389104.31 rows=7738513 width=4) (actual time=0.082..2065.784 rows=10000 loops=1) Index Cond: (project_id = 13083) Filter: ((type)::text = 'Ci::Build'::text) Buffers: shared hit=6079 read=3942 I/O Timings: read=2002.545 Planning time: 2.760 ms Execution time: 2070.346 ms
What are the relevant issue numbers?
Does this MR meet the acceptance criteria?
- Changelog entry added, if necessary
- Documentation created/updated
- Tests added for this feature/bug
- [-] Tested in all supported browsers
- Conforms to the code review guidelines
- Conforms to the merge request performance guidelines
- Conforms to the style guides
- Conforms to the database guides
- [-] Link to e2e tests MR added if this MR has Requires e2e tests label. See the Test Planning Process.
- Security reports checked/validated by reviewer