Skip to content

[API] Omit `X-Total` and `X-Total-Pages` headers when items count is more than 10,000

What does this MR do?

This is inspired by what we do in NumbersHelper and Projects::JobsController#index:

  1. We issue a COUNT limited to 10,001
  2. If the total is more than 10,000, we then call without_count on the paginated data (thus the X-Total, X-Total-Pages, and rel="last" Link are omitted).
  3. This feature is behind the api_kaminari_count_with_limit feature flag to ensure that if there is a regression, or if this is breaking users' workflow in a significant way, we can rollback easily.
  4. The plan is to remove the flag after at least 1 release of it being enabled on GitLab.com.

Note: This is a breaking API change in the sense that API clients cannot expect the X-Total and X-Total-Pages headers to always be present anymore.

10,000 seems like a good number, it takes 2 seconds to count jobs for gitlab-ce on GitLab.com with a 10,000 limit, and timeouts with a 100,000 limit:

SELECT COUNT(count_column) FROM (SELECT  1 AS count_column FROM “ci_builds” WHERE “ci_builds”.“type” IN (‘Ci::Build’) AND “ci_builds”.“project_id” = 13083 LIMIT 10000) subquery_for_count

Aggregate (cost=12258.53..12258.54 rows=1 width=8) (actual time=2070.293..2070.293 rows=1 loops=1)
 Buffers: shared hit=6079 read=3942
 I/O Timings: read=2002.545
 -> Limit (cost=0.57..12133.53 rows=10000 width=4) (actual time=0.083..2067.605 rows=10000 loops=1)
    Buffers: shared hit=6079 read=3942
    I/O Timings: read=2002.545
    -> Index Scan using index_ci_builds_on_project_id_and_id on ci_builds (cost=0.57..9389104.31 rows=7738513 width=4) (actual time=0.082..2065.784 rows=10000 loops=1)
       Index Cond: (project_id = 13083)
       Filter: ((type)::text = 'Ci::Build'::text)
       Buffers: shared hit=6079 read=3942
       I/O Timings: read=2002.545
Planning time: 2.760 ms
Execution time: 2070.346 ms

What are the relevant issue numbers?

Closes #52674 (closed), https://gitlab.com/gitlab-org/gitlab-ce/issues/42194.

Does this MR meet the acceptance criteria?

Edited by Rémy Coutable

Merge request reports