Count REST runner job lists newest partition first
What does this MR do and why?
Context
GET /runners/:id/jobs counts the runner's jobs, up to 10,001, to fill the X-Total headers. The count reads p_ci_builds partitions in the order Postgres keeps them, oldest first, so for a busy runner it fills the cap from its oldest jobs, which are slow, uncached reads. In this step, we count from the newest partition instead. The figures behind it are in https://gitlab.com/gitlab-org/gitlab/-/work_items/638239#note_4175478465.
This MR
- Counts the jobs one
p_ci_buildspartition at a time, newest first, under the same 10,001 cap, so theX-Totalheaders don't change. Untilci_partitionshas rows, as on a new instance beforeCi::PartitioningWorkerfirst runs, it uses the current count. - Adds a
counter:option to offset pagination, so an endpoint can supply its own count without changing how it paginates.
This keeps the count on a busy runner's recent jobs instead of its oldest ones.
Feature Flag: ci_count_rest_runner_job_list_newest_partitions_first
References
- https://gitlab.com/gitlab-org/gitlab/-/work_items/638239+
- https://gitlab.com/gitlab-org/gitlab/-/work_items/638981+
Database queries
See MR comments.
How to set up and validate locally
- Run this with an administrator's personal access token (with the
admin_modescope if Admin Mode is enabled) and the ID of a runner that has run jobs:Note thecurl --head --header "PRIVATE-TOKEN: <your_access_token>" "http://gdk.test:3000/api/v4/runners/<id>/jobs"X-Totalheader. - In a Rails console, run
Feature.enable(:ci_count_rest_runner_job_list_newest_partitions_first)and wait a minute. - Run the
curlagain.X-Totalis the same, andlog/development.logshows the count asSELECT COUNT(*) FROM (SELECT 1 AS one FROM "ci_partitions" CROSS JOIN LATERAL (....
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.