Count REST runner job lists newest partition first

What does this MR do and why?

Context

GET /runners/:id/jobs counts the runner's jobs, up to 10,001, to fill the X-Total headers. The count reads p_ci_builds partitions in the order Postgres keeps them, oldest first, so for a busy runner it fills the cap from its oldest jobs, which are slow, uncached reads. In this step, we count from the newest partition instead. The figures behind it are in https://gitlab.com/gitlab-org/gitlab/-/work_items/638239#note_4175478465.

This MR

  • Counts the jobs one p_ci_builds partition at a time, newest first, under the same 10,001 cap, so the X-Total headers don't change. Until ci_partitions has rows, as on a new instance before Ci::PartitioningWorker first runs, it uses the current count.
  • Adds a counter: option to offset pagination, so an endpoint can supply its own count without changing how it paginates.

This keeps the count on a busy runner's recent jobs instead of its oldest ones.

Feature Flag: ci_count_rest_runner_job_list_newest_partitions_first

References

Database queries

See MR comments.

How to set up and validate locally

  1. Run this with an administrator's personal access token (with the admin_mode scope if Admin Mode is enabled) and the ID of a runner that has run jobs:
    curl --head --header "PRIVATE-TOKEN: <your_access_token>" "http://gdk.test:3000/api/v4/runners/<id>/jobs"
    Note the X-Total header.
  2. In a Rails console, run Feature.enable(:ci_count_rest_runner_job_list_newest_partitions_first) and wait a minute.
  3. Run the curl again. X-Total is the same, and log/development.log shows the count as SELECT COUNT(*) FROM (SELECT 1 AS one FROM "ci_partitions" CROSS JOIN LATERAL (....

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Leaminn Ma

Merge request reports

Loading
Loading