Rate limit runner job lists filtered by completed status

What does this MR do and why?

Context

CiRunner.jobs with a statuses filter reads the runner's jobs in ID order until it fills a page, so the cost of a call grows with the number of jobs the runner has run. The figures behind the statuses we bound are in https://gitlab.com/gitlab-org/gitlab/-/work_items/638237#note_4165629642.

This MR

  • CiRunner.jobs drops statuses a runner's jobs never hold from its filter. A filter left with no statuses returns no jobs without querying.
  • Rate limits CiRunner.jobs calls whose statuses include SUCCESS, FAILED or CANCELED, per user, including queries sent over ActionCable. The limit is a new admin CI/CD setting (default 330 a minute, 0 disables it). Over the limit, the field returns a GraphQL error and the rest of the request still resolves.
  • Keeps the set of statuses a runner's jobs hold in Ci::Build::RUNNER_JOB_LIST_STATUSES, so the REST runner jobs endpoint can drop the same set.

This caps how often one user can make GitLab read through a runner's job history.

Feature Flag: ci_rate_limit_runner_job_list_completed_statuses

No changelog entry while the limit is behind the disabled flag. The entry comes with the flag-removal MR.

References

Screenshots or screen recordings

Before After
before_master_admin_ci_cd after_branch_admin_ci_cd

How to set up and validate locally

  1. In a Rails console:
    Feature.enable(:ci_rate_limit_runner_job_list_completed_statuses)
    ApplicationSetting.current.update!(runner_job_list_completed_statuses_limit_per_user: 1)
  2. As an administrator, in /-/graphql-explorer, run this twice, with the ID of any runner that has run jobs:
    query { runner(id: "gid://gitlab/Ci::Runner/<id>") { jobs(statuses: [FAILED]) { nodes { id } } } }
    The second run returns This field has been requested with the SUCCESS, FAILED, or CANCELED statuses too many times. Try again later.
  3. Run it again with statuses: [RUNNING]. It returns jobs and no error.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Leaminn Ma

Merge request reports

Loading
Loading