Rate limit REST runner job lists filtered by completed status

Leaminn Marequested to merge
ci-rest-runner-jobs-status-limit into master

What does this MR do and why?

Context

In !268206 (merged), we rate limited CiRunner.jobs when it is filtered by a completed status, because the filter makes the database read the runner's jobs until it fills a page. GET /runners/:id/jobs filters the same way, so in this step we apply the same bound to the REST endpoint. The figures behind it are in https://gitlab.com/gitlab-org/gitlab/-/work_items/638239#note_4175478465.

This MR

  • Rate limits calls filtered by success, failed, or canceled, per user, on the same limit and application setting as CiRunner.jobs, so a user has one budget across both APIs. Over the limit, the endpoint returns 429 Too Many Requests.
  • Logs calls over that limit instead of blocking them, in both APIs, until a second flag enforces it. New rate limits roll out in log mode first, so the logs show who a limit would block before it blocks anyone. The limit key now uses staged_rule. No instance enforces it yet, because CiRunner.jobs's flag is still off everywhere.
  • Returns no jobs without querying when the status is one a runner's jobs cannot have, or when the user has no authorized project. Both requests matched nothing but still read every job of the runner.

This caps how often one user can make GitLab read through a runner's job history over REST.

Feature Flags:

  • ci_rate_limit_rest_runner_job_list_completed_statuses: the REST changes above, with the limit in log mode
  • ci_enforce_runner_job_list_completed_statuses_limit: blocks calls over the limit, in GraphQL and REST

No changelog entry while the limit is behind the disabled flag. The entry comes with the flag-removal MR.

References

How to set up and validate locally

  1. In a Rails console: 
    Feature.enable(:ci_rate_limit_rest_runner_job_list_completed_statuses)
ApplicationSetting.current.update!(runner_job_list_completed_statuses_limit_per_user: 1)
  2. Wait a minute, so the web server picks up the new flag and setting.
  3. Run this twice, with an administrator's personal access token and the ID of any runner that has run jobs: 
    curl --header "PRIVATE-TOKEN: <your_access_token>" "http://gdk.test:3000/api/v4/runners/<id>/jobs?status=failed"
    Both return 200. In log/api_json.log, the second request's rate_limit_state includes applimiter_runner_job_list_completed_statuses:limit_runner_job_list_completed_status_filters_by_user_log:log.
  4. Run Feature.enable(:ci_enforce_runner_job_list_completed_statuses_limit) and wait a minute.
  5. Run the curl twice more. The second returns 429 with {"message":{"error":"This endpoint has been requested too many times. Try again later."}}. The enforced limit counts from zero, so the first still returns 200.
  6. Run it with status=running. It returns 200, because the limit applies only to success, failed, and canceled.
  7. Run it with status=pending. It returns []. In log/api_json.log, its db_ci_count is 2 lower than the status=running request, because it skips the count and the page queries.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Leaminn Ma

Merge request reports