Limit runner managers listed per user in the REST API

What does this MR do and why?

Context

REST GET /runners/:id/managers reads, renders and returns every manager of a runner in one response, so the work behind each call grows with the number of managers, and nothing bounds it per user. The figures and the agreed approach are in https://gitlab.com/gitlab-org/gitlab/-/work_items/638238#note_4185424743 (part 3 of 3).

This MR

  • The endpoint counts the runner's managers toward a new per-user rate limit, runner_manager_list, before it reads the list. The count stops one past the limit, so a list larger than the whole limit is refused without being read.
  • A user already over the limit is refused before the managers are counted. Refused requests count toward the limit too, as with other count-based limits, so a user stays refused until the window ends.
  • The limit is a new rate_limits application setting, runner_manager_list_limit_per_user (default 300,000 managers per minute, 0 disables), on Admin > Settings > CI/CD and in the application settings API.
  • The new key starts in log mode: requests over the limit are logged until the ci_rate_limit_rest_runner_manager_list flag is enabled, and then get 429 Too Many Requests.
  • Docs: a new section in the CI/CD limits page, a line on the runners API, the settings API, and the GitLab.com settings table. The Changelog entry comes with the flag-removal MR.

Feature Flag: ci_rate_limit_rest_runner_manager_list

References

Screenshots or screen recordings

The new Maximum runner managers listed per user field on Admin > Settings > CI/CD > Continuous Integration and Deployment. Before (master) and After (this branch) are attached at the end of this description.

How to set up and validate locally

  1. In a Rails console, set a small limit:
    ApplicationSetting.current.update!(runner_manager_list_limit_per_user: 3)
  2. Pick a runner with two managers (each host that runs gitlab-runner with the runner's token adds one).
  3. With a personal access token (read_api scope), list its managers twice:
    curl --include --header "PRIVATE-TOKEN: <your_access_token>" "http://gdk.test:3000/api/v4/runners/<runner_id>/managers"
    Both return 200, and log/api_json.log shows rate_limit_state ending in _by_user_log:log for the second request.
  4. Run Feature.enable(:ci_rate_limit_rest_runner_manager_list), wait a minute, and repeat. The second request returns 429 with RateLimit-Name: runner_manager_list without reading the list, and a third is refused before the count statement runs.
  5. Set the limit to 0 and repeat: both return 200, with no count statement.
  6. Clean up: Feature.remove(:ci_rate_limit_rest_runner_manager_list) and reset the setting to 300000.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

before after

Edited by Leaminn Ma

Merge request reports

Loading
Loading