Add a per-runner limit on runner manager creation
What does this MR do and why?
Runners register a runner manager for each new system_id they send to POST /runners/verify, POST /jobs/request, or the Job Router. This MR adds a per-runner limit on how many runner managers a runner can create.
- Only creations are counted. Requests for an existing
system_idreturn the existing manager without touching the limit, so periodicverifycalls, such as the runner Helm chart's liveness probe, are unaffected. - Requests without a
system_idare not counted. They all use the runner's<legacy>manager, so a runner has at most one of them. This keeps endpoints that don't send asystem_id, such asPOST /runners/reset_authentication_token, from being rejected. - Per runner, every 10 minutes. A new Labkit rule,
runner_manager_creation, keyed byci_runner. The limit comes from a new application setting,runner_manager_creation_limit(Settings API only,0disables it). - Two feature flags, both default off.
log_runner_manager_creation_limit: counts creations and logs the ones over the limit, but still creates them.enforce_runner_manager_creation_limit(runner actor): rejects creations over the limit with429andRetry-After: 600. It also logs.- With both disabled,
current_runner_managercallsensure_manager(system_xid)exactly as before: same queries, no Redis call.
- The default of 1,000 per 10 minutes is provisional. We will choose the enforced value after the log-only stage, by lowering the threshold step by step and checking which runners exceed it. See the rollout issue.
How it works
Ci::Runner#ensure_manager takes an optional before_creating: callable. When given, it looks up the manager first and calls before_creating only when it has to create one. The API helper passes the limit check as before_creating when either flag is enabled and the request has a system_id.
| Path | Before | After (a flag enabled) |
|---|---|---|
Existing system_id (almost all requests) |
1 SELECT |
1 SELECT |
New system_id |
1 SELECT + INSERT |
2 SELECT + 1 Redis call + INSERT |
The extra SELECT on the creation path is the same query ensure_manager already runs, on the existing unique index.
The log line (Gitlab::AppJsonLogger) has message: 'Runner manager creation limit exceeded', runner_id, runner_type, system_xid (truncated to 64 characters), path, limit, and enforced. When enforcement rejects a creation, the standard Application_Rate_Limiter_Request event (env: runner_manager_creation_request_limit) is also logged to auth_json.log, like other application rate limits.
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636728
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/637761
Points for reviewers
I left these as they are and would like your opinion:
- Bypass header:
Gitlab::ApplicationRateLimiter.throttled?doesn't honorX-GitLab-RateLimit-Bypass, so the limit also applies to allowlisted runner managers. I expect that to be fine, since theirsystem_ids are stable. - Changelog: none, because both flags are default off. The setting is visible in the Settings API regardless of the flags, though.
Retry-Afteris a fixed 600 seconds. The runner waits for it and retries up to 5 times, so a runner that hits the limit can wait longer than the window needs.throttled?returns only a boolean, so the actual reset time isn't available here.
How to set up and validate locally
-
In the Rails console, enable logging and set a low limit:
Feature.enable(:log_runner_manager_creation_limit) ApplicationSetting.current.update!(runner_manager_creation_limit: 1) -
Wait a minute for the settings and feature flag caches, then verify a runner with two new system IDs (replace the GDK URL and the token):
for id in s_first s_second; do curl --request POST "https://gdk.test:3443/api/v4/runners/verify" \ --form "token=<runner_token>" --form "system_id=$id" doneBoth return
200, andlog/application_json.loghas aRunner manager creation limit exceededline withenforced: falsefors_second. -
Enable enforcement, wait a minute again, and try a third new system ID:
Feature.enable(:enforce_runner_manager_creation_limit)system_id=s_thirdreturns429withRetry-After: 600, the log line hasenforced: true,log/auth_json.loghas anApplication_Rate_Limiter_Requestline, and no runner manager is created.s_firststill returns200.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.