Block runner manager creation at a per-runner ceiling
What does this MR do and why?
!267942 (merged) trims a runner back to a cap by deleting managers that haven't contacted GitLab for two hours. It never deletes managers inside that grace period, and it pauses while loose foreign key cleanup is behind, so on its own it isn't a hard limit.
This MR adds a ceiling above the cap: a runner with at least ci_runner_manager_creation_ceiling managers can't create new ones, and the request that would create one gets 429 Too Many Requests. Existing managers keep working, and nothing is deleted.
- Counted by the cron worker, looked up on creation. Counting a runner's managers up to the ceiling is too slow for the request path.
Ci::Runners::RunnerManagerCapServicefrom !267942 (merged) already counts them every 10 minutes, so it now also records the runners at or above the ceiling in a Redis sorted set (Ci::Runners::RunnerManagerCreationCeiling). Creating a manager only runs aZSCOREon that set. - One count for both limits. Each runner is counted once, up to
max(cap, ceiling) + 1, and the count is used for both. - Recorded runners are counted on every run. A runner at the ceiling stops creating managers, so discovery would no longer find it. The worker reads the recorded runners (up to 100) and counts them before the discovered ones, so a run cut short by the runtime limit still renews them. A runner that drops below the ceiling is removed from the set.
- Recording doesn't pause with deletions. It happens before any deletion and continues while loose foreign key back-pressure pauses deletions. The log line has
trimming_pausedto show this. - Fails open. Entries expire after 30 minutes (three runs) unless a run renews them, so the ceiling stops applying if the worker stops. Redis errors allow the creation and are tracked.
- Bound. The set changes only when the worker runs, so a runner can pass the ceiling by one run's worth of new managers, which the creation limit from !267108 (merged) bounds.
Retry-Afteris 600 seconds, the worker's interval. - Only new managers are checked. As with !267108 (merged), requests with an existing
system_id, and requests without one (<legacy>), aren't affected. - Setting:
ci_runner_manager_creation_ceiling, inci_cd_settingsand the Settings API. Default150000, provisional, to be tuned in the log-only stage.0disables it. - Feature flags:
log_runner_manager_cap(from !267942 (merged)) now also logs runners at the ceiling and the requests that would be rejected.enforce_runner_manager_ceiling(new, runner actor, default off) rejects those requests. As withenforce_runner_manager_cap, the worker runs only when a flag is enabled for the instance, so enabling it for specific runners requireslog_runner_manager_cap.- The ceiling has its own enforcement flag so that it can be enforced only after existing runners are below it.
Logs
| Where | message |
Fields |
|---|---|---|
| Worker, for each runner at the ceiling | Runner manager creation ceiling reached |
class_name, runner_id, runner_managers_count (capped at max(cap, ceiling) + 1), limit, trimming_paused |
| Worker, at the end of each run | (existing log) | adds runners_at_ceiling |
| Request that would create a manager | Runner manager creation ceiling exceeded |
class_name, runner_id, runner_type, system_xid (truncated to 64 characters), path, limit, enforced |
The 429 response body:
{"message":{"error":"This runner has too many runner managers. Reuse the system ID of an existing runner manager, or delete unused runner managers."}}Follow-ups from the !267942 (merged) review
- The last deletion step now pads the two-hour online timeout by
Ci::RunnerManager::UPDATE_CONTACT_COLUMN_EVERY.max, likeCi::Runner.recent_queue_deadline, because the databasecontacted_atcan lag the real last contact by up to 55 minutes. MAX_RUNTIMEis now 4 minutes, so that a run stops before the 300-second apdex target ofurgency :lowjobs.
References
- Issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/636728
- Builds on !267942 (merged) (cron worker and cap, merged).
- Builds on !267108 (merged) (per-runner limit on runner manager creation, merged).
- Rollout issue: https://gitlab.com/gitlab-org/gitlab/-/work_items/638868
Points for reviewers
- Redis instance and structure. One sorted set in
Gitlab::Redis::SharedState: members are runner IDs, scores are expiry times. Reads ignore expired scores, and each write removes them, so one key holds the whole list with per-entry expiry, and the worker can list the recorded runners to count them again. I choseSharedStatebecause the set has to persist between runs: other sorted sets with expiring members use it, andRateLimitingonly holds counters. verifyis rejected too. As with !267108 (merged), any request that would create a manager is checked, includingPOST /runners/verify. A liveness probe that runsverifywith a newsystem_ideach time would fail at the ceiling. The cap keeps runners below the ceiling in normal operation, and the rollout issue checks the logs for such runners before enforcing.- No validation between the cap and the ceiling. Both only need to be
>= 0. A ceiling below the cap is allowed, though it makes the cap rarely apply.
Database review
No migrations. The new setting is a key in the ci_cd_settings JSON column. Two queries run by Ci::Runners::RunnerManagerCapService change or are new, every 10 minutes. Plans are from Database Lab (gitlab-production-ci). Timings and row counts are in an internal note.
1. Count of a runner's managers, up to the larger limit
The query from !267942 (merged), with LIMIT max(cap, ceiling) + 1 instead of LIMIT cap + 1:
SELECT COUNT(*) FROM (SELECT 1 AS one FROM "ci_runner_machines" WHERE "ci_runner_machines"."runner_id" = <runner_id> AND "ci_runner_machines"."runner_type" = <runner_type> LIMIT 150001) subquery_for_countPlans: first run https://console.postgres.ai/gitlab/projects/gitlab-production-ci/sessions/81095/commands/189223, second run https://console.postgres.ai/gitlab/projects/gitlab-production-ci/sessions/81095/commands/189224
(Index Only Scan on index_proj_type_ci_runner_machines_on_runner_id_type_system_xid)
For comparison, LIMIT 100001 from !267942 (merged): https://console.postgres.ai/gitlab/projects/gitlab-production-ci/sessions/81095/commands/189226
Only runners with more managers than the cap or the ceiling reach the LIMIT.
2. Recorded runners
SELECT "ci_runners".* FROM "ci_runners" WHERE "ci_runners"."id" IN (<up to 100 runner_ids>)Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-ci/sessions/81095/commands/189228 (Index Scan on each partition's primary key)
How to set up and validate locally
-
In the Rails console, disable the cap so that nothing is deleted, set a low ceiling, and enable logging only:
Feature.enable(:log_runner_manager_cap) ApplicationSetting.current.update!(ci_max_runner_managers_per_runner: 0, ci_runner_manager_creation_ceiling: 2) -
Create two managers for one runner, and run the worker:
runner = Ci::Runner.last 2.times { |i| runner.ensure_manager("s_ceiling_#{i}") } Ci::Runners::RunnerManagerCapCronWorker.new.performlog/application_json.loghas aRunner manager creation ceiling reachedline. -
Verify the runner with a new system ID:
curl --request POST "http://gdk.test:3000/api/v4/runners/verify" --form "token=<runner_token>" --form "system_id=s_ceiling_new"The request succeeds, and the log has a
Runner manager creation ceiling exceededline withenforced: false. -
Enable enforcement and verify with another new system ID:
Feature.enable(:enforce_runner_manager_ceiling)curl --include --request POST "http://gdk.test:3000/api/v4/runners/verify" --form "token=<runner_token>" --form "system_id=s_ceiling_other"The response is
429withRetry-After: 600. Verifying withsystem_id=s_ceiling_0still succeeds, because that manager exists.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist.