Block runner manager creation at a per-runner ceiling

What does this MR do and why?

!267942 (merged) trims a runner back to a cap by deleting managers that haven't contacted GitLab for two hours. It never deletes managers inside that grace period, and it pauses while loose foreign key cleanup is behind, so on its own it isn't a hard limit.

This MR adds a ceiling above the cap: a runner with at least ci_runner_manager_creation_ceiling managers can't create new ones, and the request that would create one gets 429 Too Many Requests. Existing managers keep working, and nothing is deleted.

  • Counted by the cron worker, looked up on creation. Counting a runner's managers up to the ceiling is too slow for the request path. Ci::Runners::RunnerManagerCapService from !267942 (merged) already counts them every 10 minutes, so it now also records the runners at or above the ceiling in a Redis sorted set (Ci::Runners::RunnerManagerCreationCeiling). Creating a manager only runs a ZSCORE on that set.
  • One count for both limits. Each runner is counted once, up to max(cap, ceiling) + 1, and the count is used for both.
  • Recorded runners are counted on every run. A runner at the ceiling stops creating managers, so discovery would no longer find it. The worker reads the recorded runners (up to 100) and counts them before the discovered ones, so a run cut short by the runtime limit still renews them. A runner that drops below the ceiling is removed from the set.
  • Recording doesn't pause with deletions. It happens before any deletion and continues while loose foreign key back-pressure pauses deletions. The log line has trimming_paused to show this.
  • Fails open. Entries expire after 30 minutes (three runs) unless a run renews them, so the ceiling stops applying if the worker stops. Redis errors allow the creation and are tracked.
  • Bound. The set changes only when the worker runs, so a runner can pass the ceiling by one run's worth of new managers, which the creation limit from !267108 (merged) bounds. Retry-After is 600 seconds, the worker's interval.
  • Only new managers are checked. As with !267108 (merged), requests with an existing system_id, and requests without one (<legacy>), aren't affected.
  • Setting: ci_runner_manager_creation_ceiling, in ci_cd_settings and the Settings API. Default 150000, provisional, to be tuned in the log-only stage. 0 disables it.
  • Feature flags:
    • log_runner_manager_cap (from !267942 (merged)) now also logs runners at the ceiling and the requests that would be rejected.
    • enforce_runner_manager_ceiling (new, runner actor, default off) rejects those requests. As with enforce_runner_manager_cap, the worker runs only when a flag is enabled for the instance, so enabling it for specific runners requires log_runner_manager_cap.
    • The ceiling has its own enforcement flag so that it can be enforced only after existing runners are below it.

Logs

Where message Fields
Worker, for each runner at the ceiling Runner manager creation ceiling reached class_name, runner_id, runner_managers_count (capped at max(cap, ceiling) + 1), limit, trimming_paused
Worker, at the end of each run (existing log) adds runners_at_ceiling
Request that would create a manager Runner manager creation ceiling exceeded class_name, runner_id, runner_type, system_xid (truncated to 64 characters), path, limit, enforced

The 429 response body:

{"message":{"error":"This runner has too many runner managers. Reuse the system ID of an existing runner manager, or delete unused runner managers."}}

Follow-ups from the !267942 (merged) review

  • The last deletion step now pads the two-hour online timeout by Ci::RunnerManager::UPDATE_CONTACT_COLUMN_EVERY.max, like Ci::Runner.recent_queue_deadline, because the database contacted_at can lag the real last contact by up to 55 minutes.
  • MAX_RUNTIME is now 4 minutes, so that a run stops before the 300-second apdex target of urgency :low jobs.

References

Points for reviewers

  1. Redis instance and structure. One sorted set in Gitlab::Redis::SharedState: members are runner IDs, scores are expiry times. Reads ignore expired scores, and each write removes them, so one key holds the whole list with per-entry expiry, and the worker can list the recorded runners to count them again. I chose SharedState because the set has to persist between runs: other sorted sets with expiring members use it, and RateLimiting only holds counters.
  2. verify is rejected too. As with !267108 (merged), any request that would create a manager is checked, including POST /runners/verify. A liveness probe that runs verify with a new system_id each time would fail at the ceiling. The cap keeps runners below the ceiling in normal operation, and the rollout issue checks the logs for such runners before enforcing.
  3. No validation between the cap and the ceiling. Both only need to be >= 0. A ceiling below the cap is allowed, though it makes the cap rarely apply.

Database review

No migrations. The new setting is a key in the ci_cd_settings JSON column. Two queries run by Ci::Runners::RunnerManagerCapService change or are new, every 10 minutes. Plans are from Database Lab (gitlab-production-ci). Timings and row counts are in an internal note.

1. Count of a runner's managers, up to the larger limit

The query from !267942 (merged), with LIMIT max(cap, ceiling) + 1 instead of LIMIT cap + 1:

SELECT COUNT(*) FROM (SELECT 1 AS one FROM "ci_runner_machines" WHERE "ci_runner_machines"."runner_id" = <runner_id> AND "ci_runner_machines"."runner_type" = <runner_type> LIMIT 150001) subquery_for_count

Plans: first run https://console.postgres.ai/gitlab/projects/gitlab-production-ci/sessions/81095/commands/189223, second run https://console.postgres.ai/gitlab/projects/gitlab-production-ci/sessions/81095/commands/189224 (Index Only Scan on index_proj_type_ci_runner_machines_on_runner_id_type_system_xid)

For comparison, LIMIT 100001 from !267942 (merged): https://console.postgres.ai/gitlab/projects/gitlab-production-ci/sessions/81095/commands/189226

Only runners with more managers than the cap or the ceiling reach the LIMIT.

2. Recorded runners

SELECT "ci_runners".* FROM "ci_runners" WHERE "ci_runners"."id" IN (<up to 100 runner_ids>)

Plan: https://console.postgres.ai/gitlab/projects/gitlab-production-ci/sessions/81095/commands/189228 (Index Scan on each partition's primary key)

How to set up and validate locally

  1. In the Rails console, disable the cap so that nothing is deleted, set a low ceiling, and enable logging only:

    Feature.enable(:log_runner_manager_cap)
    ApplicationSetting.current.update!(ci_max_runner_managers_per_runner: 0, ci_runner_manager_creation_ceiling: 2)
  2. Create two managers for one runner, and run the worker:

    runner = Ci::Runner.last
    2.times { |i| runner.ensure_manager("s_ceiling_#{i}") }
    Ci::Runners::RunnerManagerCapCronWorker.new.perform

    log/application_json.log has a Runner manager creation ceiling reached line.

  3. Verify the runner with a new system ID:

    curl --request POST "http://gdk.test:3000/api/v4/runners/verify" --form "token=<runner_token>" --form "system_id=s_ceiling_new"

    The request succeeds, and the log has a Runner manager creation ceiling exceeded line with enforced: false.

  4. Enable enforcement and verify with another new system ID:

    Feature.enable(:enforce_runner_manager_ceiling)
    curl --include --request POST "http://gdk.test:3000/api/v4/runners/verify" --form "token=<runner_token>" --form "system_id=s_ceiling_other"

    The response is 429 with Retry-After: 600. Verifying with system_id=s_ceiling_0 still succeeds, because that manager exists.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Taka Nishida

Merge request reports

Loading
Loading