Add a per-runner limit on runner manager creation

What does this MR do and why?

Runners register a runner manager for each new system_id they send to POST /runners/verify, POST /jobs/request, or the Job Router. This MR adds a per-runner limit on how many runner managers a runner can create.

  • Only creations are counted. Requests for an existing system_id return the existing manager without touching the limit, so periodic verify calls, such as the runner Helm chart's liveness probe, are unaffected.
  • Requests without a system_id are not counted. They all use the runner's <legacy> manager, so a runner has at most one of them. This keeps endpoints that don't send a system_id, such as POST /runners/reset_authentication_token, from being rejected.
  • Per runner, every 10 minutes. A new Labkit rule, runner_manager_creation, keyed by ci_runner. The limit comes from a new application setting, runner_manager_creation_limit (Settings API only, 0 disables it).
  • Two feature flags, both default off.
    • log_runner_manager_creation_limit: counts creations and logs the ones over the limit, but still creates them.
    • enforce_runner_manager_creation_limit (runner actor): rejects creations over the limit with 429 and Retry-After: 600. It also logs.
    • With both disabled, current_runner_manager calls ensure_manager(system_xid) exactly as before: same queries, no Redis call.
  • The default of 1,000 per 10 minutes is provisional. We will choose the enforced value after the log-only stage, by lowering the threshold step by step and checking which runners exceed it. See the rollout issue.

How it works

Ci::Runner#ensure_manager takes an optional before_creating: callable. When given, it looks up the manager first and calls before_creating only when it has to create one. The API helper passes the limit check as before_creating when either flag is enabled and the request has a system_id.

Path Before After (a flag enabled)
Existing system_id (almost all requests) 1 SELECT 1 SELECT
New system_id 1 SELECT + INSERT 2 SELECT + 1 Redis call + INSERT

The extra SELECT on the creation path is the same query ensure_manager already runs, on the existing unique index.

The log line (Gitlab::AppJsonLogger) has message: 'Runner manager creation limit exceeded', runner_id, runner_type, system_xid (truncated to 64 characters), path, limit, and enforced. When enforcement rejects a creation, the standard Application_Rate_Limiter_Request event (env: runner_manager_creation_request_limit) is also logged to auth_json.log, like other application rate limits.

References

Points for reviewers

I left these as they are and would like your opinion:

  1. Bypass header: Gitlab::ApplicationRateLimiter.throttled? doesn't honor X-GitLab-RateLimit-Bypass, so the limit also applies to allowlisted runner managers. I expect that to be fine, since their system_ids are stable.
  2. Changelog: none, because both flags are default off. The setting is visible in the Settings API regardless of the flags, though.
  3. Retry-After is a fixed 600 seconds. The runner waits for it and retries up to 5 times, so a runner that hits the limit can wait longer than the window needs. throttled? returns only a boolean, so the actual reset time isn't available here.

How to set up and validate locally

  1. In the Rails console, enable logging and set a low limit:

    Feature.enable(:log_runner_manager_creation_limit)
    ApplicationSetting.current.update!(runner_manager_creation_limit: 1)
  2. Wait a minute for the settings and feature flag caches, then verify a runner with two new system IDs (replace the GDK URL and the token):

    for id in s_first s_second; do
      curl --request POST "https://gdk.test:3443/api/v4/runners/verify" \
        --form "token=<runner_token>" --form "system_id=$id"
    done

    Both return 200, and log/application_json.log has a Runner manager creation limit exceeded line with enforced: false for s_second.

  3. Enable enforcement, wait a minute again, and try a third new system ID:

    Feature.enable(:enforce_runner_manager_creation_limit)

    system_id=s_third returns 429 with Retry-After: 600, the log line has enforced: true, log/auth_json.log has an Application_Rate_Limiter_Request line, and no runner manager is created. s_first still returns 200.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Taka Nishida

Merge request reports

Loading
Loading