Limit runner version update enqueues per runner

What does this MR do and why?

Ci::RunnerManager#heartbeat enqueues Ci::Runners::ProcessRunnerVersionUpdateWorker whenever the version a runner reports differs from the cached one. This runs outside the throttle that limits database writes, and the first heartbeat of every new runner manager also enqueues it. The version string is set by the runner, and deduplicate :until_executing only collapses identical strings while they wait in the queue, so most of these jobs recompute a version whose status is already known.

This MR enqueues the worker only for versions a runner has not sent in the current hour, and caps the number of distinct versions per runner.

  • Only inside the existing new_version != version branch. Heartbeats with an unchanged version make no extra Redis calls or feature flag checks.
  • Per runner, not per runner manager. Keying by manager would be bypassed by sending a new system_id.
  • A Redis set of version digests per runner, with a fixed 1-hour window. SADD, EXPIRE ... NX and SCARD run in one MULTI on Gitlab::Redis::Cache. The worker is enqueued only when SADD adds a new member and the set has at most 50 members (MAX_RUNNER_VERSIONS_PER_WINDOW).
  • Two feature flags, both default off.
    • log_runner_version_update_limit: tracks versions and logs runners over the limit, but still enqueues as today.
    • enforce_runner_version_update_limit (runner actor): skips the enqueue for repeated versions and for versions over the limit. It also logs.
    • With both disabled, heartbeats behave exactly as before, with no Redis call.
  • Redis errors fail open. The exception is tracked and the worker is enqueued as before, so the heartbeat is not affected.

Effect of skipping an enqueue

  • ci_runner_versions is keyed by the version string and shared by the whole instance, so skipping a repeat enqueue for a known version does not change any upgrade status.
  • If a runner goes over the limit, only the enqueue is skipped. Ci::Runners::ReconcileExistingRunnerVersionsCronWorker still picks up every version in ci_runner_machines. The request and the rest of the heartbeat are not affected.

How it works

Path Before After (a flag enabled)
Version unchanged (almost all heartbeats) No enqueue No enqueue, no Redis call
Version changed, new to the runner this hour, at most 50 Enqueue 1 Redis MULTI + enqueue
Version changed, already seen for the runner this hour Enqueue 1 Redis MULTI, enqueue only in log mode
Version changed, over 50 for the runner this hour Enqueue 1 Redis MULTI + log, enqueue only in log mode

The log line (Gitlab::AppJsonLogger) has message: 'Runner version update limit exceeded', runner_id, runner_manager_id, versions_count, limit, and enforced. It does not include the version string, which is set by the runner.

References

Points for reviewers

I left these as they are and would like your opinion:

  1. Deduplicate per runner or per instance? The worker only takes the version string, and its result is shared by the whole instance. A per-version SET ... NX EX 3600 key would deduplicate across runners as well, and the per-runner set would then only count new strings for the limit. I kept per runner to match the plan in the issue, but the change would stay inside track_runner_version.
  2. The limit is per runner, not per user. A user with many runners can send up to 50 distinct versions per runner per hour. Bounding runner creation is covered by the related issue.
  3. The limit is a constant, not an application setting. A false positive only delays an upgrade status until the reconcile cron worker runs, so I did not think it needed to be configurable.
  4. The set keeps growing after the limit. SADD still adds members over the limit until the window expires. Each member is a 64-character digest. Adding only while under the limit would cap the set, at the cost of one more round trip.
  5. Gitlab::Redis::Cache or SharedState. I used Cache, like RedisCacheable. An evicted set only causes an extra enqueue.
  6. Changelog: none, because both flags are default off.

How to set up and validate locally

  1. Make sure the update_runner_versions_enabled application setting is on (the default).

  2. In the Rails console, enable logging and send 51 distinct versions from one runner manager:

    Feature.enable(:log_runner_version_update_limit)
    
    manager = Ci::RunnerManager.last
    51.times { |i| manager.heartbeat(version: "17.0.#{i}") }
    
    Gitlab::Redis::Cache.with { |redis| redis.scard("ci:runner:#{manager.runner_id}:version_update_digests") }
    # => 51

    log/application_json.log has one Runner version update limit exceeded line with enforced: false, and 51 jobs are enqueued.

  3. Enable enforcement and send one more new version and one repeated version:

    Feature.enable(:enforce_runner_version_update_limit)
    
    manager.heartbeat(version: '17.0.99')
    manager.heartbeat(version: '17.0.0')

    Neither is enqueued, and the log line for 17.0.99 has enforced: true.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Taka Nishida

Merge request reports

Loading
Loading