Add per-instance jitter to the malware advisory sync cron

What does this MR do and why?

Adds a per-instance jitter (0–5 min) to the malware advisory sync cron (#604792 (closed)), so instances don't all call the PMDB Distribution Service (PDS) on the same wall-clock tick.

What

PackageMetadata::MalwareAdvisoriesSyncWorker (the */5 cron from #602432 (closed)) now applies a deterministic per-instance phase offset before running the sync. The polling interval is unchanged — only the phase.

Why

PDS is a shared, cluster-wide dependency. Every instance's Sidekiq-cron fires on the same wall-clock ticks, so without an offset they call PDS simultaneously — a thundering herd, dominated by the large number of self-managed instances. Staggering the phase smooths the arrival distribution and lowers peak requests/region (the PREP assessment sizes for the concentrated-burst case precisely because of this — see #602708).

How

  • ee/config/schedule.yml unchanged — a per-instance phase can't be expressed in a shared crontab, so the offset is applied inside the worker.
  • On the cron tick (perform), the worker re-enqueues its real run via perform_in(jitter_offset, true) and returns; the delayed run (perform(true)) does the actual sync. Same perform_in stagger idiom as SecretsManagement::ReconcileNamespaceSecretCountsCronWorker, but the offset is deterministic per instance rather than rand per tick.
  • Offset = SHA256(Gitlab::CurrentSettings.uuid) mod MAX_JITTER seconds (MAX_JITTER = 5.minutes):
    • stable across restartsuuid is the stable instance identifier (same one used for Service Ping / seat link), so an instance keeps its slot instead of re-randomising each boot;
    • evenly distributed — SHA256 spreads the fleet across the window; each self-managed instance carries a distinct uuid.
  • Safe under repeated ticks — a stacked jittered enqueue simply no-ops on the existing ExclusiveLeaseGuard lease.
  • Gating unchangedshould_run? (dependency_scanning license + sync_malware_advisories FF, default off + dev guard) is still checked before both the re-enqueue and the sync, so this stays inert until the flag is enabled.
  • Scoped to self-managed and Dedicated. GitLab.com (and staging) is a single instance, so jitter would only delay its sync with no herd to smooth. The worker gates the re-enqueue on Gitlab.com? (true for both gitlab.com and staging): on .com/staging it runs on the plain 5-minute tick; on self-managed/Dedicated it applies the per-instance offset. The instance check is scoped and both branches are covered by specs, so the Gitlab/AvoidGitlabInstanceChecks cop is disabled with a justification.

Local testing

Steps

# bin/rails runner — inspect this instance's deterministic offset + the fleet spread
worker = PackageMetadata::MalwareAdvisoriesSyncWorker
w = worker.new
w.send(:jitter_offset)                                # this instance's offset (seconds)
w.send(:jitter_offset) == w.send(:jitter_offset)      # stable across calls

require 'digest'
Array.new(12) { Digest::SHA256.hexdigest(SecureRandom.uuid).to_i(16) % worker::MAX_JITTER.to_i }.sort

Verification performed

instance uuid          : e40c4b82-4277-4fa7-8be1-a7578ab1af59
MAX_JITTER             : 300s (5m)
this instance's offset : 219s (3.65m)
  in [0, MAX_JITTER)?  : true
  stable on re-call?   : true

12 sample instance offsets (s): [2, 5, 18, 48, 56, 150, 202, 223, 225, 231, 251, 263]
spread across 0..299: min=2s max=263s

Offset is inside the 0–5 min window, deterministic/stable for a given uuid, and spreads evenly across a sample fleet.

Specs

bundle exec rspec ee/spec/workers/package_metadata/malware_advisories_sync_worker_spec.rb
# 11 examples, 0 failures

Covers: on self-managed/Dedicated the cron tick re-enqueues with the per-instance offset (and does not sync directly); on GitLab.com/staging the cron tick syncs directly via the lease (and does not re-enqueue); the delayed run syncs via the lease; gating (license / FF / dev) blocks all paths; and jitter_offset is in [0, MAX_JITTER) and equals SHA256(uuid) mod MAX_JITTER.

MR acceptance checklist

  • 5-minute polling interval unchanged; only the per-instance phase is offset.
  • Jitter scoped to self-managed and Dedicated; GitLab.com and staging run on the plain 5-minute tick.
  • Offset deterministic per instance (stable across restarts) and evenly distributed across the fleet.
  • Safe under repeated cron ticks (lease no-op); still FF-gated (sync_malware_advisories, default off).
  • Specs for the two-phase (tick → re-enqueue → sync) behaviour and the offset derivation; verified locally.

Related to #604792 (closed).

Edited by Bala Kumar

Merge request reports

Loading
Loading