Add per-instance jitter to the malware advisory sync cron
What does this MR do and why?
Adds a per-instance jitter (0–5 min) to the malware advisory sync cron (#604792 (closed)), so instances don't all call the PMDB Distribution Service (PDS) on the same wall-clock tick.
What
PackageMetadata::MalwareAdvisoriesSyncWorker (the */5 cron from #602432 (closed)) now applies a deterministic per-instance phase offset before running the sync. The polling interval is unchanged — only the phase.
Why
PDS is a shared, cluster-wide dependency. Every instance's Sidekiq-cron fires on the same wall-clock ticks, so without an offset they call PDS simultaneously — a thundering herd, dominated by the large number of self-managed instances. Staggering the phase smooths the arrival distribution and lowers peak requests/region (the PREP assessment sizes for the concentrated-burst case precisely because of this — see #602708).
How
ee/config/schedule.ymlunchanged — a per-instance phase can't be expressed in a shared crontab, so the offset is applied inside the worker.- On the cron tick (
perform), the worker re-enqueues its real run viaperform_in(jitter_offset, true)and returns; the delayed run (perform(true)) does the actual sync. Sameperform_instagger idiom asSecretsManagement::ReconcileNamespaceSecretCountsCronWorker, but the offset is deterministic per instance rather thanrandper tick. - Offset =
SHA256(Gitlab::CurrentSettings.uuid) mod MAX_JITTERseconds (MAX_JITTER = 5.minutes):- stable across restarts —
uuidis the stable instance identifier (same one used for Service Ping / seat link), so an instance keeps its slot instead of re-randomising each boot; - evenly distributed — SHA256 spreads the fleet across the window; each self-managed instance carries a distinct
uuid.
- stable across restarts —
- Safe under repeated ticks — a stacked jittered enqueue simply no-ops on the existing
ExclusiveLeaseGuardlease. - Gating unchanged —
should_run?(dependency_scanning license +sync_malware_advisoriesFF, default off + dev guard) is still checked before both the re-enqueue and the sync, so this stays inert until the flag is enabled. - Scoped to self-managed and Dedicated. GitLab.com (and staging) is a single instance, so jitter would only delay its sync with no herd to smooth. The worker gates the re-enqueue on
Gitlab.com?(true for both gitlab.com and staging): on.com/staging it runs on the plain 5-minute tick; on self-managed/Dedicated it applies the per-instance offset. The instance check is scoped and both branches are covered by specs, so theGitlab/AvoidGitlabInstanceCheckscop is disabled with a justification.
Local testing
Steps
# bin/rails runner — inspect this instance's deterministic offset + the fleet spread
worker = PackageMetadata::MalwareAdvisoriesSyncWorker
w = worker.new
w.send(:jitter_offset) # this instance's offset (seconds)
w.send(:jitter_offset) == w.send(:jitter_offset) # stable across calls
require 'digest'
Array.new(12) { Digest::SHA256.hexdigest(SecureRandom.uuid).to_i(16) % worker::MAX_JITTER.to_i }.sortVerification performed
instance uuid : e40c4b82-4277-4fa7-8be1-a7578ab1af59
MAX_JITTER : 300s (5m)
this instance's offset : 219s (3.65m)
in [0, MAX_JITTER)? : true
stable on re-call? : true
12 sample instance offsets (s): [2, 5, 18, 48, 56, 150, 202, 223, 225, 231, 251, 263]
spread across 0..299: min=2s max=263sOffset is inside the 0–5 min window, deterministic/stable for a given uuid, and spreads evenly across a sample fleet.
Specs
bundle exec rspec ee/spec/workers/package_metadata/malware_advisories_sync_worker_spec.rb
# 11 examples, 0 failuresCovers: on self-managed/Dedicated the cron tick re-enqueues with the per-instance offset (and does not sync directly); on GitLab.com/staging the cron tick syncs directly via the lease (and does not re-enqueue); the delayed run syncs via the lease; gating (license / FF / dev) blocks all paths; and jitter_offset is in [0, MAX_JITTER) and equals SHA256(uuid) mod MAX_JITTER.
MR acceptance checklist
- 5-minute polling interval unchanged; only the per-instance phase is offset.
- Jitter scoped to self-managed and Dedicated; GitLab.com and staging run on the plain 5-minute tick.
- Offset deterministic per instance (stable across restarts) and evenly distributed across the fleet.
- Safe under repeated cron ticks (lease no-op); still FF-gated (
sync_malware_advisories, default off). - Specs for the two-phase (tick → re-enqueue → sync) behaviour and the offset derivation; verified locally.
Related to #604792 (closed).