## Summary The malware advisory sync cronjob ([#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432)) runs on a fixed ~5-minute schedule. Without a per-instance offset, **all GitLab instances — especially the large number of self-managed ones — fire the cron on the same wall-clock ticks and call PDS simultaneously** (a thundering herd). The PREP assessment already calls this out: per-region burst targets are validated for the pessimistic *concentrated-burst* case precisely because "with no jitter many instances fire near-simultaneously" (see [#602708](https://gitlab.com/gitlab-org/gitlab/-/work_items/602708)). Introduce a **jitter of up to 5 minutes** on the sync cron so runs are staggered across instances, smoothing PDS request load. ## Proposal - Add a **per-instance offset (0–5 min)** to the sync worker's schedule so instances don't align on the same minute. - Derive the offset **deterministically per instance** (e.g. from a stable instance/license identifier) so it is: - **stable across restarts** — an instance keeps its slot rather than re-randomising each boot, and - **well-distributed across the fleet** — different instances land on different offsets. - Affects only the *phase* of the cadence introduced in [#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432); the ~5-minute polling interval itself is unchanged. ## Why - PDS is a shared, cluster-wide dependency; smoothing the arrival distribution lowers peak req/region. - Self-managed instances vastly outnumber cells and are the primary source of synchronised bursts. ## Implementation plan The sync cadence is introduced in [#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432): a `CronjobQueue` worker (mirroring `PackageMetadata::AdvisoriesSyncWorker`) scheduled in `ee/config/schedule.yml` at `*/5 * * * *`. Every instance's Sidekiq-cron fires on the same wall-clock ticks, so the PDS calls align. This change keeps the 5-minute **interval** and adds a deterministic per-instance **phase offset** inside the worker: 1. **`ee/config/schedule.yml` — unchanged.** The `*/5 * * * *` entry stays; a per-instance phase cannot be expressed in a shared crontab, so the offset is applied in the worker. 2. **Sync worker — apply the offset.** The cron trigger re-enqueues the worker after this instance's offset (via `perform_in`) and returns; the delayed run does the actual sync. This reuses the established stagger idiom (`Worker.perform_in(<offset>, …)`) from e.g. [`SecretsManagement::ReconcileNamespaceSecretCountsCronWorker`](https://gitlab.com/gitlab-org/gitlab/-/blob/master/ee/app/workers/secrets_management/reconcile_namespace_secret_counts_cron_worker.rb) and `Search::NamespaceIndexIntegrityWorker` — except the offset is **deterministic per instance** rather than `rand` per tick. - **Offset** = `SHA256(Gitlab::CurrentSettings.uuid) mod MAX_JITTER` seconds, with `MAX_JITTER = 5.minutes`. The instance `uuid` is the same stable identifier used for Service Ping / seat link, so the offset is **stable across restarts**; SHA256 spreads the fleet **evenly** across the window; self-managed instances each carry a distinct `uuid`. - Safe under repeated cron ticks: the existing `ExclusiveLeaseGuard` (`LEASE_TIMEOUT`) means a stacked jittered enqueue simply no-ops on the lease. 3. **Specs.** Freeze `Gitlab::CurrentSettings.uuid` and assert: the cron hop calls `perform_in(sync_offset, true)`; the delayed hop invokes `SyncService`; `sync_offset` is identical for a fixed `uuid`, differs across `uuid`s, and always falls in `[0, 5.minutes)`. **Alternatives considered:** (a) `perform_in(rand(MAX_JITTER), …)` — simpler and matches the existing idiom, but re-randomises each tick (no stable slot per instance); (b) splitting into a thin cron worker that enqueues a separate sync worker. This plan follows the deterministic requirement stated above; (a) is a one-line change if determinism is dropped. <details> <summary>Diff — jitter added to the sync worker (relative to #602432, which mirrors <code>PackageMetadata::AdvisoriesSyncWorker</code>)</summary> ```diff # ee/app/workers/package_metadata/malware_advisories_sync_worker.rb # (worker name illustrative; the worker itself is introduced in #602432) module PackageMetadata class MalwareAdvisoriesSyncWorker include ApplicationWorker include CronjobQueue # rubocop:disable Scalability/CronWorkerContext include ExclusiveLeaseGuard LEASE_TIMEOUT = 5.minutes + MAX_JITTER = 5.minutes data_consistency :always feature_category :software_composition_analysis urgency :low idempotent! sidekiq_options retry: false worker_has_external_dependencies! - def perform + # `scheduled` distinguishes the cron trigger (false) from the delayed, + # per-instance-jittered run (true). + def perform(scheduled = false) return unless should_run? + # The cron fires on every instance at the same tick; defer the actual PDS + # call by this instance's deterministic offset to avoid a thundering herd. + unless scheduled + self.class.perform_in(self.class.sync_offset, true) + return + end + try_obtain_lease do SyncService.execute(data_type: 'malware_advisories', lease: exclusive_lease) end end + # Deterministic per-instance offset in [0, MAX_JITTER): hashing the stable + # instance UUID keeps each instance in a fixed slot across restarts and + # spreads the fleet evenly across the jitter window. + def self.sync_offset + Digest::SHA256.hexdigest(Gitlab::CurrentSettings.uuid.to_s).hex % MAX_JITTER.to_i + end + private def should_run? # ... feature-flag / add-on gate (see #602432) end def lease_timeout LEASE_TIMEOUT end end end ``` `ee/config/schedule.yml` stays as introduced in #602432 — interval unchanged, phase handled in the worker: ```yaml malware_advisories_sync_worker: # added in #602432 class: PackageMetadata::MalwareAdvisoriesSyncWorker cron: "*/5 * * * *" # interval unchanged; per-instance phase offset applied in the worker ``` </details> ## Out of scope - The retry/backoff behaviour and the 5-minute interval itself (owned by [#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432)). - Server-side rate limiting / capacity on PDS (PMDB ingestion epic).