Speed up the licenses v3 sync throttle

Why

The v3 package-metadata licenses bootstrap on GitLab.com has been running for over a day and is still working through npm.

LicenseV3SyncService sleeps throttle_rate after every ingest_slice_size records. At the current values (0.75s / 200) that's 3.75 ms of sleep per record. npm alone has 5.67 million records, so about 5.9 hours of npm's total time is just sleeping.

Measurements from production show sleep is a significant chunk of the time, but not the majority: npm is tracking toward roughly 25 hours total, of which about 6 hours is sleep — so roughly a third. This MR changes throttle_rate to 0.25s and ingest_slice_size to 1000, cutting sleep per record to 0.25 ms, a 15x reduction. These are the same values MalwareAdvisorySyncService already uses in production against the same pm_packages / pm_licenses tables, so there is precedent for this pacing.

max_sync_duration (4 minutes) and MAX_LEASE_LENGTH (5 minutes) are left unchanged, so the worker lease, the v2 route and StopSignal's arithmetic are all untouched. There is a useful side effect: faster throttling means many more shards now fit inside the existing 4-minute budget. Previously a shard used about 3 minutes of sleep out of that budget, so a run finished one shard and partially worked a second, which then got re-downloaded on the next run because the checkpoint only advances on completed shards. That repeated work mostly goes away here without changing the budget.

One caveat: because the budget is unchanged, there is still little headroom if a single shard's work ever exceeds 4 minutes. A shard that cannot finish inside the budget can never be checkpointed, so progress would stop. Raising max_sync_duration is a possible follow-up, out of scope here.

Risk: write pressure goes up during the bootstrap — larger transactions (1000 rows per upsert instead of 200) and less pause between them. Worth watching database saturation during the first bootstrap. All of this is behind the sync_v3_license_expressions feature flag.

Validate the v3 license expression sync end to ... (#627171 - closed)

Edited by Nick Ilieskou

Merge request reports

Loading
Loading