Support delta downloads on the offline malware advisory path
The offline malware advisory path only ever downloads whole snapshots. The script added in !250093 (merged) calls /v1/malware/advisories/all and writes v3/<registry_id>/full_dataset/, so every time the service publishes a new snapshot the admin pulls the whole thing again. PDS already serves /v1/malware/advisories/delta and the online connector uses it, but there's no way to consume a delta offline.
OfflineV3 gets in the way twice. It was called MalwareOffline when this was filed, so the permalinks below still point at the old name. It only globs inside full_dataset/ (malware_offline.rb#L58-71):
def shard_files
Dir.glob('*.tar.zst', base: full_dataset_dir).sort
end
def full_dataset_dir
File.join(file_prefix, FULL_DATASET_DIR)
endand it works out the checkpoint chunk by reading the filename as base 16 (#L33), so a delta archive named for its unix timestamp comes out with a garbage chunk:
chunk = shard_file.delete_suffix('.tar.zst').to_i(16)Proposal
After the first sync, the vendor directory is read as a timeline: deltas live at v3/<registry_id>/deltas/<unix_seconds>.tar.zst next to full_dataset/, and everything newer than pm_checkpoints.sequence gets applied oldest first, whether it's a delta or the snapshot, with the snapshot's timestamp taken from the until in its own checkpoint.json. On a tie the snapshot goes first, since it's a full replacement. The first sync still reads the snapshot only and ignores deltas — deltas can't bootstrap a registry, and it's the only path with per-shard resume. Because a newer snapshot can overtake the deltas beside it, copying in a fresh full dataset is how an admin forces a full resync, with no manual clearing of the deltas directory.
In the docs, give the download script an input to pick either a whole snapshot or the deltas since the local checkpoint.
Decided
- Continuity is assumed rather than checked. A delta's filename only names the end of its window, so a directory missing one in the middle looks identical to a complete one — there's no reliable way to detect it.
- Known follow-up, not handled here: a snapshot read outside a first sync has no per-shard resume, so an interrupted one restarts from the first shard. Same for a single delta archive. See Shard large /delta responses (same large-file r... (#608196) • Bala Kumar • 19.7
Paging
/delta returns at most DELTA_URL_LIMIT URLs per registry per call, 20 by default, and PDS hands back the oldest first so a client can page forward without gaps (handlers.go#L78-84):
// Timestamps arrive newest-first,
// so trimming from the front keeps the OLDEST n items and lets incremental
// clients paginate forward without gaps.
func OldestN(timestamps []string, n int) []string {So the script would need a paging loop, and an instance that has been off for a month would make a lot of round trips. Add #608196 to that, where one delta can grow to the size of a whole snapshot with no shard dimension and no resume, so the snapshot stays in the picture rather than deltas alone.
The script already skips the download when the service's until matches the local checkpoint.json, so the recurring cost today is one metadata call per registry. This is about the case where the snapshot has actually moved.
Relates to
Licenses
The offline licenses path has the same gap, raised in review of !252827 (merged) (thread). The download procedure documented there pulls a whole snapshot per run. Both blockers above are in the same reader, which !251674 (merged) made dataset-neutral, so the fix covers licenses and malware advisories together. The docs change applies to both scripts.