PackageMetadata v2 sync skips unread chunks when the bucket-label path reports caught up

The bucket-label shortcut decides "caught up" from the sequence alone and never looks at the chunk, so a sync interrupted partway through a multi-file export never reads the rest of it.

Introduced by the GCP bucket-label optimization that skips listObjects: 57fcdcfb

What happens

  • caught_up? in ee/lib/gitlab/package_metadata/connector/gcp.rb is checkpoint.sequence >= entries.first.timestamp. The chunk is never consulted.
  • An equal sequence doesn't mean "finished that export", it means "somewhere inside it". A checkpoint sitting mid-export reads as caught up.
  • npm's checkpoint was sequence 1788246925, chunk 4. That export directory holds 11 files (000000000–000000010), so five were read and six were not.
  • filtered_entries compounds it: entries.select { |e| e.timestamp > checkpoint.sequence } drops the partially-read export entirely. Fixing caught_up? on its own still skips — both have to change together.

Why it went unnoticed

  • The sync ran every 5 minutes for six days, decided "caught up" each time, logged success and ingested nothing. No error, no alert, and the checkpoint looked healthy.
  • You need a multi-file export interrupted partway to hit it. Almost every export is a single file, so the checkpoint sits at chunk 0 and the equality case genuinely does mean finished. cargo had 2 files and read both, correct there.
  • The other paths get it right. The listObjects fallback uses file.checkpoint?, which compares sequence and chunk, and the offline connector does the same. The label shortcut returns before any of that runs.

What we lost

  • v2 exports are incremental deltas, not snapshots. The exporter runs ExportBetween(since, until) and derives since from the newest export directory already in the bucket, which is its own bookkeeping and independent of our checkpoint. A window we skip is never republished, so those npm rows don't come back on their own.
  • The npm license gap is closed today only because licenses moved to v3 and that migration ran a full snapshot that replaced the data wholesale. Luck, not design.
  • Advisories are still on v2 over GCP and still take the label path, so the same bug is live for them. cve_enrichment is explicitly excluded from the label path.

Note

The chunk count is already there to use. The exporter writes it into the label (1788246925-11) and we already parse it into LabelEntry#chunks before ignoring it. Nothing needs to change in the exporter's label writing.

Proposal

  1. Start by investigating if there are licenses v2 PURL types since 19.0 with more than 1 file in the deltas. In theory only npm should have big deltas with more than 1 file.

Answer: It can happen for Go but probably the number is very low.

Click to expand
purl_type/timestamp file number
npm/1788851695 11
go/1788959675 3
cargo/1788887105 2
go/1788877035 2
go/1788907502 2
go/1788963421 2
go/1788970535 2
go/1788992520 2
  1. Extend exporter so that based on a cli flag we can do the following:
    • When a delta (advisory or license) has more than 1 file (default size file) instead of creating one timestamp dir with all the files we should create a new timestamp dir (delta) for each file. For example if we have 3 files instead of using 1789035496 with 3 files we should have 1789035494, 1789035495 and 1789035496 with one file each.
  2. Update the run_exporter script in Deployment project so that we use this new flag. In theory this can be done for every PURL type but only for v2. License and Advisory.
  3. We need to fill in the gaps. So run the exporter to re-export all license data between 19.0 and now only for the affect PURL types. This should fill in the gaps for all GitLab instances syncing v2 data.
  4. Fix the source of the bug and backport it up to 2 milestones back (more if allowed).

Open questions

  • Investigate if this happens also for advisory data.
    • Answer: Yes it can happen. Usually we get up to 5 files which should be easy to ingest but there is no guarantee.
Results querying v2 bucket
purl_type/timestamp file number
rpm/1788941419 5
deb/1788941471 2
rpm/1789027789 2
wolfi/1788941151 2
  • Will we have the same problem for v3?
    • Answer: The bug will not be present in v3. However, we should implement the sharding of big deltas and resumable deltas on the Rails instance. My understanding is that with our 20mins budget an instance has 20mins to unzip the delta and upsert it. This should be enough time to upsert the majority of deltas.

To Dos

Tests and verifications:

  • Test fix on GDK

    Test results

    The fix works for everything tested.

    The GDK was checked out 2367 commits before ec40ad2f89cc, the Rails-side fix to caught_up?, so Rails still ran the buggy version comparing only checkpoint.sequence and ignoring chunk. This proves the exporter-side split alone is enough for an instance without the Rails fix — the case for self-managed instances on older versions reading the same buckets. Advisories were used because advisory_configs hardcodes VERSION_FORMAT_V2 and no vendored advisories directory exists, so this is v2 over GCS, no v3 or feature flag involved.

    Synced from a personal bucket gs://onaaman-pmdb-split-test holding real exporter output from https://gitlab.com/gitlab-org/security-products/license-db/license-exporter/-/merge_requests/288+s at commit bd0919c, into local Postgres with 2500 synthetic GLAD advisories for cargo. The only local change was swapping Google::Cloud::Storage.anonymous for an authenticated client, since the bucket is private; nothing in the sync path or repo was modified.

    Case Starting checkpoint Rows Final checkpoint Time
    Far behind, below the compacted block 1789562000 0 → 2500 1789649494 41.8s
    Just behind, inside the label window 1789649157 2500 → 2500 1789649494 11.2s
    Interrupted after the first new delta 1789649492 2500 → 2500 1789649494 7.6s
    Already at the newest 1789649494 2500 → 2500 unchanged 1.3s

    Case 1 is the main one: a consumer below T_compact received the complete dataset, 11 files in the right order — the compacted block first, then the kept deltas on top. Case 4 took 1.3s against case 1's 41.8s, showing that compaction placing its block below T_latest does not make an up-to-date consumer re-download anything.

    The strongest result: the sync was interrupted after every single file, 11 times in a row. The final state was identical to an uninterrupted sync — 2500 rows, checkpoint at the newest directory. Every interruption landed on a delta boundary, so even the buggy caught_up? resumed correctly. That is exactly what the split buys.

    Not covered:

    • Scale. Everything was tested at 6 files. The real npm compaction run was 72. With --delta-padding at its default of 60, an export of more than 61 chunks fails after writing everything. Whether a real npm delta approaches that was not measured.
    • Bucket label capacity. The label holds about 5 timestamps, so a consumer more than 5 deltas behind falls back to listObjects. Case 1 took that path and was correct, but the cost was not measured.
    • Only advisories were exercised on the consumer side. v2 licenses use a different ingestion service, though the same connector, which is where the bug lives.
    • Synthetic data, not production volumes.

    Unrelated to this run, noted here for completeness: if one export's cutoff falls inside the previous run's directory range, those directories are overwritten with no error. This should not happen in practice — the exporters run on an hourly cadence, and the job setup alone takes longer than the few seconds a range spans.

    Separate finding: PackageMetadata::AdvisoryDataObject.create raises a clear ArgumentError for a missing packages attribute, but a well-formed JSON line with no advisory key falls through to nil.has_key? and raises NoMethodError. It takes corrupt bucket data to reach, so it's low priority and out of scope here.

  • Test auto compaction - https://gitlab.com/gitlab-org/security-products/license-db/deployment/-/merge_requests/488#note_3850365055

Edited by Orin Naaman