The bucket-label shortcut decides "caught up" from the sequence alone and never looks at the chunk, so a sync interrupted partway through a multi-file export never reads the rest of it. Introduced by the GCP bucket-label optimization that skips `listObjects`: https://gitlab.com/gitlab-org/gitlab/-/commit/57fcdcfb5831 ## What happens - `caught_up?` in `ee/lib/gitlab/package_metadata/connector/gcp.rb` is `checkpoint.sequence >= entries.first.timestamp`. The chunk is never consulted. - An equal sequence doesn't mean "finished that export", it means "somewhere inside it". A checkpoint sitting mid-export reads as caught up. - npm's checkpoint was `sequence 1788246925, chunk 4`. That export directory holds 11 files (`000000000`–`000000010`), so five were read and six were not. - `filtered_entries` compounds it: `entries.select { |e| e.timestamp > checkpoint.sequence }` drops the partially-read export entirely. Fixing `caught_up?` on its own still skips — both have to change together. ## Why it went unnoticed - The sync ran every 5 minutes for six days, decided "caught up" each time, logged success and ingested nothing. No error, no alert, and the checkpoint looked healthy. - You need a multi-file export interrupted partway to hit it. Almost every export is a single file, so the checkpoint sits at chunk 0 and the equality case genuinely does mean finished. cargo had 2 files and read both, correct there. - The other paths get it right. The `listObjects` fallback uses `file.checkpoint?`, which compares sequence and chunk, and the offline connector does the same. The label shortcut returns before any of that runs. ## What we lost - v2 exports are incremental deltas, not snapshots. The exporter runs `ExportBetween(since, until)` and derives `since` from the newest export directory already in the bucket, which is its own bookkeeping and independent of our checkpoint. A window we skip is never republished, so those npm rows don't come back on their own. - The npm license gap is closed today only because licenses moved to v3 and that migration ran a full snapshot that replaced the data wholesale. Luck, not design. - Advisories are still on v2 over GCP and still take the label path, so the same bug is live for them. `cve_enrichment` is explicitly excluded from the label path. > [!note] > > The chunk count is already there to use. The exporter writes it into the label (`1788246925-11`) and we already parse it into `LabelEntry#chunks` before ignoring it. Nothing needs to change in the exporter's label writing. ## Proposal 1. Start by investigating if there are licenses v2 PURL types since 19.0 with more than 1 file in the deltas. In theory only `npm` should have big deltas with more than 1 file. **Answer:** It can happen for Go but probably the number is very low. <details> <summary>Click to expand</summary> | purl_type/timestamp | file number | |---------------------|-------------| | npm/1788851695 | 11 | | go/1788959675 | 3 | | cargo/1788887105 | 2 | | go/1788877035 | 2 | | go/1788907502 | 2 | | go/1788963421 | 2 | | go/1788970535 | 2 | | go/1788992520 | 2 | </details> 3. Extend exporter so that based on a cli flag we can do the following: * When a delta (advisory or license) has more than 1 file (default size file) instead of creating one timestamp dir with all the files we should create a new timestamp dir (delta) for each file. For example if we have 3 files instead of using `1789035496` with 3 files we should have `1789035494`, `1789035495` and `1789035496` with one file each. 4. Update the run_exporter script in Deployment project so that we use this new flag. In theory this can be done for every PURL type but only for v2. License and Advisory. 5. We need to fill in the gaps. So run the exporter to re-export all license data between 19.0 and now only **for the affect PURL types.** This should fill in the gaps for all GitLab instances syncing v2 data. 6. Fix the source of the bug and backport it up to 2 milestones back (more if allowed). ## Open questions * Investigate if this happens also for advisory data. * **Answer:** Yes it can happen. Usually we get up to 5 files which should be easy to ingest but there is no guarantee. <details> <summary>Results querying v2 bucket</summary> | purl_type/timestamp | file number | |---------------------|-------------| | rpm/1788941419 | 5 | | deb/1788941471 | 2 | | rpm/1789027789 | 2 | | wolfi/1788941151 | 2 | </details> * Will we have the same problem for v3? * **Answer:** The bug will not be present in v3. However, we should implement the sharding of big deltas and resumable deltas on the Rails instance. My understanding is that with our 20mins budget an instance has 20mins to unzip the delta and upsert it. This should be enough time to upsert the majority of deltas. ## To Dos - [x] Exporter workaround https://gitlab.com/gitlab-org/security-products/license-db/license-exporter/-/merge_requests/288+ - [x] Deployment changes https://gitlab.com/gitlab-org/security-products/license-db/deployment/-/merge_requests/488+ - [x] Set the flag to `true` where needed. - [x] Update auto-compaction. - [x] Bump exporter. - [x] Run the exporter for the NPM data loss period - [x] Rails fix https://gitlab.com/gitlab-org/gitlab/-/merge_requests/255559+ - [x] Backport Rails fix - [x] 19.4 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/255958+ - [x] 19.3 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/255959+ - [x] 19.2 https://gitlab.com/gitlab-org/gitlab/-/merge_requests/255960+ - [ ] Ideally: - [ ] See if we want to export the 19.0-19.3 window for other purl types: https://gitlab.com/gitlab-org/gitlab/-/work_items/628038#note_3885376292 - [ ] Change auto compaction frequency (?) **Tests and verifications:** - [x] Test fix on GDK <details><summary>Test results</summary> The fix works for everything tested. The GDK was checked out 2367 commits before `ec40ad2f89cc`, the Rails-side fix to `caught_up?`, so Rails still ran the buggy version comparing only `checkpoint.sequence` and ignoring `chunk`. This proves the exporter-side split alone is enough for an instance without the Rails fix — the case for self-managed instances on older versions reading the same buckets. Advisories were used because `advisory_configs` hardcodes `VERSION_FORMAT_V2` and no vendored advisories directory exists, so this is v2 over GCS, no v3 or feature flag involved. Synced from a personal bucket `gs://onaaman-pmdb-split-test` holding real exporter output from https://gitlab.com/gitlab-org/security-products/license-db/license-exporter/-/merge_requests/288+s at commit `bd0919c`, into local Postgres with 2500 synthetic GLAD advisories for `cargo`. The only local change was swapping `Google::Cloud::Storage.anonymous` for an authenticated client, since the bucket is private; nothing in the sync path or repo was modified. | Case | Starting checkpoint | Rows | Final checkpoint | Time | |---|---|---|---|---| | Far behind, below the compacted block | `1789562000` | 0 → 2500 | `1789649494` | 41.8s | | Just behind, inside the label window | `1789649157` | 2500 → 2500 | `1789649494` | 11.2s | | Interrupted after the first new delta | `1789649492` | 2500 → 2500 | `1789649494` | 7.6s | | Already at the newest | `1789649494` | 2500 → 2500 | unchanged | 1.3s | Case 1 is the main one: a consumer below `T_compact` received the complete dataset, 11 files in the right order — the compacted block first, then the kept deltas on top. Case 4 took 1.3s against case 1's 41.8s, showing that compaction placing its block below `T_latest` does not make an up-to-date consumer re-download anything. The strongest result: the sync was interrupted after every single file, 11 times in a row. The final state was identical to an uninterrupted sync — 2500 rows, checkpoint at the newest directory. Every interruption landed on a delta boundary, so even the buggy `caught_up?` resumed correctly. That is exactly what the split buys. Not covered: - **Scale.** Everything was tested at 6 files. The real npm compaction run was 72. With `--delta-padding` at its default of 60, an export of more than 61 chunks fails after writing everything. Whether a real npm delta approaches that was not measured. - **Bucket label capacity.** The label holds about 5 timestamps, so a consumer more than 5 deltas behind falls back to `listObjects`. Case 1 took that path and was correct, but the cost was not measured. - **Only advisories were exercised on the consumer side.** v2 licenses use a different ingestion service, though the same connector, which is where the bug lives. - Synthetic data, not production volumes. Unrelated to this run, noted here for completeness: if one export's cutoff falls inside the previous run's directory range, those directories are overwritten with no error. This should not happen in practice — the exporters run on an hourly cadence, and the job setup alone takes longer than the few seconds a range spans. Separate finding: `PackageMetadata::AdvisoryDataObject.create` raises a clear `ArgumentError` for a missing `packages` attribute, but a well-formed JSON line with no `advisory` key falls through to `nil.has_key?` and raises `NoMethodError`. It takes corrupt bucket data to reach, so it's low priority and out of scope here. </details> - [x] Test auto compaction - https://gitlab.com/gitlab-org/security-products/license-db/deployment/-/merge_requests/488#note_3850365055