Read CVE enrichment from a verbatim offline mirror

What does this MR do and why?

Offline Package Metadata Database sync cannot read a verbatim mirror of the CVE enrichment export, and the failure is silent.

CVE enrichment is the one dataset configured without a purl_type, and its export nests the data type one level deeper than the registry-partitioned ones:

Dataset Export layout Offline prefix before this MR
Advisories, licenses v2/<registry_id>/<sequence>/<chunk>.ndjson <base_uri>/v2/<registry_id>
CVE enrichment v2/cve_enrichment/<sequence>/<chunk>.ndjson <base_uri>/v2

Connector::Offline#data_after globs a fixed depth (*/*.<suffix>) under that prefix, so a mirror made the way the offline documentation says to make one (an rsync of the bucket) puts every enrichment chunk one level out of reach. The glob returns nothing, the sync ingests no rows, and nothing is logged. The online path is unaffected because Connector::Gcp#all_files lists by bucket prefix, which is depth-agnostic.

Verified on sealed instances at 19.2.2 and 19.3.1: flattening the mirror by one level makes ingest work. Operators have been applying that flatten by hand, so this MR accepts both layouts.

Approach

Offline#file_prefix descends into the data type directory when a purl_type-less configuration has one on disk, and stays at the shallower prefix when it does not.

Widening the glob is the smaller-looking change and the unsafe one. BaseConnector#sequence_and_chunk_from splits the path relative to the prefix and keeps the first two fields, so under a wider glob the literal cve_enrichment segment would be parsed as the sequence and the real chunk dropped. Resolving the prefix leaves the same <sequence>/<chunk> pair to parse.

Flattened installations keep working: the descent is gated on Dir.exist?, so their resolved prefix is byte-identical to today's, and both layouts yield the same <sequence>/<chunk> pairs, so a later verbatim re-mirror resumes from the existing checkpoint. Configurations with a purl_type return before the filesystem check, which covers advisories, licenses, and MalwareOffline.

The existing spec stubs Dir.glob with the exact pattern and base, which cannot show which prefix was resolved, so the three added examples walk a real Dir.mktmpdir tree: verbatim layout, flattened layout, and a purl_type configuration with a decoy data type directory. The describe-level before that installs the stub moved into the shared example that owns the paths it reads.

References

Screenshots or screen recordings

Not applicable: no user interface change.

How to set up and validate locally

bin/rspec ee/spec/lib/gitlab/package_metadata/connector/offline_spec.rb

Stash only the offline.rb hunk and re-run to see the verbatim-mirror example fail on master.

On an instance, mirror the enrichment bucket verbatim into <Rails.root>/vendor/package_metadata/cve_enrichment (the tree reads cve_enrichment/v2/cve_enrichment/<sequence>/<chunk>.ndjson), run PackageMetadata::CveEnrichmentSyncWorker.new.perform, and check PackageMetadata::CveEnrichment.count. Then flatten by one level and run the sync again.

Mirror layout Before After
Verbatim (v2/cve_enrichment/<sequence>/<chunk>) Nothing ingested, nothing logged Ingests
Flattened by hand (v2/<sequence>/<chunk>) Ingests Ingests
Advisories or licenses (v2/<registry_id>/<sequence>/<chunk>) Ingests Unchanged

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

This change was written with AI assistance: Claude Code drafted the fix and its specs, and the commit carries the matching AI-Assisted: yes and AI-Tools: Claude Code trailers. The export layouts were confirmed against the live buckets before the specs were written.

Merge request reports

Loading
Loading