Read CVE enrichment from a verbatim offline mirror
What does this MR do and why?
Offline Package Metadata Database sync cannot read a verbatim mirror of the CVE enrichment export, and the failure is silent.
CVE enrichment is the one dataset configured without a purl_type, and its
export nests the data type one level deeper than the registry-partitioned ones:
| Dataset | Export layout | Offline prefix before this MR |
|---|---|---|
| Advisories, licenses | v2/<registry_id>/<sequence>/<chunk>.ndjson |
<base_uri>/v2/<registry_id> |
| CVE enrichment | v2/cve_enrichment/<sequence>/<chunk>.ndjson |
<base_uri>/v2 |
Connector::Offline#data_after globs a fixed depth (*/*.<suffix>) under that
prefix, so a mirror made the way the offline documentation says to make one (an
rsync of the bucket) puts every enrichment chunk one level out of reach. The
glob returns nothing, the sync ingests no rows, and nothing is logged. The online
path is unaffected because Connector::Gcp#all_files lists by bucket prefix,
which is depth-agnostic.
Verified on sealed instances at 19.2.2 and 19.3.1: flattening the mirror by one level makes ingest work. Operators have been applying that flatten by hand, so this MR accepts both layouts.
Approach
Offline#file_prefix descends into the data type directory when a
purl_type-less configuration has one on disk, and stays at the shallower
prefix when it does not.
Widening the glob is the smaller-looking change and the unsafe one.
BaseConnector#sequence_and_chunk_from splits the path relative to the prefix
and keeps the first two fields, so under a wider glob the literal
cve_enrichment segment would be parsed as the sequence and the real chunk
dropped. Resolving the prefix leaves the same <sequence>/<chunk> pair to
parse.
Flattened installations keep working: the descent is gated on Dir.exist?, so
their resolved prefix is byte-identical to today's, and both layouts yield the
same <sequence>/<chunk> pairs, so a later verbatim re-mirror resumes from the
existing checkpoint. Configurations with a purl_type return before the
filesystem check, which covers advisories, licenses, and MalwareOffline.
The existing spec stubs Dir.glob with the exact pattern and base, which cannot
show which prefix was resolved, so the three added examples walk a real
Dir.mktmpdir tree: verbatim layout, flattened layout, and a purl_type
configuration with a decoy data type directory. The describe-level before
that installs the stub moved into the shared example that owns the paths it
reads.
References
- Related to #561085. This MR fixes one defect behind that issue and does not close it.
- The mirror procedure this makes work for enrichment: https://gitlab.com/gitlab-org/gitlab/-/blob/master/doc/topics/offline/quick_start_guide.md
Screenshots or screen recordings
Not applicable: no user interface change.
How to set up and validate locally
bin/rspec ee/spec/lib/gitlab/package_metadata/connector/offline_spec.rbStash only the offline.rb hunk and re-run to see the verbatim-mirror example
fail on master.
On an instance, mirror the enrichment bucket verbatim into
<Rails.root>/vendor/package_metadata/cve_enrichment (the tree reads
cve_enrichment/v2/cve_enrichment/<sequence>/<chunk>.ndjson), run
PackageMetadata::CveEnrichmentSyncWorker.new.perform, and check
PackageMetadata::CveEnrichment.count. Then flatten by one level and run the
sync again.
| Mirror layout | Before | After |
|---|---|---|
Verbatim (v2/cve_enrichment/<sequence>/<chunk>) |
Nothing ingested, nothing logged | Ingests |
Flattened by hand (v2/<sequence>/<chunk>) |
Ingests | Ingests |
Advisories or licenses (v2/<registry_id>/<sequence>/<chunk>) |
Ingests | Unchanged |
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
This change was written with AI assistance: Claude Code drafted the fix and its
specs, and the commit carries the matching AI-Assisted: yes and
AI-Tools: Claude Code trailers. The export layouts were confirmed against the
live buckets before the specs were written.