<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=602430) </details> <!--IssueSummary end--> Part of the malware advisory Rails sync (epic gitlab-org&20876). Extracted from the POC !226660. ## Scope Add the connectors that fetch malware advisory archives — from **PDS** (online) and from the **vendor path** (offline/air-gapped) — decompress them via `TarZstReader`, and yield data files to `PackageMetadata::SyncService`, conforming to the existing connector interface (`data_after(checkpoint) => Enumerator<DataFile>`, same as `Connector::Gcp` / `Connector::Offline`). ### Online — `Gitlab::PackageMetadata::Connector::Pds` - **Auth:** send the full `CloudConnector.headers(nil)` bundle **plus** an IJWT `Authorization: Bearer` token — i.e. `CloudConnector.headers(nil).merge('Authorization' => "Bearer #{ijwt}")`, with the token from `CloudConnector::Tokens.get(unit_primitive: :malware_advisories, resource: :instance)`. A bare `Authorization` header is **not** enough: PDS's auth middleware also requires `X-Gitlab-Realm` (matching the JWT `gitlab_realm` claim) and `X-Gitlab-Instance-Id` (matching `sub`), and returns `401 header mismatch` without them (verified against staging). Sync is instance-scoped, so pass `nil` for the user. The `malware_advisories` unit primitive is available in `gitlab-cloud-connector` 1.52.0, pinned on master — #602433, **done** ✅. - **Endpoint dispatch driven by the checkpoint** (`full_sync_target_sequence` marker landed in !242914): | Checkpoint state | `full_sync_target_sequence` | `sequence` | Request | |---|---|---|---| | Never synced | `NULL` | `0` | `GET /all` (full point-in-time snapshot) | | Bootstrap in progress | `= snapshot until` | last shard done | resume `GET /all`, skip already-ingested shards (**#603628**) | | Finalized (on deltas) | `NULL` | `> 0` | `GET /delta?timestamp=<sequence>` | - Parse the signed-URL response, download each archive, hand the bytes to `TarZstReader`, and yield entries as data files tagged with the sequence (delta `timestamp`, or `/all`'s `TarZstReader#checkpoint_until`) and `Entry#shard_index` / `#chunk`. ### Offline — `Gitlab::PackageMetadata::Connector::MalwareOffline` - Vendor-dir glob for air-gapped instances (mirrors `Connector::Offline`); reads the extracted archive's `TarZstReader#checkpoint_until` as the full-dataset sequence. ### Integration - `PackageMetadata::SyncConfiguration`: add `malware_advisory_configs` (`data_type: malware_advisories`, `version_format: v3`, per-`purl_type`) and a `Location.for_malware_advisories` returning `[:pds, <endpoint>]` online or `[:offline, <vendor path>]`. - `PackageMetadata::SyncService`: add the `:pds` branch in `#connector` and a `malware_advisories?` branch in `#ingest` (→ the ingestion service from #602431). ## Groundwork already merged — consume, don't rebuild (!242914) The resumable-first-sync groundwork is on `master`; this connector **consumes** it: - `pm_checkpoints.full_sync_target_sequence` (nullable `integer`) — the "bootstrap in progress vs finalized" marker. - `TarZstReader::Entry#shard_index` — base-16 `00..ff` → `0..255` (`nil` for flat delta entries). - `TarZstReader#checkpoint_until` — the `until` epoch from the bundled `checkpoint.json`. ## First sync — resumability is owned by #603628 The `/all` snapshot is a single point-in-time unit (hash-sharded `00..ff`, every chunk sharing one `until`), so it can't be resumed mid-way and continued via `/delta` without skipping un-ingested shards. **This MR's connector only picks the right endpoint from the checkpoint marker and yields files tagged with `shard_index`.** The per-shard checkpointing during `/all` and the finalize transition (`sequence = full_sync_target_sequence; full_sync_target_sequence = NULL`) are implemented in **#603628** (resumable per-shard first sync) alongside the sync worker/config (#602432). No first-sync sizing/strategy decision is made here — see #603628 for that. ## Single bulk `/delta` endpoint (#603947 / PDS !27) — no DB impact If the bulk multi-registry `/delta` lands ([pmdb-distribution-service!27](https://gitlab.com/gitlab-org/security-products/license-db/pmdb-distribution-service/-/merge_requests/27)): - `request_query` → `purl_type[]=<reg>` + `timestamp[<reg>]=<checkpoint.sequence>` (per-registry cursor) instead of one `purl_type`/`timestamp`. - `Connector::Pds.new` takes the **set** of registries + their checkpoints. - `parse_signed_urls` parses `{purl_types: {<reg>: [{delta, signed_url}]}}` and tags each archive with its `purl_type`. - `/all` (first sync) stays per-registry — the connector issues per-registry `/all` for fresh registries and one bulk `/delta` for synced ones. Full Rails impact + no-DB-change rationale: #603947. ## Dependencies - **Blocked by** the archive reader `TarZstReader` — #602428 (**done** ✅, merged). - **Blocked by** `pm_checkpoints` malware support — #602427 (**done** ✅, merged). - The `malware_advisories` unit primitive — #602433 (**done** ✅) — ships in `gitlab-cloud-connector` 1.52.0 (pinned on master), so real IJWT auth via `CloudConnector::Tokens` works now; a stubbed token stays convenient for specs. ## Acceptance - Given a checkpoint, the connector requests the correct endpoint (`/all` | resume `/all` | `/delta`) and yields data files with the right sequence. - Offline connector ingests a vendor-extracted dataset. - PDS requests carry the full `CloudConnector.headers` bundle (`X-Gitlab-Realm`, `X-Gitlab-Instance-Id`, …) alongside the IJWT `Authorization`; a bare `Authorization` header is rejected by PDS with `401 header mismatch`. - **Corrupt `checkpoint.json`:** if `TarZstReader#checkpoint_until` returns `nil` due to a parse error (corrupt/malformed file from PDS), log via `Gitlab::ErrorTracking` and **skip the sync run** — do not fall back to a full `/all` fetch and do not proceed with an unknown cursor.