PDS connector abstraction

TLDR

Extract a reusable PDS connector. The malware PDS connector is generic code with a malware name. Split it so the shared half is shared and malware keeps only its endpoint, token scope and log wording. No licenses code.

Overview

  • Rails already talks to PDS, for malware advisories. The connector that does it is named for malware, but almost nothing in it is malware specific.
  • The request shapes, the instance token and identity headers, snapshot and delta selection, per-shard resume, the delta cursor, the registry identifier mapping and the per-registry skip when PDS rejects a request are all properties of the v3 distribution contract, not of one dataset.
  • What is malware specific is small: which scope the token carries, which endpoint it calls, and the wording of its log lines.

Note

Malware only. No licenses configuration, no new endpoint. The extraction and the dispatch are behaviour preserving; scope item 4 is not, see Behavioural changes below.

Scope

  • The dataset-neutral parts of the malware PDS connector become shared, with malware as a subclass carrying only the three things above.

  • Asking for a PDS connector returns the one for the dataset in hand rather than always the malware one.

  • The per-registry skip stays in the shared part, so any dataset inherits it. A registry PDS does not serve is logged and skipped rather than failing the run.

Behavioural changes

Scope item 4 cannot be delivered without changing behaviour. The guardrail above holds for the extraction and the dispatch, but the marker is a different matter: the reason a snapshot does not survive across runs today is that nothing sets it. Three changes, all confined to v3 PDS sync and all behind the default-off sync_malware_advisories flag.

  1. A v3 first sync now resumes its snapshot across runs. MalwareAdvisorySyncService, the service that actually syncs malware, never set full_sync_target_sequence at all. An interrupted first sync therefore lost first_sync?, flipped to /delta on the next run, and dropped the remaining shards.

  2. Snapshot shards are checkpointed individually. The same service batched its checkpoint per archive, advancing only when file.sequence changed. Every shard of a snapshot shares the snapshot's until, so the sequence never changed across a first sync and nothing was committed until the whole snapshot drained: an interrupted first sync committed nothing and the next run re-downloaded from shard 0. A shard is a complete archive on its own, so it is now committed as soon as it is ingested. Delta syncs keep their per-archive batching.

  3. SyncService's marker gate is now sync_config.v3? rather than a malware data-type check. The old predicate was wrong in principle, since the marker belongs to the v3 contract rather than to one dataset, and unreachable in practice, since no v3 data type reaches this service today.

Not changed: no licenses configuration, no new endpoint, and no change to malware's request shapes, token scope, or checkpoint semantics on the delta path. Malware's existing connector specs are untouched and still pass.

One log line changed wording as a side effect of interpolating the dataset label: Failed to download malware advisory archive from signed URL now reads ... malware advisories archive .... Nothing matches on it.

Definition of done

  • Malware behaves identically to today, asserted against its existing tests rather than assumed.

  • A second dataset can be constructed in tests and reaches its own endpoint with its own scope.

  • An unsupported registry is skipped rather than failing the run, for either dataset.

Key files

  • ee/lib/gitlab/package_metadata/connector/malware_pds.rb: the connector to extract from
  • ee/lib/gitlab/package_metadata/connector/base_connector.rb: registry identifiers and shared helpers
  • ee/app/services/package_metadata/sync_service.rb and ee/app/services/package_metadata/malware_advisory_sync_service.rb: both hold a connector dispatch, and both hardcode the malware one
Edited by Orin Naaman