Switch package-licenses sync to PDS, the v3 format and the v3 sync service

Orin Naamanrequested to merge
611228-licenses-pds-switch into master

Overview

  • Switches package-licenses metadata sync from GCS to PDS (PMDB Distribution Service).
  • On PDS, the licenses version format moves from v2 to v3 and sync runs through LicenseV3SyncService instead of SyncService.
  • Gated by the sync_v3_license_expressions feature flag, off by default, so this merges as a no-op until enabled. Only the jitter below takes effect immediately.
  • Disabling the flag puts the sync back on GCS and v2, resuming from the stored v2 checkpoint, which never advances while v3 is active. It does not remove v3 rows already ingested. Fallback is being scoped in Investigate v3-to-v2 fallback feasibility (#616368 - closed) • Orin Naaman • 19.4.

Routing

Vendored dir Flag Storage Format Service
present on :offline v2 SyncService
present off :offline v2 SyncService
absent on :pds v3 LicenseV3SyncService
absent off :gcp v2 SyncService

A vendored directory wins over the flag, so air-gapped installs stay on v2 either way. The flag only chooses between the two online destinations.

Changes

  • Location.for_licenses returns PDS when the flag is on, GCS when off. The vendored-directory checks still run first.
  • license_configs derives the version format from storage type: v3 on PDS, v2 on GCS and offline. Both connectors build their path prefix from the version format, so the two move together.
  • LicensesSyncWorker picks its sync service from the version format rather than reading the flag itself. That is what keeps the offline rows above on SyncService.
  • Adds a per-instance jitter of up to 5 minutes, seeded from the instance UUID, so self-managed instances do not all call PDS on the same cron tick. GitLab.com is a single instance and runs on the tick, matching MalwareAdvisoriesSyncWorker.
  • Adds the sync_v3_license_expressions flag definition, beta type, off by default.

Files

  • sync_configuration.rb - flag gate in Location.for_licenses; version format follows storage type.
  • licenses_sync_worker.rb - service routing, jitter re-enqueue, LEASE_TIMEOUT.
  • license_v3_sync_service.rb - class comment only.
  • checkpoint.rb - comment only, first_sync? now covers both v3 datasets.
  • beta/sync_v3_license_expressions.yml - new flag definition.
  • keela_excluded.yml - drops the licenses_pds_endpoint exclusion, it has a caller now.
  • sync_configuration_spec.rb - PDS, GCS and offline configs; 8-row for_licenses table; fresh-checkpoint regression.
  • licenses_sync_worker_spec.rb - routing table for every row above, cron tick vs delayed run, jitter_offset.
  • license_v3_sync_service_spec.rb - v3 checkpoint load; refuses a vendored v2 config.
  • v3_sync_service_spec.rb - asserts LicenseV3SyncService satisfies the dataset contract.

Warning

This must be merged last, after the other branches in the licenses PDS work. blocked by Prefer spdx_expression over spdx_identifier in ... (!250978 - merged), Ingest v3 PMDB license data via new spdx_expres... (!250674 - merged) and Add PDS endpoint constants for licenses (withou... (!251085 - merged)

Related issues

Edited by Orin Naaman

Merge request reports