We have to Sync the malware advisories data from the private bucket which is being handled in https://gitlab.com/groups/gitlab-org/-/work_items/20538 into GitLab. ### Unknowns 1. Auth for connecting to the private bucket for performing the data sync. - Can use ADS and cloud connector framework to get the token to communicate with the endpoint. Details available in https://gitlab.com/gitlab-com/content-sites/handbook/-/merge_requests/18229#note_3159494508 ### Implementation pieces 1. Tables and data model for the GLAM advisories. 2. Sync framework which ingests data into the tables. --- ## Work breakdown & structure Following the [POC outcome](https://gitlab.com/gitlab-org/gitlab/-/work_items/596491#note_3437298702) ([!226660](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/226660), closed), the Rails sync is decomposed into the issues below, grouped by dependency wave (everything within a wave can be worked in parallel; `is_blocked_by` links are set on each issue). Live status is in the GLQL table below; MRs are called out per row. **Issue status:** ✅ done · 🔄 in progress · 🚫 blocked · 📅 scheduled · ❌ not required. ### Wave 0 — foundations | Issue | Status | Scope | MR | |---|---|---|---| | [#602427](https://gitlab.com/gitlab-org/gitlab/-/work_items/602427) | ✅ | Reuse `pm_checkpoints` for malware checkpoints (enum + `for_malware_advisories` scope) | [!239992](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/239992) (merged) | | [#602428](https://gitlab.com/gitlab-org/gitlab/-/work_items/602428) | ✅ | `tar.zst` archive reader + `zstd-ruby` dependency | [!240026](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240026) (merged) | | [#589555](https://gitlab.com/gitlab-org/gitlab/-/work_items/589555) | ✅ | Determine DB schema; tables/models land via the two stacked table MRs | [!242951](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/242951) + [!242954](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/242954) (merged) | | ~~[#602429](https://gitlab.com/gitlab-org/gitlab/-/work_items/602429)~~ | ❌ | ~~Dedicated admin setting for malware PURL types~~ — **superseded** by product direction: reuse the existing `package_metadata_purl_types` setting; wiring folded into [#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432) | closed ([!240035](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240035) closed) | | [#602433](https://gitlab.com/gitlab-org/gitlab/-/work_items/602433) | ✅ | `gitlab-cloud-connector` `malware_advisories` unit primitive | **shipped**; `gitlab-cloud-connector` bumped to 1.52.0 (pinned on master) | ### Wave 1 — connector & ingestion | Issue | Status | Scope | Blocked by | MR | |---|---|---|---|---| | [#602430](https://gitlab.com/gitlab-org/gitlab/-/work_items/602430) | ✅ | PDS + offline connectors | ~~[#602427](https://gitlab.com/gitlab-org/gitlab/-/work_items/602427)~~, ~~[#602428](https://gitlab.com/gitlab-org/gitlab/-/work_items/602428)~~ | [!243687](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/243687) (merged) | | [#602431](https://gitlab.com/gitlab-org/gitlab/-/work_items/602431) | ✅ | Ingestion pipeline (data object, fabricator, ingestion service + tasks) — event publish dropped, see [#606613](https://gitlab.com/gitlab-org/gitlab/-/work_items/606613) | schema (~~[#589555](https://gitlab.com/gitlab-org/gitlab/-/work_items/589555)~~, via [!242951](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/242951)/[!242954](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/242954)) | [!240168](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240168) (merged) | | [#603628](https://gitlab.com/gitlab-org/gitlab/-/work_items/603628) | ✅ | Resumable per-shard first sync so a large `/all` bootstrap completes across many short, restart-safe jobs | ~~[#602430](https://gitlab.com/gitlab-org/gitlab/-/work_items/602430)~~, ~~[#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432)~~ | [!242914](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/242914) (merged — DB + reader groundwork) + [!244523](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/244523) (merged) | ### Wave 2 — orchestration | Issue | Status | Scope | Blocked by | MR | |---|---|---|---|---| | [#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432) | ✅ | Sync worker, configuration (reads the existing `package_metadata_purl_types`), scheduling, atomic per-archive checkpoint, lifecycle logging — FF-gated (`sync_malware_advisories`, default off); merged ahead of [#602431](https://gitlab.com/gitlab-org/gitlab/-/work_items/602431) | ~~[#602427](https://gitlab.com/gitlab-org/gitlab/-/work_items/602427)~~, ~~[#602430](https://gitlab.com/gitlab-org/gitlab/-/work_items/602430)~~, ~~[#602431](https://gitlab.com/gitlab-org/gitlab/-/work_items/602431)~~ | [!246641](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/246641) (merged) | | [#604792](https://gitlab.com/gitlab-org/gitlab/-/work_items/604792) | 🔄 | Per-instance jitter (up to 5 min) on the sync cron so instances (especially self-managed) don't all call PDS on the same tick | ~~[#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432)~~ | [!246820](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/246820) (ready for review) | ### Wave 3 — downstream & operational | Issue | Status | Scope | Blocked by | MR | |---|---|---|---|---| | [#602417](https://gitlab.com/gitlab-org/gitlab/-/work_items/602417) | 🚫 | Monitoring & metrics for the sync | ~~[#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432)~~ | — | | [#604583](https://gitlab.com/gitlab-org/gitlab/-/work_items/604583) | 📅 | Feature flag rollout — `sync_malware_advisories` (instance-wide): gates the sync worker and all PDS calls | ~~[#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432)~~ | flag defined in [!246641](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/246641) (merged); rollout via ChatOps | | [#604584](https://gitlab.com/gitlab-org/gitlab/-/work_items/604584) | 📅 | Feature flag rollout — `ingest_malware_advisories` (instance-wide): gates the DB upsert into `pm_malware_*` | ~~[#602431](https://gitlab.com/gitlab-org/gitlab/-/work_items/602431)~~, ~~[#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432)~~ | flag + upsert gate in [!240168](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240168) (merged); rollout via ChatOps | | [#606640](https://gitlab.com/gitlab-org/gitlab/-/work_items/606640) | 📅 | Make the sync available to **Premium** customers without the SSCS add-on (`dependency_scanning` is Ultimate-only, so the current gate excludes Premium) | ~~[#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432)~~ | — | | [#596488](https://gitlab.com/gitlab-org/gitlab/-/work_items/596488) | 🚫 | GitLab.com sync via offline vendor mode | ~~[#602430](https://gitlab.com/gitlab-org/gitlab/-/work_items/602430)~~, ~~[#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432)~~ | — | | [#594758](https://gitlab.com/gitlab-org/gitlab/-/work_items/594758) | 📅 | Air-gapped auth & download workflow | relates to [#602430](https://gitlab.com/gitlab-org/gitlab/-/work_items/602430) | — | | [#594792](https://gitlab.com/gitlab-org/gitlab/-/work_items/594792) | 📅 | Cleanup advisory data when the add-on expires | schema / ingestion | — | | [#589362](https://gitlab.com/gitlab-org/gitlab/-/work_items/589362) | 🔄 | [Spike] Identify data store for filtering use cases | — | — | ### Wave 4 — large-file / GA hardening follow-ups A single Sidekiq job can't run long (deploys interrupt jobs — [job-duration limit](https://docs.gitlab.com/development/sidekiq/#job-duration-limit)), and an `/all` snapshot decompresses to ~13× its compressed size, so large bootstraps need: | Issue | Status | Scope | Blocked by | MR | |---|---|---|---|---| | [#602885](https://gitlab.com/gitlab-org/gitlab/-/work_items/602885) | 📅 | Stream `.tar.zst` decompression to bound worker memory (the whole-archive read OOMs on large files) | ~~[#602428](https://gitlab.com/gitlab-org/gitlab/-/work_items/602428)~~ | — | > [#603627](https://gitlab.com/gitlab-org/gitlab/-/work_items/603627) was closed as a duplicate of [#602885](https://gitlab.com/gitlab-org/gitlab/-/work_items/602885). ### Related cross-epic change - **Bulk multi-registry `/delta`** ([#603947](https://gitlab.com/gitlab-org/gitlab/-/work_items/603947), PDS epic gitlab-org&21313 / pmdb-distribution-service!27): one `/delta` poll covers all registries (~7× fewer PDS calls). Rails impact (single bulk request + per-registry response parse, per-registry checkpoint advance) lands in [#602430](https://gitlab.com/gitlab-org/gitlab/-/work_items/602430) / [#602432](https://gitlab.com/gitlab-org/gitlab/-/work_items/602432); **no Rails DB change** — the existing per-PURL `pm_checkpoints` rows already supply the `timestamp[<purl>]` cursors. ### Out of scope (other epics) - **PDS service** (`/all` + `/delta` endpoints, OIDC middleware) — PMDB ingestion epic gitlab-org&20538. - **CVS scanning integration** (`GlobalMalwareAdvisoryScanWorker`, scanner) — companion MR [!226519](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/226519) / scanning epic. - **Ingestion event emission** (`IngestedMalwareAdvisoryEvent`) — not required by the sync; it exists only to feed the downstream SBOM/CVS scanning integration, so it was dropped from [#602431](https://gitlab.com/gitlab-org/gitlab/-/work_items/602431) ([!240168](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/240168)) and is tracked in [#606613](https://gitlab.com/gitlab-org/gitlab/-/work_items/606613) under integration epic gitlab-org&21156. ### Future scope (tracked, not scheduled) - [#604392](https://gitlab.com/gitlab-org/gitlab/-/work_items/604392) — widen `pm_checkpoints.sequence` + `first_sync_until` from `integer` to `bigint`. They hold Unix-second epoch cursors; `integer` overflows at **2038-01-19** (~11.6 yr headroom). Low priority; shared with public-advisory checkpoints so coordinate as a general `pm_checkpoints` change. ## Issue status ```glql display: table fields: title, state, labels("workflow::*"), milestone, assignee, updated sort: milestone asc query: epic = &20876 ``` > The `workflow::*` label column reflects in-dev / in-review status; `state` reflects open/closed. Add per-issue notes as comments on each issue.