<!--IssueSummary start--> <details> <summary> Everyone can contribute. [Help move this issue forward](https://handbook.gitlab.com/handbook/marketing/developer-relations/contributor-success/community-contributors-workflows/#contributor-links) while earning points, leveling up and collecting rewards. </summary> - [Close this issue](https://contributors.gitlab.com/manage-issue?action=close&projectId=278964&issueIid=612097) </details> <!--IssueSummary end--> ## Summary Ship a post-deploy migration that resets the malware rows in `pm_checkpoints`, so every GitLab instance re-drives the full advisory snapshot once and CVS picks up the malware advisories that were ingested before the scanner existed. The migration ships in a release **after** the CVS code is merged and confirmed working, not alongside it. That ordering is what makes a single migration correct for every instance, self-managed and GitLab.com alike. ## A migration of this name already shipped in %19.3, and it does not count `db/post_migrate/20260813000002_reset_malware_advisory_checkpoints.rb` is already on master, merged 2026-08-14 in `216e370c8e59` as part of the package-name **normalization** work ([#613223](https://gitlab.com/gitlab-org/gitlab/-/work_items/613223) / [!249953](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/249953)). It is mechanically identical to what this issue asks for: it zeroes `sequence` and `chunk` for `data_type = 4` so the next sync re-drives `/all`. **It does not satisfy this issue, and it made the situation worse rather than better.** The whole premise here is that the reset must run in a release where the CVS scanner is *already enabled*, so that re-driving `/all` republishes ingestion events for advisories inside the 14-day window. When that migration ran in %19.3 there was no scanner, no scan service, and no event publication at all — [#606613](https://gitlab.com/gitlab-org/gitlab/-/work_items/606613) is still unbuilt. The one-shot snapshot was consumed with nothing listening, which is exactly the failure this issue exists to avoid and the same shape as [#607059](https://gitlab.com/gitlab-org/gitlab/-/work_items/607059). So a **second, separate reset is required in %19.4**, once the scanner is live. Two consequences for whoever picks this up: - The new migration needs a **different class name**. Rails requires migration class names to be unique, so `ResetMalwareAdvisoryCheckpoints` is taken. Something like `ResetMalwareAdvisoryCheckpointsForCvs` makes the distinct purpose obvious in `db/post_migrate/`. - Do not treat the existing migration as prior art for "this already ran, so the tables are fresh". By %19.4 the checkpoints have advanced again through normal delta sync, so the advisories ingested between the %19.3 reset and the scanner going live are in exactly the same skipped state described below. This issue was also carrying a `workflow::complete` label while still open, most likely because the %19.3 migration was mistaken for this work. That label has been removed. ## Problem CVS scans an advisory once, at ingestion, and never revisits it. There is no cron that re-scans advisories, so the gap does not self-heal. Malware advisory ingestion is enabled ahead of the scanner. Every advisory ingested in that window would never be scanned, even after the scanner and its event subscription go live. For malware this matters more than for CVEs, because no separate pipeline path creates these vulnerabilities today. This is the same shape as https://gitlab.com/gitlab-org/gitlab/-/work_items/607059, where the sync advanced its checkpoint while ingestion was disabled, consuming the snapshot without the data reaching the database. ## What the reset recovers, given the 14-day window Delta sync publishes an event for an advisory only as it **arrives**. Rows already present in `pm_malware_advisories` are not re-upserted by a delta, so they never produce one. Ingestion is enabled in %19.3 and the scanner ships later, so by then the table already holds a month or more of advisories that arrived before any consumer existed. Normal delta sync will never revisit them, which means even advisories published inside the trailing 14 days are permanently skipped purely because of when they happened to arrive. The reset fixes exactly that: re-driving `/all` re-upserts every row and publishes events for everything still inside the window. It aligns CVS with its own policy at the point of enablement. Advisories older than 14 days are out of CVS scope by policy rather than by accident, and stay covered by the SBOM/CI path (https://gitlab.com/gitlab-org/gitlab/-/work_items/612091), which applies no age filter. ## Why a checkpoint reset, and not a backfill `Checkpoint#first_sync?` returns true when `full_sync_target_sequence` is present or `sequence` is zero. Zeroing the cursor makes the next sync fetch the full `/all` snapshot again and re-upsert every advisory through the normal ingestion path. Because the upsert uses `on_duplicate: :update`, re-upserted rows come back in the `RETURNING` clause and publish events for updates as well as inserts. So the reset needs no backfill code, no event emission from the migration, and no new code path to test. It reuses the ingestion path that is already exercised in production. A migration that publishes ingestion events directly was rejected: it recovers only the subset inside the publication-age window rather than everything missed, and migrations must not depend on application models or emit application events. ## Gates: what must be true before this MR merges The migration is only correct if the scanner is already doing work by the time it runs. All of the following must hold first: 1. Event publication merged — https://gitlab.com/gitlab-org/gitlab/-/work_items/606613 2. CVS scanner, scan service and worker merged — https://gitlab.com/gitlab-org/gitlab/-/work_items/612094 3. ~~Publication-age window decided and implemented~~ — **done.** https://gitlab.com/gitlab-org/gitlab/-/work_items/612096 settled on reusing `14.days` unchanged. The reset is scoped accordingly: it recovers the advisories inside that window, which is the set CVS treats as in scope. 4. CVS feature flag at 100% on production, and default-enabled or removed in the code self-managed receives — https://gitlab.com/gitlab-org/gitlab/-/work_items/612098 5. Validated on staging: a manual checkpoint reset there produces re-ingestion, scan events, and vulnerabilities on a known-affected project. **Why this ordering means one migration covers everything.** Because the migration ships a release later than the scanner, by the time it runs on GitLab.com the flag is already at 100%, and by the time a self-managed instance upgrades to that release the scanner is on by default in the code they receive. Shipping the migration in the same release as the flag-gated scanner would reset the checkpoint while scanning was still disabled, burn the snapshot a second time, and reproduce https://gitlab.com/gitlab-org/gitlab/-/work_items/607059 — with the fix as the cause. ## The migration Post-deploy migration in `db/post_migrate/`. `pm_checkpoints` is `gitlab_schema: gitlab_pm`, and `malware_advisories` is `data_type = 4` in `Enums::PackageMetadata::DATA_TYPES`. ```ruby # frozen_string_literal: true class ResetMalwareAdvisoryCheckpoints < Gitlab::Database::Migration[2.3] restrict_gitlab_migration gitlab_schema: :gitlab_pm milestone '<release in which this ships>' MALWARE_ADVISORIES_DATA_TYPE = 4 # Zeroing the cursor makes Checkpoint#first_sync? true, so the next sync fetches # the full /all snapshot and re-upserts every advisory through the normal path, # which publishes the ingestion events CVS needs. See the issue for why this runs # a release after the scanner rather than alongside it. def up execute(<<~SQL) UPDATE pm_checkpoints SET sequence = 0, chunk = 0, full_sync_target_sequence = NULL, updated_at = NOW() WHERE data_type = #{MALWARE_ADVISORIES_DATA_TYPE} SQL end # Irreversible: the previous cursor positions are not recorded anywhere, and # restoring them would re-open the gap this migration exists to close. def down # no-op end end ``` Notes on the shape: - No application models and no events, which is what makes this acceptable as a migration. - The table holds one row per purl type per data type, so a single `UPDATE` is enough. No batching, no `disable_ddl_transaction!`. - `updated_at` is set so the reset is visible when debugging. - Scoped by `data_type` so the `advisories`, `licenses` and `cve_enrichment` cursors are untouched. ## What happens after it runs 1. The next `PackageMetadata::MalwareAdvisoriesSyncWorker` tick (every 5 minutes) sees `first_sync?` as true and requests `/all` per configured registry. 2. Ingestion re-upserts every advisory. Rows that already exist are updated, so they return from `RETURNING` and publish `IngestedMalwareAdvisoryEvent`. 3. `GlobalMalwareAdvisoryScanWorker` consumes those events and scans `sbom_occurrences`, subject to its `concurrency_limit` of 10 and `deduplicate :until_executed`. 4. The resumable per-shard first sync (https://gitlab.com/gitlab-org/gitlab/-/work_items/603628) spreads the bootstrap across many short jobs rather than one long one. ## Verification Capture before and after, on staging first and then production: - Before: `pm_checkpoints` rows for `data_type = 4`, and counts from `pm_malware_advisories` / `pm_malware_affected_packages`. - Immediately after the migration: those checkpoint rows are zeroed and nothing else changed. - Over the following sync cycles: `malware_advisory_sync` logs show `sync_mode` of `full`, `malware_advisory_ingestion` logs show advisories upserted, and `updated_at` on existing advisory rows advances. - End to end: a project with a known malicious dependency gains a malware vulnerability. The verification fixture can be used for this: https://gitlab.com/gitlab-org/govern/threat-insights-demos/verification-projects/bala-test-group/malware-sbom-verification ## Risks - **PDS load.** Every instance re-downloads the full `/all` snapshot per registry. Self-managed is staggered by upgrade timing; GitLab.com is a single hit. Confirm capacity with the PDS team before merging. - **Event volume.** The re-drive republishes events for advisories that were already scanned. Vulnerability creation is idempotent through the UUID upsert, so this is duplicated work rather than duplicated data, but worker queue depth should be watched during the first cycles. - **Window dependency.** If the publication-age window is left at the public-advisory default, the re-drive is filtered and the snapshot re-download is largely wasted. This is why gate 3 exists. - **Runs once.** A post-deploy migration cannot be re-run. If the timing turns out wrong, recovery is a manual reset, so the runbook below is needed regardless. ## Testing - Migration spec asserting that only `data_type = 4` rows are reset, and that `advisories`, `licenses` and `cve_enrichment` checkpoints keep their `sequence`, `chunk` and `full_sync_target_sequence`. - Spec asserting `full_sync_target_sequence` is nulled, not just `sequence` zeroed, since a stale marker would also route to `/all` but from a resume position. ## Acceptance criteria - [ ] Post-deploy migration resets `pm_checkpoints` rows with `data_type = 4` to `sequence = 0`, `chunk = 0`, `full_sync_target_sequence = NULL`. - [ ] Migration declares `restrict_gitlab_migration gitlab_schema: :gitlab_pm` and touches no application models. - [ ] All five gates above are satisfied before the MR merges, and the MR description records which release the CVS scanner shipped in. - [ ] Migration spec covers scoping to malware rows only. - [ ] Reset validated on staging end to end before the migration merges. - [ ] PDS capacity for a fleet-wide `/all` re-download confirmed with the PDS team. - [ ] A manual reset runbook exists as a fallback, in case the migration runs at the wrong moment on any given instance. ## Technical notes Reference for the manual equivalent, used on GitLab.com during the earlier incident: https://gitlab.com/gitlab-com/gl-infra/production/-/work_items/22589 Parent epic: https://gitlab.com/groups/gitlab-org/-/epics/21156