Recompute inventory stats when vulnerabilities go undetected
What does this MR do and why?
Recompute inventory stats when vulnerabilities go undetected
Security Inventory severity counts are backed by the vulnerability_statistics table. When a pipeline scan stops reporting a previously detected vulnerability, MarkAsResolvedService flips resolved_on_default_branch to true with a bulk update_all. That bulk write bypasses the incremental statistics path, which relies on dirty tracking of a loaded record, so the inventory count stayed stale until the twice-daily AdjustmentWorker recompute corrected it.
Enqueue a per-project AdjustmentWorker run from MarkAsResolvedService when it marks vulnerabilities as no longer detected, so the counts are corrected right away. This reuses the same recompute path the archival, removal, and auto-resolve flows already use for their bulk writes. The enqueue fires once per execute and only when a real transition happened, and it is gated by the security_inventory_no_longer_detected_vulnerabilities flag so the live and batch paths filter consistently.
Re-detection resolved_on_default_branch back to false during ingestion) stays on the cron cadence for now, since those writes run on every pipeline and have no cheap signal for a genuine flip.
Risks and trade-offs
- Recompute cost: one extra per-project recompute each time a pipeline marks something no longer detected ( once per pipeline ). Bounded by project size and by the
no_longer_detected.present?guard. Comparable to the existing archival, removal, and auto-resolve enqueues. - Possible duplicate enqueue with auto-resolve in the same run. Harmless; the worker is safe to run twice.
AdjustmentWorkeris intentionally not idempotent-deduplicated, so we cannot rely on Sidekiq dedup to collapse them; the once-per-execute flag keeps our own contribution to a single enqueue. - Re-detection (
resolved_on_default_branchchanges from true to false ) stays on the cron cadence.
Rejected alternative
The incremental approach ( calculating and propagating diff instead of the SQL based recompute ) was rejected because the bulk writes that flip the flag bypass the very mechanism it depends on, and replicating that mechanism by hand is strictly more complex and riskier than reusing the existing recompute.
References
How to set up and validate locally
It is difficult to come up with a plausible testing scenario that doesn't rely to heavy on console operations.
Steps
Console-driven test against a seeded project. It exercises the exact new code (MarkAsResolvedService -> enqueue -> AdjustmentWorker recompute), is deterministic, and avoids CI/scanner flakiness.
Prerequisites
# in rails console
Feature.enable(:security_inventory_no_longer_detected_vulnerabilities) # global; works for :instance or namespaceMake sure background jobs run so the enqueued recompute executes: gdk start rails-background-jobs.
Step 1 - pick a currently-counted vuln and snapshot the count
project = Project.find_by_full_path('superorg/depb/redteam/megascript') # any seeded project with criticals
def counts(project)
project.reload.vulnerability_statistic
&.slice('total', 'critical', 'high', 'medium', 'low', 'unknown', 'info', 'letter_grade')
end
target = project.vulnerability_reads
.where(state: Vulnerability.active_states, resolved_on_default_branch: false, severity: :critical)
.where.not(report_type: :generic) # generic = manual, the service skips these
.first
before = counts(project)Step 2 - run the real service the way ingestion does
scanner = target.scanner
tracked_context = target.tracked_context
pipeline = project.ci_pipelines.last
# everything this scanner still reports EXCEPT our target => only target is "missing"
ingested = project.vulnerability_reads.by_scanner(scanner).pluck(:vulnerability_id) - [target.vulnerability_id]
Security::Ingestion::MarkAsResolvedService.execute(pipeline, scanner, ingested, tracked_context)
sleep 3 # let the enqueued AdjustmentWorker runStep 3 - assert the live correction
[before, counts(project)]Pass: critical is exactly one lower than before, and you did not run AdjustmentWorker yourself. That is the live path: the service flipped the flag and enqueued the recompute.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.