Defer storing security reports on sec database health

Bala Kumarrequested to merge
fix/defer-store-security-reports-on-db-health into master

What does this MR do and why?

Adds database-health deferral to Security::StoreSecurityReportsByProjectWorker, mirroring the pattern already merged for Sbom::IngestReportsWorker.

  • Adds defer_on_database_health_signal :gitlab_sec, [::Vulnerabilities::Finding.table_name], 1.minute and a self.defer_on_database_health_signal? override gated on a new feature flag.
  • The worker calls Security::Ingestion::IngestReportsService, the same vulnerability ingestion path used by Security::Scans::IngestReportsWorker, so it declares the same table. Vulnerabilities::Finding.table_name resolves to vulnerability_occurrences, derived from the model rather than hardcoded since that exact name has previously been mistyped.
  • No actor is available: the middleware evaluates the predicate at job-fetch time with no project or user in scope, and deferral is an all-or-nothing throttle. Hence the inline rubocop:disable Gitlab/FeatureFlagWithoutActor.
  • New flag defer_store_security_reports_on_database_health (gitlab_com_derisk, default disabled, group::security foundations, milestone 19.5). Default-off, so no behaviour change until enabled.

Evidence

sec-db-writes-per-job-20260912

The Kibana panel "Sec DB writes per job by worker class (gprd)", over a 12-hour window ending 2026-09-12 08:30 UTC, shows Security::StoreSecurityReportsByProjectWorker peaking at 2,807 sec-database writes in a single job — the second-highest of any worker, behind Sbom::IngestReportsWorker at 16,896. Every other worker is two orders of magnitude lower: Security::StoreScansWorker at 346, Sbom::RemoveOldDependencyGraphsWorker at 152, Sbom::BuildDependencyGraphWorker at 86.

Note: the panel is titled "p99" but its legend series are prefixed (100), so these are the maximum per-job write count, not a 99th percentile — treat them as peak per-job writes.

Sbom::IngestReportsWorker has already been gated for exactly this reason, in !254955 (merged).

Behaviour when the flag is on

  • If a database health indicator returns a stop signal, Gitlab::SidekiqMiddleware::SkipJobs reschedules the job instead of running it.
  • First deferral waits 1 minute; consecutive deferrals back off exponentially to a 30 minute cap. That escalation is itself behind the pre-existing incremental_database_health_defer_delay flag.
  • Deferred jobs appear in logs as job_status: deferred with job_deferred_by: database_health_check.

Known limitation: the AutovacuumActiveOnTable indicator does not currently observe the sec database (#628693), so today the effective gating comes from cluster-wide indicators, chiefly WalRate. The table list is declared for correctness and for when that gap closes.

References

Screenshots or screen recordings

Backend-only change, no UI. The evidence image above covers the production data behind this change.

How to set up and validate locally

  1. bundle exec rspec ee/spec/workers/security/store_security_reports_by_project_worker_spec.rb — covers database_health_check_attrs and both feature flag states.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Bala Kumar

Merge request reports