Gate SBOM ingestion on sec database health, fix table typo

Bala Kumarrequested to merge
fix/sec-db-health-deferral-ingest-workers into master

What does this MR do and why?

Commit 1: fix misspelled table name

  • ee/app/workers/security/scans/ingest_reports_worker.rb: vulnerability_occurences → vulnerability_occurrences.
  • Only AutovacuumActiveOnTable reads that table list, so the typo meant it always checked a table that doesn't exist.
  • Spec coverage added for the corrected name.

Commit 2: gate Sbom::IngestReportsWorker on database health

  • Adds defer_on_database_health_signal :gitlab_sec, [:sbom_occurrences], 1.minute, behind flag defer_sbom_ingest_reports_on_database_health (type gitlab_com_derisk, default disabled).
  • Default-off: no behaviour change until enabled.
  • Worker writes to the sec database inline with no concurrency_limit; sibling workers are already gated.
  • Production log analysis into sec-database WAL generation pressure identified Sbom::IngestReportsWorker as a significant contributor; see References for supporting evidence.
  • Spec coverage added.
  • Low risk: no migrations, no user-facing change.

References

Screenshots or screen recordings

All from the production investigation, window 2026-09-10 22:00 to 2026-09-11 00:00 UTC.

Sec DB writes per job, p99 by worker class. Sbom::IngestReportsWorker reaches 38,896 writes in a single job at 22:09. The next-highest worker in the same bucket is Security::StoreSecurityReportsByProjectWorker at 587.

shot1

Open in Kibana

Total sec DB write count, Sbom::IngestReportsWorker. 97,352 writes in the 22:09 minute bucket, against a baseline of roughly 5,000 to 25,000 per minute.

shot2

Open in Kibana

patroni-sec WAL generation saturation. Peaks at 65.9% at 22:40, against a 70% soft SLO and a 90% hard SLO. The 99th percentile for the week is 42.9%.

shot3

Open in Grafana

How to set up and validate locally

  1. Enable the flag: 
    Feature.enable(:defer_sbom_ingest_reports_on_database_health)
  2. Run the specs: 
    bundle exec rspec ee/spec/workers/sbom/ingest_reports_worker_spec.rb ee/spec/workers/security/scans/ingest_reports_worker_spec.rb
Notes for reviewers: pre-existing quirks in the deferral mechanism

Autovacuum indicator doesn't see the sec connection. AutovacuumActiveOnTable is the only indicator that reads the tables: list; it queries via PostgresAutovacuumActivity, a SharedModel. SkipJobs never wraps it in SharedModel.using_connection, unlike app/workers/database/batched_background_migration/execution_worker.rb:52, so it evaluates against main for every :gitlab_sec worker. The table-name fix here doesn't activate anything today; SBOM gating relies on the cluster-wide WalRate indicator instead. A fix belongs in SkipJobs, out of scope here.

Deduplication during deferral. SkipJobs runs after DuplicateJobs::Server, so deferring releases the until_executed key, and re-enqueued deferred jobs skip deduplication. A repeat event for the same pipeline_id can enqueue a second job. Bounded in practice.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Bala Kumar

Merge request reports