Watch more sec tables in security report ingestion health check

What does this MR do and why?

Extends the defer_on_database_health_signal table list on Security::StoreSecurityReportsByProjectWorker from one table to six:

  • Before: vulnerability_occurrences
  • After: vulnerability_reads, vulnerability_occurrences, vulnerabilities, vulnerability_identifiers, security_finding_enrichments, vulnerability_occurrence_identifiers

This comes out of the investigation into write-ahead-log (WAL) saturation alerts on the GitLab.com sec PostgreSQL database (patroni-sec), tracked in &23596. On 2026-09-15 this worker's p99 was roughly 372 writes to the sec database per job.

How deferral works. Gitlab::SidekiqMiddleware::SkipJobs evaluates Gitlab::Database::HealthStatus before perform and reschedules the job if any indicator returns a Stop signal. There are four indicators, but only AutovacuumActiveOnTable reads the table list passed to defer_on_database_health_signal. The other three (WriteAheadLog, PatroniApdex, WalRate) are cluster-wide checks and ignore the list. AutovacuumActiveOnTable uses OR semantics: it stops the job if autovacuum is running on any one of the listed tables. That is why the list is not every table the worker writes to — listing everything would make the worker defer far more often than the evidence justifies.

How the tables were chosen. Two independent signals, and here they agree cleanly: the six included tables are exactly the intersection of this worker's write path with the dead-tuple panel. That is why this list is longer than the one for Sbom::IngestReportsWorker.

  1. Index count per table, counted from db/structure.sql (CREATE INDEX statements plus primary-key and unique constraints, since each of those is backed by an index too). Every index on a row has to be maintained whenever that row is written, so index count is a direct proxy for how much WAL a table generates per row written.
  2. Observed dead tuples on patroni-sec (see the panel below). Dead tuples are what triggers autovacuum, so a table high on that panel is one autovacuum actually visits.
Index counts for the 18 SEC_DB_TASKS in ee/app/services/security/ingestion/ingest_report_slice_service.rb (16 distinct tables)
table indexes columns in dead-tuple panel? included?
vulnerability_reads 25 25 yes yes, added
vulnerability_occurrences 15 26 yes yes, already declared
vulnerabilities 13 29 yes yes, added
vulnerability_identifiers 7 10 yes yes, added
vulnerability_statistics 6 16 no no
security_finding_enrichments 6 10 yes yes, added
vulnerability_occurrence_identifiers 5 7 yes yes, added
vulnerability_finding_signatures 4 7 no no
vulnerability_flags 4 11 no no
vulnerability_namespace_statistics 4 12 no no
vulnerability_findings_remediations 4 6 no no
vulnerability_finding_links 3 7 no no
vulnerability_finding_evidences 3 6 no no
vulnerability_finding_risk_scores 3 5 no no
vulnerability_remediations 2 8 no no
vulnerability_representation_information 2 7 no no

vulnerability_statistics has a middling index count (6) but is deliberately excluded: it is written once per project per ingestion run, not once per finding, so its write churn does not scale with report size the way the included tables' does.

Note that the dead-tuple column is only meaningful for the tables named in the panel query below. The nine tables with 4 or fewer indexes were never measured, so their exclusion rests on index count alone.

Feature flag. This MR reuses the worker's existing flag defer_store_security_reports_on_database_health (gitlab_com_derisk, default off), introduced in !255196 (merged) with rollout tracked in #628841. No new flag or rollout issue is needed.

Reviewer note: this takes effect on merge. Both gates are open on gprd today. The ops flag autovacuum_indicator_uses_context_connection was set to true on 2026-09-12 (gitlab-com/gl-infra/feature-flag-log#64250). The worker flag defer_store_security_reports_on_database_health was set to true on 2026-09-15 (gitlab-com/gl-infra/feature-flag-log#64357). AutovacuumActiveOnTable therefore observes the sec database. The five added tables start to defer this worker when this merge request merges. Watch sidekiq_jobs_skipped_total{action="deferred", reason="database_health_check"} after the merge.

Out of scope, worth follow-up. No issues are filed for these yet:

  • Security::Scans::IngestReportsWorker runs through this exact same IngestReportSliceService write path and still declares only vulnerability_occurrences. This MR deliberately leaves it alone to keep the change reviewable.
  • The dead-tuple panel shows two more high-churn sec tables that neither this worker nor Sbom::IngestReportsWorker writes to: sbom_graph_paths (8 indexes, written by Sbom::BuildDependencyGraphWorker) and security_scans (11 indexes, written by Security::StoreScansWorker).

References

  • Parent epic: &23596
  • autovacuum_indicator_uses_context_connection rollout: #628693
  • defer_store_security_reports_on_database_health rollout: #628841
  • MR that introduced the flag: !255196 (merged)
  • Related write-reduction work on the SBOM ingestion path: #628686

Screenshots or screen recordings

No UI change. The dead-tuple panel that informed the table selection, from patroni-sec on 2026-09-14:

sec-dead-tuples-2026-09-14

pg_stat_user_tables_n_dead_tup{type="patroni-sec", env="gprd", relname=~"sbom_graph_paths|vulnerability_occurrences|vulnerability_occurrence_identifiers|vulnerabilities|vulnerability_reads|vulnerability_identifiers|security_finding_enrichments|sbom_occurrences|security_scans"}

Source: Grafana explore, dead tuples on patroni-sec

How to set up and validate locally

The worker's health-check table list is exposed via described_class.database_health_check_attrs. The updated spec asserts all six tables:

bundle exec rspec ee/spec/workers/security/store_security_reports_by_project_worker_spec.rb

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Gregory Havenga

Merge request reports

Loading
Loading