Watch more sec tables in security report ingestion health check
What does this MR do and why?
Extends the defer_on_database_health_signal table list on Security::StoreSecurityReportsByProjectWorker from one table to six:
- Before:
vulnerability_occurrences - After:
vulnerability_reads,vulnerability_occurrences,vulnerabilities,vulnerability_identifiers,security_finding_enrichments,vulnerability_occurrence_identifiers
This comes out of the investigation into write-ahead-log (WAL) saturation alerts on the GitLab.com sec PostgreSQL database (patroni-sec), tracked in &23596. On 2026-09-15 this worker's p99 was roughly 372 writes to the sec database per job.
How deferral works. Gitlab::SidekiqMiddleware::SkipJobs evaluates Gitlab::Database::HealthStatus before perform and reschedules the job if any indicator returns a Stop signal. There are four indicators, but only AutovacuumActiveOnTable reads the table list passed to defer_on_database_health_signal. The other three (WriteAheadLog, PatroniApdex, WalRate) are cluster-wide checks and ignore the list. AutovacuumActiveOnTable uses OR semantics: it stops the job if autovacuum is running on any one of the listed tables. That is why the list is not every table the worker writes to — listing everything would make the worker defer far more often than the evidence justifies.
How the tables were chosen. Two independent signals, and here they agree cleanly: the six included tables are exactly the intersection of this worker's write path with the dead-tuple panel. That is why this list is longer than the one for Sbom::IngestReportsWorker.
- Index count per table, counted from
db/structure.sql(CREATE INDEXstatements plus primary-key and unique constraints, since each of those is backed by an index too). Every index on a row has to be maintained whenever that row is written, so index count is a direct proxy for how much WAL a table generates per row written. - Observed dead tuples on
patroni-sec(see the panel below). Dead tuples are what triggers autovacuum, so a table high on that panel is one autovacuum actually visits.
Index counts for the 18 SEC_DB_TASKS in ee/app/services/security/ingestion/ingest_report_slice_service.rb (16 distinct tables)
| table | indexes | columns | in dead-tuple panel? | included? |
|---|---|---|---|---|
vulnerability_reads |
25 | 25 | yes | yes, added |
vulnerability_occurrences |
15 | 26 | yes | yes, already declared |
vulnerabilities |
13 | 29 | yes | yes, added |
vulnerability_identifiers |
7 | 10 | yes | yes, added |
vulnerability_statistics |
6 | 16 | no | no |
security_finding_enrichments |
6 | 10 | yes | yes, added |
vulnerability_occurrence_identifiers |
5 | 7 | yes | yes, added |
vulnerability_finding_signatures |
4 | 7 | no | no |
vulnerability_flags |
4 | 11 | no | no |
vulnerability_namespace_statistics |
4 | 12 | no | no |
vulnerability_findings_remediations |
4 | 6 | no | no |
vulnerability_finding_links |
3 | 7 | no | no |
vulnerability_finding_evidences |
3 | 6 | no | no |
vulnerability_finding_risk_scores |
3 | 5 | no | no |
vulnerability_remediations |
2 | 8 | no | no |
vulnerability_representation_information |
2 | 7 | no | no |
vulnerability_statistics has a middling index count (6) but is deliberately excluded: it is written once per project per ingestion run, not once per finding, so its write churn does not scale with report size the way the included tables' does.
Note that the dead-tuple column is only meaningful for the tables named in the panel query below. The nine tables with 4 or fewer indexes were never measured, so their exclusion rests on index count alone.
Feature flag. This MR reuses the worker's existing flag defer_store_security_reports_on_database_health (gitlab_com_derisk, default off), introduced in !255196 (merged) with rollout tracked in #628841. No new flag or rollout issue is needed.
Reviewer note: this takes effect on merge. Both gates are open on gprd today. The ops flag autovacuum_indicator_uses_context_connection was set to true on 2026-09-12 (gitlab-com/gl-infra/feature-flag-log#64250). The worker flag defer_store_security_reports_on_database_health was set to true on 2026-09-15 (gitlab-com/gl-infra/feature-flag-log#64357). AutovacuumActiveOnTable therefore observes the sec database. The five added tables start to defer this worker when this merge request merges. Watch sidekiq_jobs_skipped_total{action="deferred", reason="database_health_check"} after the merge.
Out of scope, worth follow-up. No issues are filed for these yet:
Security::Scans::IngestReportsWorkerruns through this exact sameIngestReportSliceServicewrite path and still declares onlyvulnerability_occurrences. This MR deliberately leaves it alone to keep the change reviewable.- The dead-tuple panel shows two more high-churn
sectables that neither this worker norSbom::IngestReportsWorkerwrites to:sbom_graph_paths(8 indexes, written bySbom::BuildDependencyGraphWorker) andsecurity_scans(11 indexes, written bySecurity::StoreScansWorker).
References
- Parent epic: &23596
autovacuum_indicator_uses_context_connectionrollout: #628693defer_store_security_reports_on_database_healthrollout: #628841- MR that introduced the flag: !255196 (merged)
- Related write-reduction work on the SBOM ingestion path: #628686
Screenshots or screen recordings
No UI change. The dead-tuple panel that informed the table selection, from patroni-sec on 2026-09-14:
pg_stat_user_tables_n_dead_tup{type="patroni-sec", env="gprd", relname=~"sbom_graph_paths|vulnerability_occurrences|vulnerability_occurrence_identifiers|vulnerabilities|vulnerability_reads|vulnerability_identifiers|security_finding_enrichments|sbom_occurrences|security_scans"}Source: Grafana explore, dead tuples on patroni-sec
How to set up and validate locally
The worker's health-check table list is exposed via described_class.database_health_check_attrs. The updated spec asserts all six tables:
bundle exec rspec ee/spec/workers/security/store_security_reports_by_project_worker_spec.rbMR acceptance checklist
Evaluate this MR against the MR acceptance checklist.
