Watch more sec tables in SBOM ingestion health check

What does this MR do and why?

Extends the defer_on_database_health_signal table list on Sbom::IngestReportsWorker from one table to three:

  • Before: sbom_occurrences
  • After: sbom_occurrences, sbom_occurrence_refs, sbom_occurrences_vulnerabilities

This comes out of the investigation into write-ahead-log (WAL) saturation alerts on the GitLab.com sec PostgreSQL database (patroni-sec), tracked in &23596. Sbom::IngestReportsWorker is the heaviest writer to the sec database of any worker: on 2026-09-15 its p99 was roughly 15,800 writes to the sec database per job.

How deferral works. Gitlab::SidekiqMiddleware::SkipJobs evaluates Gitlab::Database::HealthStatus before perform and reschedules the job if any indicator returns a Stop signal. There are four indicators, but only AutovacuumActiveOnTable reads the table list passed to defer_on_database_health_signal. The other three (WriteAheadLog, PatroniApdex, WalRate) are cluster-wide checks and ignore the list. AutovacuumActiveOnTable uses OR semantics: it stops the job if autovacuum is running on any one of the listed tables. That is why the list is not every table the worker writes to — listing everything would make the worker defer far more often than the evidence justifies.

How the tables were chosen. Two independent signals:

  1. Index count per table, counted from db/structure.sql (CREATE INDEX statements plus primary-key and unique constraints, since each of those is backed by an index too). Every index on a row has to be maintained whenever that row is written, so index count is a direct proxy for how much WAL a table generates per row written.
  2. Observed dead tuples on patroni-sec, from a Grafana/Mimir query run on 2026-09-14 (36 series, top series peaking around 4 million dead tuples, with a sawtooth pattern where drops mark autovacuum completions). Dead tuples are what triggers autovacuum, so a table high on that panel is one autovacuum actually visits.
Index counts for the 8 ingestion tasks in ee/app/services/sbom/ingestion/ingest_report_slice_service.rb
table indexes columns included?
sbom_occurrences 25 22 yes, already declared
sbom_occurrence_refs 6 6 yes, added by this MR
sbom_occurrences_vulnerabilities 6 8 yes, added by this MR
sbom_sources 4 6 no
sbom_components 3 7 no
sbom_component_versions 3 7 no
sbom_source_packages 3 6 no

The four excluded tables are lookup/dedup tables with only 3-4 indexes each. The ingestion tasks only write rows to them that are genuinely new (deduped against existing rows), so their write churn is low.

A caveat on the two added tables. sbom_occurrence_refs and sbom_occurrences_vulnerabilities do not appear in the dead-tuple panel, so their inclusion rests on index count and row fan-out alone: each holds roughly one row per occurrence, so their row counts scale with the same volume that drives sbom_occurrences. A reviewer who prefers the strictly minimal change can drop these two and keep only sbom_occurrences.

Feature flag. This MR reuses the worker's existing flag defer_sbom_ingest_reports_on_database_health (gitlab_com_derisk, default off), introduced in !254955 (merged) with rollout tracked in #628675. No new flag or rollout issue is needed.

Reviewer note: this takes effect on merge. Both gates are open on gprd today. The ops flag autovacuum_indicator_uses_context_connection was set to true on 2026-09-12 (gitlab-com/gl-infra/feature-flag-log#64250). The worker flag defer_sbom_ingest_reports_on_database_health was set to true on 2026-09-14 (gitlab-com/gl-infra/feature-flag-log#64272). AutovacuumActiveOnTable therefore observes the sec database. The two added tables start to defer this worker when this merge request merges. Watch sidekiq_jobs_skipped_total{action="deferred", reason="database_health_check"} after the merge.

References

  • Parent epic: &23596
  • autovacuum_indicator_uses_context_connection rollout: #628693
  • defer_sbom_ingest_reports_on_database_health rollout: #628675
  • MR that introduced the flag: !254955 (merged)
  • Related write-reduction work on this worker: #628686

Screenshots or screen recordings

No UI change. Dead tuples on patroni-sec on 2026-09-14, the panel that frames this work. sbom_occurrences is among the top series; the sawtooth drops are autovacuum completions, which is what this worker defers on:

sec-dead-tuples-2026-09-14

pg_stat_user_tables_n_dead_tup{type="patroni-sec", env="gprd", relname=~"sbom_graph_paths|vulnerability_occurrences|vulnerability_occurrence_identifiers|vulnerabilities|vulnerability_reads|vulnerability_identifiers|security_finding_enrichments|sbom_occurrences|security_scans"}

Source: Grafana explore, dead tuples on patroni-sec

Note the relname filter: sbom_occurrence_refs and sbom_occurrences_vulnerabilities were never in this query, which is why the caveat above rests their case on index count and row fan-out rather than on this panel.

How to set up and validate locally

The worker's health-check table list is exposed via described_class.database_health_check_attrs (or Sbom::IngestReportsWorker.database_health_check_attrs in a Rails console). The updated spec asserts all three tables:

bundle exec rspec ee/spec/workers/sbom/ingest_reports_worker_spec.rb

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist.

Edited by Gregory Havenga

Merge request reports

Loading
Loading