## Summary Roll out the database-health deferral gating for `Sbom::IngestReportsWorker`, currently behind the `defer_sbom_ingest_reports_on_database_health` feature flag. There is no separate feature issue for this change; the context is described below. `Sbom::IngestReportsWorker` (`ee/app/workers/sbom/ingest_reports_worker.rb`) ingests CycloneDX SBOM reports from CI pipelines and writes the results to the `sec` database inline — `sbom_occurrences`, `sbom_components`, `sbom_component_versions`, `sbom_sources`, `sbom_source_packages`, `sbom_occurrences_vulnerabilities`. It declares no `concurrency_limit`, and its only serialization is a per-project exclusive lease, so many projects ingest in parallel. Until now it had no database-health gating, unlike its siblings (`Sbom::BuildDependencyGraphWorker`, `Security::Scans::IngestReportsWorker`). The change adds `defer_on_database_health_signal :gitlab_sec, [:sbom_occurrences], 1.minute`, and this flag controls whether that gating is active. When the flag is enabled and a database health indicator returns a stop signal, `Gitlab::SidekiqMiddleware::SkipJobs` reschedules the job instead of running it. The first deferral waits 1 minute; consecutive deferrals back off exponentially to a 30 minute cap (that escalation is itself behind the separate, pre-existing `incremental_database_health_defer_delay` flag). The health indicators evaluated are `AutovacuumActiveOnTable`, `WriteAheadLog`, `PatroniApdex` and `WalRate`. In practice the `WalRate` indicator is the one expected to trip: its configured SLO for the sec database is 70 MB/s of WAL generation. Note that `AutovacuumActiveOnTable` is currently ineffective on this code path: it queries through a `SharedModel` that `SkipJobs` never binds to the sec connection, so it evaluates against the main database regardless of the declared schema. The `sbom_occurrences` table list is declared for correctness and for when that gap is closed. - DRI: @bala.kumar - Group: `group::security foundations` - Milestone: 19.4 - Flag type: `gitlab_com_derisk`, default state `default_enabled: false` - Flag definition: `ee/config/feature_flags/gitlab_com_derisk/defer_sbom_ingest_reports_on_database_health.yml` - Introducing merge request: To be added - This is not an ops flag, not an experiment, and has no A/B measurement — a standard incremental actor rollout applies. > [!note] > Process and guidance live in the docs — this issue is just the commands and a place to track the rollout. > "Rolling out" means incrementally enabling the flag on GitLab.com to validate stability — it is not the same as releasing the feature, which happens when the flag is removed. > [Feature flag controls](https://docs.gitlab.com/development/feature_flags/controls/) · [Feature flag lifecycle](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) ## Why this flag exists Write-ahead-log (WAL) generation pressure on GitLab.com's `sec` PostgreSQL database was investigated in a Slack thread. Log analysis of Sidekiq worker write activity pointed to `Sbom::IngestReportsWorker` as a significant contributor. This worker writes to the sec database inline and has no concurrency limit on those writes. This flag gates a new `defer_on_database_health_signal` check on the worker, so the deferral can be rolled out gradually and its effect measured against the same views used in the investigation. All from the production investigation, window `2026-09-10 22:00` to `2026-09-11 00:00` UTC. **Sec DB writes per job, p99 by worker class.** `Sbom::IngestReportsWorker` reaches 38,896 writes in a single job at 22:09. The next-highest worker in the same bucket is `Security::StoreSecurityReportsByProjectWorker` at 587.  [Open in Kibana](https://log.gprd.gitlab.net/app/r/s/fKE7c) **Total sec DB write count, `Sbom::IngestReportsWorker`.** 97,352 writes in the 22:09 minute bucket, against a baseline of roughly 5,000 to 25,000 per minute.  [Open in Kibana](https://log.gprd.gitlab.net/app/r/s/zeaJX) **`patroni-sec` WAL generation saturation.** Peaks at 65.9% at 22:40, against a 70% soft SLO and a 90% hard SLO. The 99th percentile for the week is 42.9%.  [Open in Grafana](https://dashboards.gitlab.net/goto/cfxwmpoyla0w0f?orgId=1) ## What could go wrong? - The user-visible effect of deferral is that dependency lists and SBOM-derived data lag behind the pipeline that produced them, for as long as the sec database stays unhealthy. - Because deferrals back off exponentially, a sustained unhealthy period could push individual jobs out by up to 30 minutes. - The worker has no concurrency limit, so a large backlog can accumulate while deferred and then release together once health recovers. Watch for a release burst. - While a job sits deferred, its `until_executed` idempotency key is released and the re-enqueued copy is excluded from deduplication, so a repeat event for the same pipeline can enqueue a second job. Dashboards and queries to watch during rollout: - Prometheus counter `sidekiq_jobs_skipped_total{action="deferred", reason="database_health_check"}` filtered to `Sbom::IngestReportsWorker`. - Kibana: Sidekiq logs where `json.class` is `Sbom::IngestReportsWorker` and `json.job_deferred_by` is `database_health_check`; `json.deferred_count` shows backlog depth. - Sec database WAL generation rate, against the runbook https://runbooks.gitlab.com/patroni/primary_db_node_wal_generation_saturation/ - SBOM ingestion queue latency and backlog for the `sbom_reports` queue namespace. ## Rollout Run all production `/chatops` in [`#production`](https://gitlab.slack.com/archives/C101F3796) and cross-post the results to the group::security foundations team channel. Background: [incremental rollout process](https://docs.gitlab.com/development/feature_flags/controls/#process), [feature actors](https://docs.gitlab.com/development/feature_flags/#feature-actors). **Non-production** ``` /chatops gitlab run feature set defer_sbom_ingest_reports_on_database_health true --dev --pre --staging --staging-ref ``` **Production** This flag takes no actor, so it is all-or-nothing. Percentage-of-actors and per-project targeting do not apply. Enable it only after a non-production soak, and watch the views above for a deferral burst. ``` /chatops gitlab run feature set defer_sbom_ingest_reports_on_database_health true ``` ## Before global rollout Confirm the relevant gotchas before going to 100% — see [enabling a feature for GitLab.com](https://docs.gitlab.com/development/feature_flags/controls/#enabling-a-feature-for-gitlabcom): - [ ] [Docs and version history](https://docs.gitlab.com/development/documentation/feature_flags/) updated - [ ] [Breaking changes](https://docs.gitlab.com/development/documentation/release_notes/#deprecations-removals-and-breaking-changes) announced, if any — none expected, this is an internal reliability change with no API-visible contract change - [ ] [Change management issue](https://handbook.gitlab.com/handbook/engineering/infrastructure-platforms/change-management/#feature-flags-and-the-change-management-process) opened, if required - [ ] [External API consumers](https://docs.gitlab.com/development/feature_flags/#do-not-use-feature-flags-in-external-api-consumers) handled with a fail-open mechanism, if applicable — not applicable, this flag only gates internal Sidekiq scheduling behaviour ## Cleanup Remove the flag once [deemed stable](https://handbook.gitlab.com/handbook/product-development/how-we-work/product-development-flow/feature-flag-lifecycle/#feature-flag-lifecycle) — see [cleaning up](https://docs.gitlab.com/development/feature_flags/controls/#cleaning-up). Remove the flag and its YAML definition from the codebase, then: ``` /chatops gitlab run feature delete defer_sbom_ingest_reports_on_database_health --dev --pre --staging --staging-ref --production ``` ## Rollback Disable the flag. The gating disappears immediately and the worker returns to running unconditionally. No data migration or cleanup is required, and no state is left behind. ``` /chatops gitlab run feature set defer_sbom_ingest_reports_on_database_health false /chatops gitlab run feature set defer_sbom_ingest_reports_on_database_health false --dev --pre --staging --staging-ref ``` 🤖 Generated with [Claude Code](https://claude.com/claude-code)