Loading
Defer StoreScansWorker on sec database health
What does this MR do and why?
Security::StoreScansWorker now defers its jobs when the sec database is unhealthy. This is controlled by the defer_store_scans_on_database_health feature flag, which is off by default.
ee/app/workers/security/store_scans_worker.rb:21-31callsdefer_on_database_health_signal :gitlab_secwith the sec tables thatSecurity::StoreScansServicewrites (security_scans,security_findings,security_finding_enrichments) and a 1 minute delay. That delay matches most othergitlab_secworkers.- The flag check follows
Security::StoreSecurityReportsByProjectWorker, which usesdefer_store_security_reports_on_database_health. - Context: the sec database has had recurring production WAL alerts. New
StoreScansWorkerjobs are already held back by the check onSecurity::Scans::IngestReportsWorker, which enqueues them. This check covers jobs that are already queued.
Scope changes after review
- The
urgency :highchange was dropped. The worker can't meet the high-urgency execution targets: one production run in #617809 took 309s. gitlab-com/gl-infra/production#23024 (closed) found that the Sep 25 backlog came from a saturated sec PgBouncer pool on the quarantine shard, so a higher urgency would not have helped. - The check is no longer on the
SecurityScansQueueconcern.TrackSecureScansWorkerwould have queried the sec primary for a database it never writes to, and every job would have added a primary query on the quarantine shard.
References
- Sidekiq database health checks: https://docs.gitlab.com/development/sidekiq/#deferring-sidekiq-workers
- Incident findings: gitlab-com/gl-infra/production#23024 (closed)
Screenshots or screen recordings
This is not a UI change.
How to set up and validate locally
-
In a Rails console, run:
Security::StoreScansWorker.database_health_check_attrs.except(:block) # => {gitlab_schema: :gitlab_sec, tables: ["security_scans", "security_findings", "security_finding_enrichments"], delay_by: 1 minute} Security::StoreScansWorker.defer_on_database_health_signal? # => false (flag is off by default) Feature.enable(:defer_store_scans_on_database_health) Security::StoreScansWorker.defer_on_database_health_signal? # => true -
Run
bin/rspec ee/spec/workers/security/store_scans_worker_spec.rb.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.
Edited by Bala Kumar