Defer StoreScansWorker on sec database health

What does this MR do and why?

Security::StoreScansWorker now defers its jobs when the sec database is unhealthy. This is controlled by the defer_store_scans_on_database_health feature flag, which is off by default.

  • ee/app/workers/security/store_scans_worker.rb:21-31 calls defer_on_database_health_signal :gitlab_sec with the sec tables that Security::StoreScansService writes (security_scans, security_findings, security_finding_enrichments) and a 1 minute delay. That delay matches most other gitlab_sec workers.
  • The flag check follows Security::StoreSecurityReportsByProjectWorker, which uses defer_store_security_reports_on_database_health.
  • Context: the sec database has had recurring production WAL alerts. New StoreScansWorker jobs are already held back by the check on Security::Scans::IngestReportsWorker, which enqueues them. This check covers jobs that are already queued.
Scope changes after review
  • The urgency :high change was dropped. The worker can't meet the high-urgency execution targets: one production run in #617809 took 309s. gitlab-com/gl-infra/production#23024 (closed) found that the Sep 25 backlog came from a saturated sec PgBouncer pool on the quarantine shard, so a higher urgency would not have helped.
  • The check is no longer on the SecurityScansQueue concern. TrackSecureScansWorker would have queried the sec primary for a database it never writes to, and every job would have added a primary query on the quarantine shard.

References

Screenshots or screen recordings

This is not a UI change.

How to set up and validate locally

  1. In a Rails console, run:

    Security::StoreScansWorker.database_health_check_attrs.except(:block)
    # => {gitlab_schema: :gitlab_sec, tables: ["security_scans", "security_findings", "security_finding_enrichments"], delay_by: 1 minute}
    Security::StoreScansWorker.defer_on_database_health_signal?
    # => false (flag is off by default)
    Feature.enable(:defer_store_scans_on_database_health)
    Security::StoreScansWorker.defer_on_database_health_signal?
    # => true
  2. Run bin/rspec ee/spec/workers/security/store_scans_worker_spec.rb.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

Edited by Bala Kumar

Merge request reports

Loading
Loading