Bind the autovacuum health indicator to its context connection

What does this MR do and why?

Fixes Gitlab::Database::HealthStatus::Indicators::AutovacuumActiveOnTable always querying the main database regardless of the caller's declared gitlab_schema, making it inert for every worker on a non-main schema.

Changes, all in lib/gitlab/database/health_status/indicators/autovacuum_active_on_table.rb:

  • initialize now captures context.connection alongside context.tables. It previously discarded the connection.
  • active_autovacuums_for wraps the PostgresAutovacuumActivity.for_tables call in Gitlab::Database::SharedModel.using_connection(connection).
  • Gated by a new ops flag autovacuum_indicator_uses_context_connection, default disabled, group::database, milestone 19.4, defined at config/feature_flags/ops/autovacuum_indicator_uses_context_connection.yml.

Why it was broken:

  • PostgresAutovacuumActivity is a Gitlab::Database::SharedModel, so the database it queries depends on an ambient SharedModel.using_connection wrapper.
  • With no wrapper it falls back to ActiveRecord::Base, i.e. main, and its where('schema = current_schema()') predicate can then only see main.
  • Of four callers, only the batched background migration worker wraps the evaluation (app/workers/database/batched_background_migration/execution_worker.rb:52). Gitlab::SidekiqMiddleware::SkipJobs, Gitlab::Database::BackgroundOperation::Runner, and Import::ReassignPlaceholderThrottling do not.
  • Impact: no autovacuum protection for the 37 workers declaring :gitlab_sec and the 9 declaring :gitlab_ci. The 141 :gitlab_main and 15 :gitlab_main_org declarations worked by coincidence, since gitlab_main_org also lives on main.

Why fix it in the indicator rather than each caller:

  • Fixes all four call sites plus any future caller.
  • The Context already carries the connection; the indicator simply wasn't reading it.
  • Nests safely with the existing batched-migration wrapper: using_connection raises only when a different connection is already active, and that path passes the same connection object through.

Risk (why an ops flag): this activates a previously inert stop signal across 46 workers at once. Autovacuum on large sec and ci tables can run for long stretches, so enabling it could cause a jump in deferred jobs. Watch sidekiq_jobs_skipped_total{action="deferred", reason="database_health_check"} during rollout. The flag is ops type to match batched_migrations_health_status_autovacuum, the existing flag on this indicator, and because it is an operational switch for a throttle. Ops flags require documentation in the "All feature flags in GitLab" list and an operational runbook; that documentation is outstanding.

References

Screenshots or screen recordings

Not applicable — backend-only change, no UI impact.

How to set up and validate locally

  1. Run bundle exec rspec spec/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table_spec.rb.
  2. The spec gains a connection binding group: one example asserts SharedModel.using_connection receives the connection from the context, another asserts it is not called when the flag is off. 8 examples, 0 failures.
  3. Note: the existing spec already wrapped every example in SharedModel.using_connection via an around block — it was manually compensating for what production never did.
Local testing observations: which database the query actually hits

The defect is about which connection the autovacuum query runs on, so that is what this reproduces. Run against a GDK with main, ci and sec all configured.

Save as repro.rb and run bundle exec rails runner repro.rb:

# Observes which database the autovacuum indicator's query actually runs against.
FLAG = :autovacuum_indicator_uses_context_connection
INDICATOR = Gitlab::Database::HealthStatus::Indicators::AutovacuumActiveOnTable

Feature.enable(:batched_migrations_health_status_autovacuum)

checker = Struct.new(:id, :job_class_name).new('repro', 'ReproWorker')
sec_conn = Gitlab::Database.database_base_models[:sec].connection
tables   = ['vulnerability_occurrences']
context  = Gitlab::Database::HealthStatus::Context.new(checker, sec_conn, tables)

puts "context connection : #{Gitlab::Database.db_config_name(sec_conn)}"
puts "tables             : #{tables.inspect}"
puts

def observe
  seen = []
  sub = ActiveSupport::Notifications.subscribe('sql.active_record') do |_, _, _, _, payload|
    next unless payload[:sql].to_s.include?('postgres_autovacuum_activity')

    seen << Gitlab::Database.db_config_name(payload[:connection])
  end
  signal = yield
  [seen.uniq, signal]
ensure
  ActiveSupport::Notifications.unsubscribe(sub)
end

[[false, 'BEFORE (flag off = current master behaviour)'],
 [true,  'AFTER  (flag on  = this MR)']].each do |enabled, label|
  enabled ? Feature.enable(FLAG) : Feature.disable(FLAG)
  dbs, signal = observe { INDICATOR.new(context).evaluate }
  puts label
  puts "  query ran against : #{dbs.empty? ? '(no query observed)' : dbs.join(', ')}"
  puts "  signal            : #{signal.class.name.demodulize}#{signal.reason}"
  puts
end

Feature.disable(FLAG)
puts "flag restored to disabled"

Output on a GDK at this MR's HEAD:

context connection : sec
tables             : ["vulnerability_occurrences"]

BEFORE (flag off = current master behaviour)
  query ran against : main
  signal            : Normal - no autovacuum running on any relevant tables

AFTER  (flag on  = this MR)
  query ran against : sec
  signal            : Normal - no autovacuum running on any relevant tables

flag restored to disabled

Before and after, the query targets main and sec respectively, for the same Context carrying the sec connection. That is the whole defect and the whole fix.

Two honest limits on this observation:

  • Both runs return Normal because no autovacuum was running locally. The signal is not the interesting part here; the connection is.
  • A full autovacuum-triggered Stop is not reproducible outside specs. postgres_autovacuum_activity is a view over pg_stat_activity, so activity cannot be inserted; the spec suite only manages this through the swapout_view_for_table helper. The Stop path is covered by the existing examples in spec/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table_spec.rb.

MR acceptance checklist

Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.

🤖 Generated with Claude Code

Edited by Bala Kumar

Merge request reports

Loading
Loading