Bind the autovacuum health indicator to its context connection
What does this MR do and why?
Fixes Gitlab::Database::HealthStatus::Indicators::AutovacuumActiveOnTable always querying the main database regardless of the caller's declared gitlab_schema, making it inert for every worker on a non-main schema.
Changes, all in lib/gitlab/database/health_status/indicators/autovacuum_active_on_table.rb:
initializenow capturescontext.connectionalongsidecontext.tables. It previously discarded the connection.active_autovacuums_forwraps thePostgresAutovacuumActivity.for_tablescall inGitlab::Database::SharedModel.using_connection(connection).- Gated by a new ops flag
autovacuum_indicator_uses_context_connection, default disabled,group::database, milestone 19.4, defined atconfig/feature_flags/ops/autovacuum_indicator_uses_context_connection.yml.
Why it was broken:
PostgresAutovacuumActivityis aGitlab::Database::SharedModel, so the database it queries depends on an ambientSharedModel.using_connectionwrapper.- With no wrapper it falls back to
ActiveRecord::Base, i.e. main, and itswhere('schema = current_schema()')predicate can then only see main. - Of four callers, only the batched background migration worker wraps the evaluation (
app/workers/database/batched_background_migration/execution_worker.rb:52).Gitlab::SidekiqMiddleware::SkipJobs,Gitlab::Database::BackgroundOperation::Runner, andImport::ReassignPlaceholderThrottlingdo not. - Impact: no autovacuum protection for the 37 workers declaring
:gitlab_secand the 9 declaring:gitlab_ci. The 141:gitlab_mainand 15:gitlab_main_orgdeclarations worked by coincidence, sincegitlab_main_orgalso lives on main.
Why fix it in the indicator rather than each caller:
- Fixes all four call sites plus any future caller.
- The
Contextalready carries the connection; the indicator simply wasn't reading it. - Nests safely with the existing batched-migration wrapper:
using_connectionraises only when a different connection is already active, and that path passes the same connection object through.
Risk (why an ops flag): this activates a previously inert stop signal across 46 workers at once. Autovacuum on large sec and ci tables can run for long stretches, so enabling it could cause a jump in deferred jobs. Watch sidekiq_jobs_skipped_total{action="deferred", reason="database_health_check"} during rollout. The flag is ops type to match batched_migrations_health_status_autovacuum, the existing flag on this indicator, and because it is an operational switch for a throttle. Ops flags require documentation in the "All feature flags in GitLab" list and an operational runbook; that documentation is outstanding.
References
- Issue: #628693
- Discovered in: !254955 (merged)
Screenshots or screen recordings
Not applicable — backend-only change, no UI impact.
How to set up and validate locally
- Run
bundle exec rspec spec/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table_spec.rb. - The spec gains a
connection bindinggroup: one example assertsSharedModel.using_connectionreceives the connection from the context, another asserts it is not called when the flag is off. 8 examples, 0 failures. - Note: the existing spec already wrapped every example in
SharedModel.using_connectionvia anaroundblock — it was manually compensating for what production never did.
Local testing observations: which database the query actually hits
The defect is about which connection the autovacuum query runs on, so that is what this reproduces. Run against a GDK with main, ci and sec all configured.
Save as repro.rb and run bundle exec rails runner repro.rb:
# Observes which database the autovacuum indicator's query actually runs against.
FLAG = :autovacuum_indicator_uses_context_connection
INDICATOR = Gitlab::Database::HealthStatus::Indicators::AutovacuumActiveOnTable
Feature.enable(:batched_migrations_health_status_autovacuum)
checker = Struct.new(:id, :job_class_name).new('repro', 'ReproWorker')
sec_conn = Gitlab::Database.database_base_models[:sec].connection
tables = ['vulnerability_occurrences']
context = Gitlab::Database::HealthStatus::Context.new(checker, sec_conn, tables)
puts "context connection : #{Gitlab::Database.db_config_name(sec_conn)}"
puts "tables : #{tables.inspect}"
puts
def observe
seen = []
sub = ActiveSupport::Notifications.subscribe('sql.active_record') do |_, _, _, _, payload|
next unless payload[:sql].to_s.include?('postgres_autovacuum_activity')
seen << Gitlab::Database.db_config_name(payload[:connection])
end
signal = yield
[seen.uniq, signal]
ensure
ActiveSupport::Notifications.unsubscribe(sub)
end
[[false, 'BEFORE (flag off = current master behaviour)'],
[true, 'AFTER (flag on = this MR)']].each do |enabled, label|
enabled ? Feature.enable(FLAG) : Feature.disable(FLAG)
dbs, signal = observe { INDICATOR.new(context).evaluate }
puts label
puts " query ran against : #{dbs.empty? ? '(no query observed)' : dbs.join(', ')}"
puts " signal : #{signal.class.name.demodulize} — #{signal.reason}"
puts
end
Feature.disable(FLAG)
puts "flag restored to disabled"Output on a GDK at this MR's HEAD:
context connection : sec
tables : ["vulnerability_occurrences"]
BEFORE (flag off = current master behaviour)
query ran against : main
signal : Normal - no autovacuum running on any relevant tables
AFTER (flag on = this MR)
query ran against : sec
signal : Normal - no autovacuum running on any relevant tables
flag restored to disabledBefore and after, the query targets main and sec respectively, for the same Context carrying the sec connection. That is the whole defect and the whole fix.
Two honest limits on this observation:
- Both runs return
Normalbecause no autovacuum was running locally. The signal is not the interesting part here; the connection is. - A full autovacuum-triggered
Stopis not reproducible outside specs.postgres_autovacuum_activityis a view overpg_stat_activity, so activity cannot be inserted; the spec suite only manages this through theswapout_view_for_tablehelper. TheStoppath is covered by the existing examples inspec/lib/gitlab/database/health_status/indicators/autovacuum_active_on_table_spec.rb.
MR acceptance checklist
Evaluate this MR against the MR acceptance checklist. It helps you analyze changes to reduce risks in quality, performance, reliability, security, and maintainability.