Rails schema queries can cause migrations to fail with key "unknown"
@peterhegman had an issue where running bundle exec rake db:migrate failed:
StandardError: An error has occurred, all later migrations canceled:
key not found: "unknown"
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database.rb:207:in `gitlab_schemas_for_connection'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/query_analyzers/gitlab_schemas_validate_connection.rb:21:in `analyze'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/query_analyzer.rb:89:in `block in process_sql'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/query_analyzer.rb:86:in `each'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/query_analyzer.rb:86:in `process_sql'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/query_analyzer.rb:28:in `block (2 levels) in hook!'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/query_analyzer.rb:114:in `with_ignored_recursive_calls'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/query_analyzer.rb:27:in `block in hook!'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:111:in `block in read_write'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:172:in `retry_with_backoff'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:110:in `read_write'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:118:in `write_using_load_balancer'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:88:in `method_missing'
/Users/peterhegman/Work/gdk-2/gitlab/app/models/concerns/transactions.rb:13:in `inside_transaction?'
/Users/peterhegman/Work/gdk-2/gitlab/config/initializers/forbid_sidekiq_in_transactions.rb:45:in `inside_transaction?'
/Users/peterhegman/Work/gdk-2/gitlab/config/initializers/forbid_sidekiq_in_transactions.rb:23:in `block (2 levels) in <module:NoEnqueueingFromTransactions>'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/background_migration/job_coordinator.rb:43:in `perform_in'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/migrations/background_migration_helpers.rb:205:in `block in migrate_in'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/application_context.rb:100:in `block in use'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/application_context.rb:100:in `use'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/application_context.rb:46:in `with_context'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/migrations/background_migration_helpers.rb:230:in `with_migration_context'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/migrations/background_migration_helpers.rb:204:in `migrate_in'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/migrations/background_migration_helpers.rb:88:in `block in queue_background_migration_jobs_by_range_at_intervals'
/Users/peterhegman/Work/gdk-2/gitlab/app/models/concerns/each_batch.rb:98:in `block (2 levels) in each_batch'
/Users/peterhegman/Work/gdk-2/gitlab/app/models/concerns/each_batch.rb:98:in `block in each_batch'
/Users/peterhegman/Work/gdk-2/gitlab/app/models/concerns/each_batch.rb:68:in `step'
/Users/peterhegman/Work/gdk-2/gitlab/app/models/concerns/each_batch.rb:68:in `each_batch'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/migrations/background_migration_helpers.rb:75:in `queue_background_migration_jobs_by_range_at_intervals'
/Users/peterhegman/Work/gdk-2/gitlab/db/post_migrate/20220407163559_schedule_purging_stale_security_scans.rb:14:in `up'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:33:in `block in migrate'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/query_analyzer.rb:37:in `within'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/migration_helpers/restrict_gitlab_schema.rb:30:in `migrate'
/Users/peterhegman/Work/gdk-2/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:36:in `ddl_transaction'20220407163559_schedule_purging_stale_security_scans.rb attempts to remove stale security_scans data (i.e. those with created_at entries older than 90 days). This migration only runs in test/development or GitLab.com. However, this migration crashed when a Rails schema query attempted to run:
SELECT t.oid, t.typname\nFROM pg_type as t\nWHERE t.typname IN ('int2', 'int4', 'int8', 'oid', 'float4', 'float8', 'numeric', 'bool', 'timestamp', 'timestamptz')\n /*application:web,correlation_id:2de05935d51087b960e8d1fb5446993c,endpoint_id:SchedulePurgingStaleSecurityScans,db_config_name:unknown,line:/app/models/concerns/transactions.rb:13:in `inside_transaction?'*/"Since db_config_name was unknown here, the migration failed.
To reproduce the issue:
- I updated my
security_scanstable:
gitlabhq_development=# update security_scans set created_at = '2021-03-17 18:13:32.573576+00';
UPDATE 3- Then I ran
bundle exec rake db:migrate:redo:main VERSION=20220407163559.
If I look at the debugger:
[16, 25] in /Users/stanhu/gdk-ee/gitlab/lib/gitlab/database/query_analyzers/gitlab_schemas_validate_connection.rb
16: def analyze(parsed)
17: tables = parsed.pg.select_tables + parsed.pg.dml_tables
18: table_schemas = ::Gitlab::Database::GitlabSchema.table_schemas(tables)
19: return if table_schemas.empty?
20:
=> 21: allowed_schemas = ::Gitlab::Database.gitlab_schemas_for_connection(parsed.connection)
22: return unless allowed_schemas
23:
24: invalid_schemas = table_schemas - allowed_schemas
25: if invalid_schemas.any?
(byebug) table_schemas
#<Set: {:gitlab_shared}>
(byebug) tables
["pg_type"]
(byebug) parsed.connection.class
ActiveRecord::ConnectionAdapters::PostgreSQLAdapter
(byebug) parsed.connection.pool
#<ActiveRecord::ConnectionAdapters::NullPool:0x00000001283e3680>
(byebug) parsed.connection.pool.db_config
*** NoMethodError Exception: undefined method `db_config' for #<ActiveRecord::ConnectionAdapters::NullPool:0x00000001283e3680>
nil/cc: @ayufan @DylanGriffith
Edited by Stan Hu