Master broken: ./spec/lib/backup/database_spec.rb
In https://gitlab.com/gitlab-org/gitlab/-/jobs/3515050940, I'm seeing:
1st Try error in ./spec/lib/backup/database_spec.rb:250:
expected no Exception, got #<ActiveRecord::StatementInvalid: PG::OutOfMemory: ERROR: out of shared memory
HINT: You might need to increase max_locks_per_transaction.
> with backtrace:
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'
# ./lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
# ./lib/gitlab/database/load_balancing/connection_proxy.rb:89:in `method_missing'
# ./lib/tasks/gitlab/db.rake:76:in `block (2 levels) in drop_tables'
# ./lib/tasks/gitlab/db.rake:76:in `each'
# ./lib/tasks/gitlab/db.rake:76:in `block in drop_tables'
# ./lib/gitlab/database/each_database.rb:17:in `block (2 levels) in each_database_connection'
# ./lib/gitlab/database/each_database.rb:77:in `block in with_shared_connection'
# ./lib/gitlab/database/shared_model.rb:34:in `using_connection'
# ./lib/gitlab/database/each_database.rb:74:in `with_shared_connection'
# ./lib/gitlab/database/each_database.rb:16:in `block in each_database_connection'
# ./lib/gitlab/database/each_database.rb:11:in `each_pair'
# ./lib/gitlab/database/each_database.rb:11:in `each_database_connection'
# ./lib/tasks/gitlab/db.rake:59:in `drop_tables'
# ./lib/tasks/gitlab/db.rake:53:in `block (5 levels) in <main>'
# ./lib/backup/database.rb:245:in `drop_tables'
# ./lib/backup/database.rb:96:in `block in restore'
# ./lib/backup/database.rb:76:in `each'
# ./lib/backup/database.rb:76:in `restore'
# ./spec/lib/backup/database_spec.rb:138:in `block (4 levels) in <main>'
# ./spec/spec_helper.rb:415:in `block (3 levels) in <top (required)>'
# ./spec/support/sidekiq_middleware.rb:18:in `with_sidekiq_server_middleware'
# ./spec/spec_helper.rb:407:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:403:in `block (3 levels) in <top (required)>'
# ./lib/gitlab/application_context.rb:59:in `with_raw_context'
# ./spec/spec_helper.rb:403:in `block (2 levels) in <top (required)>'
# ./spec/spec_helper.rb:239:in `block (2 levels) in <top (required)>'
# ./spec/support/system_exit_detected.rb:7:in `block (2 levels) in <main>'
# ./spec/support/flaky_tests.rb:27:in `block (2 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (3 levels) in <main>'
# ./spec/support/database/prevent_cross_joins.rb:60:in `with_cross_joins_prevented'
# ./spec/support/database/prevent_cross_joins.rb:106:in `block (2 levels) in <main>'
Also perhaps related the db:backup_and_restore
job is failing (https://gitlab.com/gitlab-org/gitlab/-/jobs/3515051094):
2022-12-22 16:25:44 +0000 -- Deleting backup and restore lock file
rake aborted!
ActiveRecord::StatementInvalid: PG::UndefinedTable: ERROR: relation "snippets" does not exist
LINE 1: SELECT "snippets".* FROM "snippets" ORDER BY "snippets"."id"...
^
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `exec_params'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql_adapter.rb:672:in `block (2 levels) in exec_no_cache'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies/interlock.rb:48:in `block in permit_concurrent_loads'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/dependencies/interlock.rb:47:in `permit_concurrent_loads'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql_adapter.rb:671:in `block in exec_no_cache'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract_adapter.rb:696:in `block (2 levels) in log'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract_adapter.rb:695:in `block in log'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activesupport-6.1.6.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract_adapter.rb:687:in `log'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql_adapter.rb:670:in `exec_no_cache'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql_adapter.rb:649:in `execute_and_clear'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/marginalia-1.11.1/lib/marginalia.rb:91:in `execute_and_clear_with_marginalia'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:55:in `exec_query'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:532:in `select'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/database_statements.rb:69:in `select_all'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/query_cache.rb:103:in `select_all'
/builds/gitlab-org/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:104:in `block in read_using_load_balancer'
/builds/gitlab-org/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:55:in `read'
/builds/gitlab-org/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:103:in `read_using_load_balancer'
/builds/gitlab-org/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:48:in `select_all'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/querying.rb:47:in `find_by_sql'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation.rb:843:in `block in exec_queries'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation.rb:858:in `block in skip_query_cache_if_necessary'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/connection_adapters/abstract/query_cache.rb:79:in `uncached'
/builds/gitlab-org/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
/builds/gitlab-org/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'
/builds/gitlab-org/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'
/builds/gitlab-org/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'
/builds/gitlab-org/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
/builds/gitlab-org/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:89:in `method_missing'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/query_cache.rb:21:in `uncached'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation/delegation.rb:108:in `public_send'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation/delegation.rb:108:in `block in method_missing'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation.rb:406:in `block in scoping'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation.rb:804:in `_scoping'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation.rb:406:in `scoping'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation/delegation.rb:108:in `method_missing'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation.rb:857:in `skip_query_cache_if_necessary'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation.rb:828:in `exec_queries'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation.rb:631:in `load'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation.rb:249:in `records'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation/batches.rb:231:in `block in in_batches'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation/batches.rb:229:in `loop'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation/batches.rb:229:in `in_batches'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation/batches.rb:137:in `find_in_batches'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/activerecord-6.1.6.1/lib/active_record/relation/batches.rb:70:in `find_each'
/builds/gitlab-org/gitlab/lib/backup/repositories.rb:119:in `each'
/builds/gitlab-org/gitlab/lib/backup/repositories.rb:119:in `cleanup_snippets_without_repositories'
/builds/gitlab-org/gitlab/lib/backup/repositories.rb:39:in `ensure in restore'
/builds/gitlab-org/gitlab/lib/backup/repositories.rb:40:in `restore'
/builds/gitlab-org/gitlab/lib/backup/manager.rb:100:in `run_restore_task'
/builds/gitlab-org/gitlab/lib/backup/manager.rb:233:in `block in run_all_restore_tasks'
/builds/gitlab-org/gitlab/lib/backup/manager.rb:231:in `each_key'
/builds/gitlab-org/gitlab/lib/backup/manager.rb:231:in `run_all_restore_tasks'
/builds/gitlab-org/gitlab/lib/backup/manager.rb:75:in `restore'
/builds/gitlab-org/gitlab/lib/tasks/gitlab/backup.rake:26:in `block (4 levels) in <main>'
/builds/gitlab-org/gitlab/lib/tasks/gitlab/backup.rake:203:in `lock'
/builds/gitlab-org/gitlab/lib/tasks/gitlab/backup.rake:23:in `block (3 levels) in <main>'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/sentry-ruby-core-5.1.1/lib/sentry/rake.rb:26:in `execute'
/builds/gitlab-org/gitlab/vendor/ruby/2.7.0/gems/rake-13.0.6/exe/rake:27:in `<top (required)>'
/usr/local/bin/bundle:23:in `load'
/usr/local/bin/bundle:23:in `<main>'