db/post_migrate/20240124043507_migrate_sidekiq_queued_and_future_jobs fails when over 30 seconds

Status

Workaround

Add disable_ddl_transaction! into the affected migration ; inserting within the Class before def up worked in our case

Description

Recently when updating from 16.9.4 to 16.10.4 in a customer production environment the post-migration stage failed with:

rails aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::ConnectionBad: PQsocket() can't get socket descriptor
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `exec'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:47:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'

rails aborted!
StandardError: An error has occurred, this and all later migrations canceled:

PG::ConnectionBad: PQsocket() can't get socket descriptor
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `exec'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:47:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:46:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/marginalia-1.11.1/lib/marginalia.rb:71:in `execute_with_marginalia'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:121:in `exec_rollback_db_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:374:in `rollback_db_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `rollback_db_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:214:in `rollback'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:311:in `block in rollback_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:309:in `rollback_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:325:in `rescue in block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:322:in `block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1408:in `execute_migration_in_transaction'
/srv/gitlab/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1481:in `block in with_advisory_lock'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1496:in `with_advisory_lock_connection'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1477:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1166:in `up'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1141:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/tasks/database_tasks.rb:262:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/railties/databases.rake:92:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:24:in `block (2 levels) in perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:24:in `block in perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/command.rb:51:in `invoke'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands.rb:18:in `<main>'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'

Caused by:
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQsocket() can't get socket descriptor
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `exec'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:47:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:46:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/marginalia-1.11.1/lib/marginalia.rb:71:in `execute_with_marginalia'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:121:in `exec_rollback_db_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:374:in `rollback_db_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `rollback_db_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:214:in `rollback'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:311:in `block in rollback_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:309:in `rollback_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:325:in `rescue in block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:322:in `block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1408:in `execute_migration_in_transaction'
/srv/gitlab/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1481:in `block in with_advisory_lock'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1496:in `with_advisory_lock_connection'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1477:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1166:in `up'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1141:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/tasks/database_tasks.rb:262:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/railties/databases.rake:92:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:24:in `block (2 levels) in perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:24:in `block in perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/command.rb:51:in `invoke'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands.rb:18:in `<main>'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'

Caused by:
PG::ConnectionBad: PQsocket() can't get socket descriptor
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `exec'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:48:in `block (2 levels) in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:47:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:46:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/marginalia-1.11.1/lib/marginalia.rb:71:in `execute_with_marginalia'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:121:in `exec_rollback_db_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:374:in `rollback_db_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/query_cache.rb:22:in `rollback_db_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:214:in `rollback'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:311:in `block in rollback_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:309:in `rollback_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:325:in `rescue in block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:322:in `block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1408:in `execute_migration_in_transaction'
/srv/gitlab/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1481:in `block in with_advisory_lock'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1496:in `with_advisory_lock_connection'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1477:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1166:in `up'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1141:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/tasks/database_tasks.rb:262:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/railties/databases.rake:92:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:24:in `block (2 levels) in perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:24:in `block in perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/command.rb:51:in `invoke'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands.rb:18:in `<main>'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'

Caused by:
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() FATAL:  terminating connection due to idle-in-transaction timeout
SSL connection has been closed unexpectedly
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `exec'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `block (2 levels) in query'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:18:in `block in query'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:17:in `query'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:99:in `query_values'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:52:in `tables'
/srv/gitlab/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:22:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:889:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:888:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:887:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1095:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1409:in `block in execute_migration_in_transaction'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1408:in `execute_migration_in_transaction'
/srv/gitlab/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1481:in `block in with_advisory_lock'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1496:in `with_advisory_lock_connection'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1477:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `migrate'
main: == [advisory_lock_connection] object_id: 53300, pg_backend_pid: 17362
main: == 20240124043507 MigrateSidekiqQueuedAndFutureJobs: migrating ================
main: == [advisory_lock_connection] object_id: 53300, pg_backend_pid: 17362
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1166:in `up'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1141:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/tasks/database_tasks.rb:262:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/railties/databases.rake:92:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:24:in `block (2 levels) in perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:24:in `block in perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/command.rb:51:in `invoke'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands.rb:18:in `<main>'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'

Caused by:
PG::ConnectionBad: PQconsumeInput() FATAL:  terminating connection due to idle-in-transaction timeout
SSL connection has been closed unexpectedly
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `exec'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:19:in `block (2 levels) in query'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/share_lock.rb:187:in `yield_shares'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/dependencies/interlock.rb:41:in `permit_concurrent_loads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:18:in `block in query'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:752:in `block in log'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/notifications/instrumenter.rb:24:in `instrument'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract_adapter.rb:743:in `log'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/postgresql/database_statements.rb:17:in `query'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:99:in `query_values'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:52:in `tables'
/srv/gitlab/lib/gitlab/database/migration_helpers/automatic_lock_writes_on_tables.rb:22:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:889:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:888:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:887:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1095:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1409:in `block in execute_migration_in_transaction'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:123:in `run_block'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:134:in `block in run_block_with_lock_timeout'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:319:in `block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activesupport-7.0.8.1/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/transaction.rb:317:in `within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/database_statements.rb:316:in `transaction'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `public_send'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:127:in `block in write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:141:in `block in read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:228:in `retry_with_backoff'
/srv/gitlab/lib/gitlab/database/load_balancing/load_balancer.rb:130:in `read_write'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:126:in `write_using_load_balancer'
/srv/gitlab/lib/gitlab/database/load_balancing/connection_proxy.rb:78:in `transaction'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:129:in `run_block_with_lock_timeout'
/srv/gitlab/lib/gitlab/database/with_lock_retries.rb:97:in `run'
/srv/gitlab/lib/gitlab/database/migrations/lock_retry_mixin.rb:52:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1408:in `execute_migration_in_transaction'
/srv/gitlab/lib/gitlab/database/migrations/runner_backoff/active_record_mixin.rb:21:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1382:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1481:in `block in with_advisory_lock'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:15:in `block in with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:215:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1496:in `with_advisory_lock_connection'
/srv/gitlab/lib/gitlab/database/migrations/pg_backend_pid.rb:12:in `with_advisory_lock_connection'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1477:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1331:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1166:in `up'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/migration.rb:1141:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/tasks/database_tasks.rb:262:in `migrate'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/activerecord-7.0.8.1/lib/active_record/railties/databases.rake:92:in `block (2 levels) in <main>'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `block in execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:281:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/sentry-ruby-5.10.0/lib/sentry/rake.rb:26:in `execute'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:219:in `block in invoke_with_call_chain'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `synchronize'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:199:in `invoke_with_call_chain'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/task.rb:188:in `invoke'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:160:in `invoke_task'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `each'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:116:in `block in top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:125:in `run_with_threads'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:110:in `top_level'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:24:in `block (2 levels) in perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/application.rb:186:in `standard_exception_handling'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:24:in `block in perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/rake-13.0.6/lib/rake/rake_module.rb:59:in `with_application'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands/rake/rake_command.rb:18:in `perform'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/command.rb:51:in `invoke'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/railties-7.0.8.1/lib/rails/commands.rb:18:in `<main>'
/srv/gitlab/vendor/bundle/ruby/3.1.0/gems/bootsnap-1.18.3/lib/bootsnap/load_path_cache/core_ext/kernel_require.rb:30:in `require'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
command terminated with exit code 1

After re-attempting and hitting the same error, we found #408922 (closed) which had same/similar causes.

We were able to proceed with the migrations by applying the workaround above.

Edited Apr 29, 2024 by Brendan McKitrick