Package upgrade fails on dev.gitlab.org
Upgrades have been failing on dev.gitlab.org starting on 2023-01-23 with the following logs
Preparing to unpack .../gitlab-ce_15.8.0+rnightly.269380.01aab3af-0_amd64.deb ...
gitlab preinstall: Checking for unmigrated data on legacy storage
gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!)
2023-01-24 03:21:55 UTC -- Dumping database ...
Dumping PostgreSQL database gitlabhq_production ... 2023-01-24 03:37:11 UTC -- Deleting tar staging files ...
2023-01-24 03:37:11 UTC -- Cleaning up /var/opt/gitlab/backups/db
2023-01-24 03:37:12 UTC -- Deleting tar staging files ... done
2023-01-24 03:37:12 UTC -- Deleting backups/tmp ...
2023-01-24 03:37:12 UTC -- Deleting backups/tmp ... done
2023-01-24 03:37:12 +0000 -- Deleting backup and restore lock file
rake aborted!
ActiveRecord::StatementInvalid: PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:89:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:65:in `block in dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:37:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:37:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:66:in `run_create_task'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:211:in `block in run_all_create_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:210:in `each_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:210:in `run_all_create_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:42:in `create'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:16:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:203:in `lock'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:13:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::ConnectionBad: PQconsumeInput() server closed the connection unexpectedly
This probably means the server terminated abnormally
before or while processing the request.
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:120:in `block in write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:129:in `block in read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:200:in `retry_with_backoff'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/load_balancer.rb:118:in `read_write'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:119:in `write_using_load_balancer'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/load_balancing/connection_proxy.rb:89:in `method_missing'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:65:in `block in dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:37:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/database.rb:37:in `dump'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:66:in `run_create_task'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:211:in `block in run_all_create_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:210:in `each_key'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:210:in `run_all_create_tasks'
/opt/gitlab/embedded/service/gitlab-rails/lib/backup/manager.rb:42:in `create'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:16:in `block (4 levels) in <top (required)>'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:203:in `lock'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/backup.rake:13:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => gitlab:backup:create
(See full trace by running task with --trace)
gitlab preinstall:
gitlab preinstall: Database backup failed! If you want to skip this backup, run the following command and try again:
gitlab preinstall:
gitlab preinstall: sudo touch /etc/gitlab/skip-auto-backup
gitlab preinstall:
dpkg: error processing archive /var/cache/apt/archives/gitlab-ce_15.8.0+rnightly.269380.01aab3af-0_amd64.deb (--unpack):
new gitlab-ce package pre-installation script subprocess returned error exit status 1
Errors were encountered while processing:
/var/cache/apt/archives/gitlab-ce_15.8.0+rnightly.269380.01aab3af-0_amd64.deb
From /var/log/apt/term.log
it seems this was first seen when upgrading to package version gitlab-ce_15.8.0+rnightly.269160.0a7b2380
, created by the nightly pipeline https://dev.gitlab.org/gitlab/omnibus-gitlab/pipelines/269160, containing the following component SHAs
gitlab-rails : eb3a8e39372e4cb3ad5d932d85d43d2b10f36744
gitlab-shell : ad172bf0c9688238bc014d2ded2b181ae0b6e45a
gitlab-pages : 7aea631ecdf2ff9a9bd6df6c9da449b5ad4ac2a8
gitaly : d7b9dcec61e29644ffc58dd2a59756435bf58bcb
gitlab-kas : 71d2bb6083d48cdc7d38eb6667a7370cd3f09428