Skip to content

Upgrading PostgreSQL to 14 fails due to Mixlib::ShellOut::CommandTimeout of 600s

Summary

Upgrading the Omnibus Debian package or running gitlab-ctl pg-upgrade fails after ~10 minutes

Steps to reproduce

Probably need a huge db (data is ~66GB) and a slow disk.

What is the current bug behavior?

The 10 minute timeout of the external command runner kills the pg_upgrade process.

What is the expected correct behavior?

Executing the upgrade command should not fail!

Relevant logs

Relevant logs 
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 14.11
Checking if disk for directory /data/gitlab/postgresql/data has enough free space for PostgreSQL upgrade: OK
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details


Please hit Ctrl-C now if you want to cancel the operation.
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 1s, normally up
ok: down: gitaly: 0s, normally up
ok: down: gitlab-exporter: 1s, normally up
ok: down: gitlab-kas: 0s, normally up
ok: down: gitlab-pages: 1s, normally up
ok: down: gitlab-sshd: 0s, normally up
ok: down: logrotate: 1s, normally up
ok: down: mailroom: 0s, normally up
ok: down: mattermost: 0s, normally up
ok: down: node-exporter: 1s, normally up
ok: down: postgres-exporter: 0s, normally up
ok: down: prometheus: 1s, normally up
ok: down: redis-exporter: 0s, normally up
ok: down: registry: 0s, normally up
ok: down: sidekiq: 0s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 0s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database: OK
Upgrading the data:/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:140:in log': wrong number of arguments (given 0, expected 1) (ArgumentError) from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:160:in rescue in block in run_pg_upgrade'
from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:146:in block in run_pg_upgrade' from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:126:in progress_message'
from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:145:in run_pg_upgrade' from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:348:in general_upgrade'
from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:299:in block in load_file' from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:204:in block in add_command_under_category'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:747:in run' from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/bin/omnibus-ctl:31:in <top (required)>'
from /opt/gitlab/embedded/bin/omnibus-ctl:25:in load' from /opt/gitlab/embedded/bin/omnibus-ctl:25:in '
/opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout/unix.rb:128:in `run_command': Command timed out after 600s: (Mixlib::ShellOut::CommandTimeout)
Command exceeded allowed execution time, process terminated
---- Begin output of su - gitlab-psql -c "/opt/gitlab/embedded/postgresql/14/bin/pg_upgrade -b /opt/gitlab/embedded/postgresql/13/bin --old-datadir=/data/gitlab/postgresql/data  --new-datadir=/data/gitlab/postgresql/data.14  -B /opt/gitlab/embedded/postgresql/14/bin " ----
STDOUT: Performing Consistency Checks


Checking cluster versions                                   ok
Checking database user is the install user                  ok
Checking database connection settings                       ok
Checking for prepared transactions                          ok
Checking for system-defined composite types in user tables  ok
Checking for reg* data types in user tables                 ok
Checking for contrib/isn with bigint-passing mismatch       ok
Checking for user-defined encoding conversions              ok
Checking for user-defined postfix operators                 ok
Checking for incompatible polymorphic functions             ok
Creating dump of global objects                             ok
Creating dump of database schemas
gitlabhq_production
mattermost_production
postgres
template1
ok
Checking for presence of required libraries                 ok
Checking database user is the install user                  ok
Checking for prepared transactions                          ok
Checking for new cluster tablespace directories             ok


If pg_upgrade fails after this point, you must re-initdb the
new cluster before continuing.



Performing Upgrade


Analyzing all rows in the new cluster                       ok
Freezing all rows in the new cluster                        ok
Deleting files from new pg_xact                             ok
Copying old pg_xact to new server                           ok
Setting oldest XID for new cluster                          ok
Setting next transaction ID and epoch for new cluster       ok
Deleting files from new pg_multixact/offsets                ok
Copying old pg_multixact/offsets to new server              ok
Deleting files from new pg_multixact/members                ok
Copying old pg_multixact/members to new server              ok
Setting next multixact ID and offset for new cluster        ok
Resetting WAL archives                                      ok
Setting frozenxid and minmxid counters in new cluster       ok
Restoring global objects in the new cluster                 ok
Restoring database schemas in the new cluster
template1
gitlabhq_production
mattermost_production
postgres
ok
Copying user relation files
/data/gitlab/postgresql/data/base/16401/2613
/data/gitlab/postgresql/data/base/16401/2683
/data/gitlab/postgresql/data/base/16401/71595
/data/gitlab/postgresql/data/base/16401/71595_fsm
/data/gitlab/postgresql/data/base/16401/71595_vm
/data/gitlab/postgresql/data/base/16401/71598


[...]


/data/gitlab/postgresql/data/base/16401/8706418
STDERR: -sh: 20: source: not found
-sh: 20: source: not found
-sh: 20: source: not found
-sh: 20: source: not found
-sh: 20: source: not found


Session terminated, killing shell... ...killed.
---- End output of su - gitlab-psql -c "/opt/gitlab/embedded/postgresql/14/bin/pg_upgrade -b /opt/gitlab/embedded/postgresql/13/bin --old-datadir=/data/gitlab/postgresql/data  --new-datadir=/data/gitlab/postgresql/data.14  -B /opt/gitlab/embedded/postgresql/14/bin " ----
Ran su - gitlab-psql -c "/opt/gitlab/embedded/postgresql/14/bin/pg_upgrade -b /opt/gitlab/embedded/postgresql/13/bin --old-datadir=/data/gitlab/postgresql/data  --new-datadir=/data/gitlab/postgresql/data.14  -B /opt/gitlab/embedded/postgresql/14/bin " returned
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/mixlib-shellout-3.2.7/lib/mixlib/shellout.rb:270:in run_command' from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:33:in run_command'
from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:16:in get_command_output' from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:73:in run_pg_command'
from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:147:in block in run_pg_upgrade' from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/util.rb:126:in progress_message'
from /opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:145:in run_pg_upgrade' from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:348:in general_upgrade'
from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:299:in block in load_file' from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:204:in block in add_command_under_category'
from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/lib/omnibus-ctl.rb:747:in run' from /opt/gitlab/embedded/lib/ruby/gems/3.1.0/gems/omnibus-ctl-0.6.12/bin/omnibus-ctl:31:in <top (required)>'
from /opt/gitlab/embedded/bin/omnibus-ctl:25:in load' from /opt/gitlab/embedded/bin/omnibus-ctl:25:in '

Details of package version

Provide the package version installation details 
ii  gitlab-ee      16.11.0-ee.0 amd64        GitLab Enterprise Edition (including NGINX, Postgres, Redis)

Environment details

  • Operating System: Debian GNU/Linux 12 (bookworm)
  • Installation Target:
    • VM: Hyper-V
  • Installation Type:
    • Upgrade from version long time ago
  • Is there any other software running on the machine: No
  • Is this a single or multiple node installation?
  • Resources
    • CPU: 24
    • Memory total: 48G

Configuration details

`gitlab-psql -c '\l+'` 
                                                                                   List of databases
         Name          |       Owner       | Encoding | Collate |  Ctype  |            Access privileges            |  Size   | Tablespace |                Description                 
-----------------------+-------------------+----------+---------+---------+-----------------------------------------+---------+------------+--------------------------------------------
 gitlabhq_production   | gitlab            | UTF8     | C.UTF-8 | C.UTF-8 | =Tc/gitlab                             +| 48 GB   | pg_default | 
                       |                   |          |         |         | gitlab=CTc/gitlab                      +|         |            | 
                       |                   |          |         |         | cmk=CTc/gitlab                          |         |            | 
 mattermost_production | gitlab_mattermost | UTF8     | C.UTF-8 | C.UTF-8 | =Tc/gitlab_mattermost                  +| 18 GB   | pg_default | 
                       |                   |          |         |         | gitlab_mattermost=CTc/gitlab_mattermost+|         |            | 
                       |                   |          |         |         | cmk=CTc/gitlab_mattermost               |         |            | 
 postgres              | gitlab-psql       | UTF8     | C.UTF-8 | C.UTF-8 |                                         | 7837 kB | pg_default | default administrative connection database
 template0             | gitlab-psql       | UTF8     | C.UTF-8 | C.UTF-8 | =c/"gitlab-psql"                       +| 7465 kB | pg_default | unmodifiable empty database
                       |                   |          |         |         | "gitlab-psql"=CTc/"gitlab-psql"         |         |            | 
 template1             | gitlab-psql       | UTF8     | C.UTF-8 | C.UTF-8 | "gitlab-psql"=CTc/"gitlab-psql"        +| 7853 kB | pg_default | default template for new databases
                       |                   |          |         |         | =c/"gitlab-psql"                        |         |            | 
(5 rows)