Error upgrading postgresql 12.12 -> 13.8
I upgrades to gitlab-ce 15.11 last week and have been getting an error with pg-upgrade. This is an attempt to run it manually outside of the package update, but showing the same results:
# gitlab-ctl pg-upgrade
Checking for an omnibus managed postgresql: OK
Checking if postgresql['version'] is set: OK
Checking if we already upgraded: NOT OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 13.8
Checking if disk for directory /var/opt/gitlab/postgresql/data has enough free space for PostgreSQL upgrade: OK
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Waiting 30 seconds to ensure tasks complete before PostgreSQL upgrade.
See https://docs.gitlab.com/omnibus/settings/database.html#upgrade-packaged-postgresql-server for details
If you do not want to upgrade the PostgreSQL server at this time, enter Ctrl-C and see the documentation for details
Please hit Ctrl-C now if you want to cancel the operation.
Toggling deploy page:cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: gitaly: 0s, normally up
ok: down: gitlab-kas: 0s, normally up
ok: down: grafana: 0s, normally up
ok: down: logrotate: 1s, normally up
ok: down: registry: 0s, normally up
ok: down: sidekiq: 0s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 1s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory:Error creating new directory: /var/opt/gitlab/postgresql/data.13
STDOUT:
STDERR: su: Permission denied
Creating temporary data directory: NOT OK
== Fatal error ==
Please check the output
== Reverting ==
ok: down: postgresql: 2s, normally up
Symlink correct version of binaries: OK
ok: run: postgresql: (pid 1870773) 1s
== Reverted ==
== Reverted to 12.12. Please check output for what went wrong ==
Toggling deploy page:rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: run: gitaly: (pid 1870785) 0s
ok: run: gitlab-kas: (pid 1870791) 1s
ok: run: grafana: (pid 1870815) 0s
ok: run: logrotate: (pid 1870826) 0s
ok: run: registry: (pid 1870832) 1s
ok: run: sidekiq: (pid 1870842) 0s
Toggling services: OK
The error is pretty clear that it is unable to create the temp directory /var/opt/gitlab/postgresql/data.13, but I'm not seeing why. The parent dir, /var/opt/gitlab/postgresql/, is owned by the gitlab-psql user with mode 755. There isn't a preexisting dir data.13 in there that is owned by another user. I'm not finding any other logs that will give me any more details on the problem. I tried adding the directory manually and assigning ownership to the gitlab-psql user and setting mode 755 and am still getting the error.
The upgrade to gitlab-ce itself seems to have otherwise been successful, and everything is running fine without the postgresql upgrade. It seems safe to pin it at 12 for now, but I would like to be able to stay as current as possible. I posted this question in the Community forum and was unable to succeed using suggestions there. I suspect there is a bug in /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb that is triggering a false positive error state, but I don't know ruby well enough to say exactly what the problem might be. I'm happy to provide further details or do further testing if needed.
I'm installing gitlab-ce from official gitlab.com repos. Running on Debian 11, also fully patched and up to date. This is a production standalone system that has been running and upgraded continuously for 6 years. I have had no problems with upgrades in the past, even when upgrading Debian from 10 to 11.