Postgres upgrade from 9.6.11 to 10.9 fails

Summary

When upgrading from 12.1.17 (pg 9.6.11) to any of 12.2.x, 12.3.x or 12.5.x (pg 10.9) it fails to start.

After countless tries and rm -R /var/opt/gitlab/postgresql/data.10 I hope anyone can help me out with this.

Steps to reproduce

Install using image 12.1.17 and upgrade to e.g. image 12.2.0.

What is the current bug behavior?

It won't start due to the postgres upgrade error.

What is the expected correct behavior?

A running 12.2.x, 12.3.x or 12.5.x instance with postgres 10.9 db 😉

Relevant logs

Running handlers:
Running handlers complete
Chef Client finished, 379/1345 resources updated in 01 minutes 25 seconds
gitlab Reconfigured!
Checking for an omnibus managed postgresql: OK
Checking for a newer version of PostgreSQL to install
Upgrading PostgreSQL to 10.9
Checking if we already upgraded: NOT OK
Checking if PostgreSQL bin files are symlinked to the expected location: OK
Checking if postgresql['version'] is set: OK
cp /opt/gitlab/embedded/service/gitlab-rails/public/deploy.html /opt/gitlab/embedded/service/gitlab-rails/public/index.html
Toggling deploy page: OK
Toggling services:ok: down: alertmanager: 1s, normally up
ok: down: gitaly: 0s, normally up
ok: down: gitlab-exporter: 0s, normally up
ok: down: grafana: 1s, normally up
ok: down: logrotate: 0s, normally up
ok: down: postgres-exporter: 1s, normally up
ok: down: prometheus: 0s, normally up
ok: down: redis-exporter: 0s, normally up
ok: down: sidekiq: 0s, normally up
ok: down: sshd: 1s, normally up
Toggling services: OK
Running stop on postgresql:ok: down: postgresql: 0s, normally up
Running stop on postgresql: OK
Symlink correct version of binaries: OK
Creating temporary data directory: OK
Initializing the new database: OK
Upgrading the data: NOT OK
Error upgrading the data to version 10.9
STDOUT: 
You must have read and write access in the current directory.
Failure, exiting
STDERR: 
/opt/gitlab/embedded/service/omnibus-ctl/lib/gitlab_ctl/pg_upgrade.rb:130:in `run_pg_upgrade': undefined method `ExecutionError' for GitlabCtl::Errors:Class (NoMethodError)
	from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:228:in `general_upgrade'
	from /opt/gitlab/embedded/service/omnibus-ctl/pg-upgrade.rb:175:in `block in load_file'
	from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:204:in `block in add_command_under_category'
	from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/lib/omnibus-ctl.rb:746:in `run'
	from /opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/omnibus-ctl-0.6.0/bin/omnibus-ctl:31:in `<top (required)>'
	from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `load'
	from /opt/gitlab/embedded/bin/omnibus-ctl:23:in `<main>'
Upgrading the existing database to 10 failed and was reverted.
Please check the output, and open an issue at:
https://gitlab.com/gitlab-org/omnibus-gitlab/issues
If you would like to restart the instance without attempting to
upgrade, add the following to your docker command:
-e GITLAB_SKIP_PG_UPGRADE=true

Details of package version

Environment details

  • Operating System: RancherOS 1.5.5
  • Installation Target, remove incorrect values:
    • VM: ProxmoxVE 6.1
  • Installation Type, remove incorrect values:
  • Is there any other software running on the machine: yes, other containers 😉
  • Is this a single or multiple node installation? single node
  • Resources
    • CPU: 4 (Intel i7)
    • Memory total: 20 GB

Configuration details

GITLAB_OMNIBUS_CONFIG: |
    external_url 'https://gitlab.my-server.com'
    gitlab_rails['time_zone'] = 'Europe/Berlin'
    gitlab_rails['gitlab_signup_enabled'] = false
    gitlab_rails['gitlab_signin_enabled'] = true
    gitlab_rails['gitlab_default_projects_limit'] = 100
    gitlab_rails['gravatar_enabled'] = false        
    gitlab_rails['gitlab_default_can_create_group'] = false
    gitlab_rails['gitlab_username_changing_enabled'] = false
    gitlab_rails['gitlab_default_theme'] = 2
    gitlab_rails['gitlab_default_projects_features_issues'] = false
    gitlab_rails['gitlab_default_projects_features_merge_requests'] = true
    gitlab_rails['gitlab_default_projects_features_wiki'] = false
    gitlab_rails['gitlab_default_projects_features_snippets'] = false
    gitlab_rails['gitlab_email_from'] = 'admin@my-server.com'
    gitlab_rails['smtp_enable'] = true
    gitlab_rails['smtp_address'] = 'mail.my-server.com'
    gitlab_rails['smtp_port'] = 465
    gitlab_rails['smtp_user_name'] = 'admin@my-server.com'
    gitlab_rails['smtp_password'] = 'p@$$w0rd'
    gitlab_rails['smtp_tls'] = true
    gitlab_shell['http_settings_self_signed_cert'] = 'true'
    nginx['listen_port'] = 80
    nginx['listen_https'] = false
    nginx['proxy_set_headers'] = {
        "X-Forwarded-Proto" => "https",
        "X-Forwarded-Ssl" => "on" }
Edited by D. Domig