Gitlab upgrade to 12.10 fails due to failed upgrade of postgres to version 11
Summary
Docker install
Upgrading from 12.9.3 to 12.10.1 fails with an error related to the upgrade of postgres to 11.7
If I update to 12.9.4 no problems, everything works...
If I upgrade to 12.10.1 with the flag "-e GITLAB_SKIP_PG_UPGRADE=true", everything works
Fo the sake of following every advice given by the logs, I tried (without much hope) :
docker exec -it gitlab update-permissions
docker restart gitlab
didn't solve anything
Steps to reproduce
The actual commands I run :
docker stop gitlab
docker rm gitlab
docker pull gitlab/gitlab-ce:latest
docker run --detach \
--hostname git.mydomain.com \
--publish 443:443 --publish 80:80 --publish 22:22 \
--name gitlab \
--restart always \
--volume /srv/gitlab/config:/etc/gitlab \
--volume /srv/gitlab/logs:/var/log/gitlab \
--volume /srv/gitlab/data:/var/opt/gitlab \
gitlab/gitlab-ce:latest
What is the current bug behavior?
The container starts up but fails and restart every 30/40 seconds or so...
What is the expected correct behavior?
The container starts up and stays up
Relevant logs
Update logs - Eror
Running handlers complete Chef Client failed. 6 resources updated in 28 seconds == Fatal error == Something went wrong during final reconfiguration, please check the output Running reconfigure: NOT OK == Reverting == ok: down: postgresql: 1s, normally up Symlink correct version of binaries: OK ok: run: postgresql: (pid 1331) 0s == Reverted == == Reverted to 10.12. Please check output for what went wrong == rm -f /opt/gitlab/embedded/service/gitlab-rails/public/index.html Toggling deploy page: OK Toggling services:ok: run: alertmanager: (pid 1344) 0s ok: run: crond: (pid 1354) 0s ok: run: gitaly: (pid 1363) 1s ok: run: gitlab-exporter: (pid 1381) 0s ok: run: grafana: (pid 1385) 1s ok: run: logrotate: (pid 1408) 0s ok: run: postgres-exporter: (pid 1415) 1s ok: run: prometheus: (pid 1424) 0s ok: run: redis-exporter: (pid 1435) 0s ok: run: registry: (pid 1437) 0s ok: run: sidekiq: (pid 1453) 1s ok: run: sshd: (pid 1459) 0s Toggling services: OK Upgrading the existing database failed and was reverted. Please check the output, and open an issue at: https://gitlab.com/gitlab-org/omnibus-gitlab/issues If you would like to restart the instance without attempting to upgrade, add the following to your docker command: -e GITLAB_SKIP_PG_UPGRADE=true Thank you for using GitLab Docker Image! Current version: gitlab-ce=12.10.1-ce.0
Postgres related logs
Recipe: postgresql::enable * runit_service[postgresql] action enable * ruby_block[restart_service] action nothing (skipped due to action :nothing) * ruby_block[restart_log_service] action nothing (skipped due to action :nothing) * ruby_block[reload_log_service] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/sv/postgresql] action create (up to date) * template[/opt/gitlab/sv/postgresql/run] action create (up to date) * directory[/opt/gitlab/sv/postgresql/log] action create (up to date) * directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date) * template[/opt/gitlab/sv/postgresql/log/run] action create (up to date) * template[/var/log/gitlab/postgresql/config] action create (up to date) * ruby_block[verify_chown_persisted_on_postgresql] action nothing (skipped due to action :nothing) * directory[/opt/gitlab/sv/postgresql/env] action create (up to date) * ruby_block[Delete unmanaged env files for postgresql service] action run (skipped due to only_if) * template[/opt/gitlab/sv/postgresql/check] action create (skipped due to only_if) * template[/opt/gitlab/sv/postgresql/finish] action create (skipped due to only_if) * directory[/opt/gitlab/sv/postgresql/control] action create (up to date) * template[/opt/gitlab/sv/postgresql/control/t] action create (up to date) * link[/opt/gitlab/init/postgresql] action create (up to date) * file[/opt/gitlab/sv/postgresql/down] action delete (up to date) * directory[/opt/gitlab/service] action create (up to date) * link[/opt/gitlab/service/postgresql] action create (up to date) * ruby_block[wait for postgresql service socket] action run (skipped due to not_if) * directory[/opt/gitlab/service/postgresql/supervise] action create (up to date) * directory[/opt/gitlab/service/postgresql/log/supervise] action create (up to date) * file[/opt/gitlab/sv/postgresql/supervise/ok] action touch (skipped due to only_if) * file[/opt/gitlab/sv/postgresql/log/supervise/ok] action touch (skipped due to only_if) * file[/opt/gitlab/sv/postgresql/supervise/status] action touch - change owner from 'root' to 'gitlab-psql' - change group from 'root' to 'gitlab-psql' - update utime on file /opt/gitlab/sv/postgresql/supervise/status * file[/opt/gitlab/sv/postgresql/log/supervise/status] action touch (skipped due to only_if) * file[/opt/gitlab/sv/postgresql/supervise/control] action touch (skipped due to only_if) * file[/opt/gitlab/sv/postgresql/log/supervise/control] action touch (skipped due to only_if) * template[/opt/gitlab/etc/gitlab-psql-rc] action create (up to date) * postgresql_user[gitlab] action create * execute[create gitlab postgresql user] action run (skipped due to not_if) (up to date) * execute[create gitlabhq_production database] action run (skipped due to not_if) * postgresql_user[gitlab_replicator] action create * execute[create gitlab_replicator postgresql user] action run (skipped due to not_if) * execute[set options for gitlab_replicator postgresql user] action run (skipped due to not_if) (up to date) * postgresql_extension[pg_trgm] action enable * postgresql_query[enable pg_trgm extension] action run (skipped due to only_if) (up to date) * ruby_block[warn pending postgresql restart] action run (skipped due to only_if) * execute[reload postgresql] action nothing (skipped due to action :nothing) * execute[start postgresql] action nothing (skipped due to action :nothing) Recipe: gitlab::database_migrations * bash[migrate gitlab-rails database] action run (skipped due to not_if) * bash[set ownership of old migration log files] action run - execute "bash" "/tmp/chef-script20200426-907-1kyn29"
Details of package version
Provide the package version installation details
Docker version 19.03.6, build 369ce74a3c
Environment details
- Operating System: Ubuntu 18.04.4
- Installation Target, remove incorrect values:
- VM: AWS
- Installation Type:
- Upgrade from version : 12.9.3
- Is there any other software running on the machine: NO
- Is this a single or multiple node installation? Single
- Resources
- CPU: 2CPU
- Memory total: 8Go
Configuration details
Provide the relevant sections of `/etc/gitlab/gitlab.rb`
external_url 'https://git.mydomain.com'
gitlab_rails['gitlab_email_enabled'] = false gitlab_rails['gitlab_email_from'] = 'ne-pas-repondre@mydomain.com' gitlab_rails['gitlab_email_display_name'] = 'gitlab @mydomain.com'
gitlab_rails['ldap_enabled'] = true gitlab_rails['ldap_servers'] = YAML.load <<-'EOS' # remember to close this block with 'EOS' below main: # 'main' is the GitLab 'provider ID' of this LDAP server [... Not relevant to this issue] EOS
gitlab_rails['omniauth_enabled'] = false gitlab_rails['omniauth_providers'] = [ { "name" => "bitbucket", "app_id" => "", "app_secret" => "", "url" => "https://bitbucket.org/" } ]
gitlab_rails['smtp_enable'] = true gitlab_rails['smtp_address'] = "email-smtp.eu-central-1.amazonaws.com" gitlab_rails['smtp_port'] = 587 gitlab_rails['smtp_user_name'] = "" gitlab_rails['smtp_password'] = "" gitlab_rails['smtp_domain'] = "mydomain.com" gitlab_rails['smtp_authentication'] = "login" gitlab_rails['smtp_enable_starttls_auto'] = true
nginx['enable'] = true nginx['redirect_http_to_https'] = true nginx['ssl_certificate'] = "/etc/gitlab/ssl/git.mydomain.com.crt" nginx['ssl_certificate_key'] = "/etc/gitlab/ssl/git.mydomain.com.key"