DB migration fails when upgrading from 9.4.5 to 9.5.0
Summary
Upgrading from GitLab CE 9.4.5 to 9.5.0 fails while performing migrations. Apparent reason: PG::ConnectionBad: FATAL: the database system is starting up
.
Steps to reproduce
Start up docker container with a configured 9.4.5, pull new image and restart.
Example Project
N/A
What is the current bug behavior?
gitlab_1 | ================================================================================
gitlab_1 | Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
gitlab_1 | ================================================================================
gitlab_1 |
gitlab_1 | Mixlib::ShellOut::ShellCommandFailed
gitlab_1 | ------------------------------------
gitlab_1 | Expected process to exit with [0], but received '1'
gitlab_1 | ---- Begin output of "bash" "/tmp/chef-script20170823-33-1youmpx" ----
gitlab_1 | STDOUT: rake aborted!
gitlab_1 | PG::ConnectionBad: FATAL: the database system is starting up
gitlab_1 | /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:22:in `load'
gitlab_1 | /opt/gitlab/embedded/bin/bundle:22:in `<main>'
gitlab_1 | Tasks: TOP => gitlab:db:configure
gitlab_1 | (See full trace by running task with --trace)
gitlab_1 | STDERR:
gitlab_1 | ---- End output of "bash" "/tmp/chef-script20170823-33-1youmpx" ----
gitlab_1 | Ran "bash" "/tmp/chef-script20170823-33-1youmpx" returned 1
gitlab_1 |
gitlab_1 | Resource Declaration:
gitlab_1 | ---------------------
gitlab_1 | # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
gitlab_1 |
gitlab_1 | 51: bash "migrate gitlab-rails database" do
gitlab_1 | 52: code <<-EOH
gitlab_1 | 53: set -e
gitlab_1 | 54: log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
gitlab_1 | 55: umask 077
gitlab_1 | 56: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
gitlab_1 | 57: STATUS=${PIPESTATUS[0]}
gitlab_1 | 58: echo $STATUS > #{db_migrate_status_file}
gitlab_1 | 59: exit $STATUS
gitlab_1 | 60: EOH
gitlab_1 | 61: environment env_variables unless env_variables.empty?
gitlab_1 | 62: notifies :run, 'execute[enable pg_trgm extension]', :before if omnibus_helper.service_enabled?('postgresql')
gitlab_1 | 63: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
gitlab_1 | 64: dependent_services.each do |svc|
gitlab_1 | 65: notifies :restart, svc, :immediately
gitlab_1 | 66: end
gitlab_1 | 67: not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
gitlab_1 | 68: only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
gitlab_1 | 69: end
gitlab_1 |
gitlab_1 | Compiled Resource:
gitlab_1 | ------------------
gitlab_1 | # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
gitlab_1 |
gitlab_1 | bash("migrate gitlab-rails database") do
gitlab_1 | action [:run]
gitlab_1 | updated true
gitlab_1 | retries 0
gitlab_1 | retry_delay 2
gitlab_1 | default_guard_interpreter :default
gitlab_1 | command "migrate gitlab-rails database"
gitlab_1 | backup 5
gitlab_1 | returns 0
gitlab_1 | code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-77bfdac\n exit $STATUS\n"
gitlab_1 | interpreter "bash"
gitlab_1 | declared_type :bash
gitlab_1 | cookbook_name "gitlab"
gitlab_1 | recipe_name "database_migrations"
gitlab_1 | not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-77bfdac) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-77bfdac | grep -Fx 0)"
gitlab_1 | only_if { #code block }
gitlab_1 | end
What is the expected correct behavior?
docker ps
-> healthy
Relevant logs and/or screenshots
see above
Output of checks
N/A
Results of GitLab environment info
on 9.4.5, cannot reproduce on 9.5.0 since container isn't coming up
root@gitlab:/# gitlab-rake gitlab:env:info RAILS_ENV=production
System information
System:
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.7
Rake Version: 10.5.0
Redis Version: 3.2.5
Git Version: 2.13.4
Sidekiq Version:5.0.0
Go Version: unknown
GitLab information
Version: 9.4.5
Revision: 140292e
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://xxx
HTTP Clone URL: https://xxx/some-group/some-project.git
SSH Clone URL: git@xxx:some-group/some-project.git
Using LDAP: no
Using Omniauth: no
GitLab Shell
Version: 5.3.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
N/A
Possible fixes
N/A
Edited by lusitania