rails_migration[gitlab-rails] action run fails from 14.8.2 to 14.10.0 (Ubuntu 20.04)
Summary
When upgrading from Self-Hosted CE 14.8.2 to 14.10.0, the database is not correctly migrated and the service does not start. Trying to reconfigure gets with the same error:
There was an error running gitlab-ctl reconfigure:
rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Steps to reproduce
On Ubuntu 20.04:
sudo apt update
sudo apt upgrade
After that, any other command will lead to the same:
sudo gitlab-ctl reconfigure
Also trying to upgrade just the database:
sudo gitlab-ctl pg-upgrade
What is the current bug behavior?
Gitlab CE is upgraded but database is not. The server cannot run or be reconfigured after that.
What is the expected correct behavior?
Gitlab CE and the database are upgraded and the server is running.
Relevant logs and/or screenshots
Recipe: gitlab::database_migrations
* ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
* rails_migration[gitlab-rails] action run
* bash[migrate gitlab-rails database] action run
================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Resource Declaration:
---------------------
suppressed sensitive resource output
Compiled Resource:
------------------
suppressed sensitive resource output
System Info:
------------
chef_version=15.17.4
platform=ubuntu
platform_version=20.04
ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
================================================================================
Error executing action `run` on resource 'rails_migration[gitlab-rails]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
51: rails_migration "gitlab-rails" do
52: rake_task 'gitlab:db:configure'
53: logfile_prefix 'gitlab-rails-db-migrate'
54: helper migration_helper
55:
56: environment env_variables
57: dependent_services dependent_services
58: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
59: notifies :run, "ruby_block[check remote PG version]", :immediately
60:
61: only_if { migration_helper.attributes_node['auto_migrate'] }
62: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
rails_migration("gitlab-rails") do
action [:run]
default_guard_interpreter :default
declared_type :rails_migration
cookbook_name "gitlab"
recipe_name "database_migrations"
rake_task "gitlab:db:configure"
logfile_prefix "gitlab-rails-db-migrate"
helper "*sensitive value suppressed*"
environment "*sensitive value suppressed*"
dependent_services []
only_if { #code block }
end
Output of checks
Results of GitLab environment info
gitlab-rake gitlab:env:info
Expand for output related to GitLab environment info
System information
System: Ubuntu 20.04
Current User: git
Using RVM: no
Ruby Version: 2.7.5p203
Gem Version: 3.1.4
Bundler Version:2.2.33
Rake Version: 13.0.6
Redis Version: 6.2.6
Sidekiq Version:6.4.0
Go Version: unknown
GitLab information
Version: 14.10.0
Revision: 88da5554d96
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: PostgreSQL
DB Version: 12.7
URL: https://gitlab-dev.autok8s.com
HTTP Clone URL: https://gitlab-dev.autok8s.com/some-group/some-project.git
SSH Clone URL: git@gitlab-dev.autok8s.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers:
GitLab Shell
Version: 13.25.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Possible fixes
The upgrade should be automated. The two failing files are:
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
on line 51
/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb
on line 16
Discussion with fixes in
Edited by Michael Friedrich