Skip to content
GitLab Next
Open
Created by Juli BCN@JuliBCN🎮

rails_migration[gitlab-rails] action run fails from 14.8.2 to 14.10.0 (Ubuntu 20.04)

Summary

When upgrading from Self-Hosted CE 14.8.2 to 14.10.0, the database is not correctly migrated and the service does not start. Trying to reconfigure gets with the same error:

There was an error running gitlab-ctl reconfigure:

rails_migration[gitlab-rails] (gitlab::database_migrations line 51) had an error: Mixlib::ShellOut::ShellCommandFailed: bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

Steps to reproduce

On Ubuntu 20.04:

sudo apt update
sudo apt upgrade

After that, any other command will lead to the same:

sudo gitlab-ctl reconfigure

Also trying to upgrade just the database:

sudo gitlab-ctl pg-upgrade

What is the current bug behavior?

Gitlab CE is upgraded but database is not. The server cannot run or be reconfigured after that.

What is the expected correct behavior?

Gitlab CE and the database are upgraded and the server is running.

Relevant logs and/or screenshots

Recipe: gitlab::database_migrations
  * ruby_block[check remote PG version] action nothing (skipped due to action :nothing)
  * rails_migration[gitlab-rails] action run
    * bash[migrate gitlab-rails database] action run

      ================================================================================
      Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
      ================================================================================

      Mixlib::ShellOut::ShellCommandFailed
      ------------------------------------
      Command execution failed. STDOUT/STDERR suppressed for sensitive resource

      Resource Declaration:
      ---------------------
      suppressed sensitive resource output

      Compiled Resource:
      ------------------
      suppressed sensitive resource output

      System Info:
      ------------
      chef_version=15.17.4
      platform=ubuntu
      platform_version=20.04
      ruby=ruby 2.7.5p203 (2021-11-24 revision f69aeb8314) [x86_64-linux]
      program_name=/opt/gitlab/embedded/bin/chef-client
      executable=/opt/gitlab/embedded/bin/chef-client


    ================================================================================
    Error executing action `run` on resource 'rails_migration[gitlab-rails]'
    ================================================================================

    Mixlib::ShellOut::ShellCommandFailed
    ------------------------------------
    bash[migrate gitlab-rails database] (/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb line 16) had an error: Mixlib::ShellOut::ShellCommandFailed: Command execution failed. STDOUT/STDERR suppressed for sensitive resource

    Resource Declaration:
    ---------------------
    # In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

     51: rails_migration "gitlab-rails" do
     52:   rake_task 'gitlab:db:configure'
     53:   logfile_prefix 'gitlab-rails-db-migrate'
     54:   helper migration_helper
     55:
     56:   environment env_variables
     57:   dependent_services dependent_services
     58:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
     59:   notifies :run, "ruby_block[check remote PG version]", :immediately
     60:
     61:   only_if { migration_helper.attributes_node['auto_migrate'] }
     62: end

    Compiled Resource:
    ------------------
    # Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'

    rails_migration("gitlab-rails") do
      action [:run]
      default_guard_interpreter :default
      declared_type :rails_migration
      cookbook_name "gitlab"
      recipe_name "database_migrations"
      rake_task "gitlab:db:configure"
      logfile_prefix "gitlab-rails-db-migrate"
      helper "*sensitive value suppressed*"
      environment "*sensitive value suppressed*"
      dependent_services []
      only_if { #code block }
    end

Output of checks

Results of GitLab environment info

gitlab-rake gitlab:env:info
Expand for output related to GitLab environment info 
System information
System:         Ubuntu 20.04
Current User:   git
Using RVM:      no
Ruby Version:   2.7.5p203
Gem Version:    3.1.4
Bundler Version:2.2.33
Rake Version:   13.0.6
Redis Version:  6.2.6
Sidekiq Version:6.4.0
Go Version:     unknown

GitLab information
Version:        14.10.0
Revision:       88da5554d96
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     12.7
URL:            https://gitlab-dev.autok8s.com
HTTP Clone URL: https://gitlab-dev.autok8s.com/some-group/some-project.git
SSH Clone URL:  git@gitlab-dev.autok8s.com:some-group/some-project.git
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers:

GitLab Shell
Version:        13.25.1
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell

(For installations with omnibus-gitlab package run and paste the output of:
`sudo gitlab-rake gitlab:env:info`)

(For installations from source run and paste the output of:
`sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)

Possible fixes

The upgrade should be automated. The two failing files are:

/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
on line 51

/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/resources/rails_migration.rb
on line 16

Discussion with fixes in

Edited by Michael Friedrich