Gitlab Upgrade is failing

Hi There,

Running into issue while upgrading to gitlab-ce-11.0.0

Gitlab reconfigure failing with following issue.

Recipe: gitlab::database_migrations

bash[migrate gitlab-rails database] action run [execute] rake aborted! StandardError: An error has occurred, all later migrations canceled:

          (<unknown>): control characters are not allowed at line 1 column 1
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in `replicate_event'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in `block in process_event'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in `process_event'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `block in perform'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `perform'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in `perform'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in `block (2 levels) in steal'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in `block in steal'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `each'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `steal'
          /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'

          Caused by:
          Psych::SyntaxError: (<unknown>): control characters are not allowed at line 1 column 1
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in `replicate_event'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in `block in process_event'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in `process_event'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `block in perform'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `perform'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in `perform'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in `block (2 levels) in steal'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in `block in steal'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `each'
          /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `steal'
          /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in `up'
          /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
          /opt/gitlab/embedded/bin/bundle:23:in `load'
          /opt/gitlab/embedded/bin/bundle:23:in `<main>'
          Tasks: TOP => db:migrate
          (See full trace by running task with --trace)
          == 20170830130119 StealRemainingEventMigrationJobs: migrating =================
================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================

Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20191021-33525-1xxvc9z" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

(<unknown>): control characters are not allowed at line 1 column 1
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in `replicate_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in `block in process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in `process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `block in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in `block (2 levels) in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in `block in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `steal'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
Psych::SyntaxError: (<unknown>): control characters are not allowed at line 1 column 1
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in `replicate_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in `block in process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in `process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `block in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in `block (2 levels) in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in `block in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `steal'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20170830130119 StealRemainingEventMigrationJobs: migrating =================
STDERR:
---- End output of "bash"  "/tmp/chef-script20191021-33525-1xxvc9z" ----
Ran "bash"  "/tmp/chef-script20191021-33525-1xxvc9z" returned 1

Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

 49: bash "migrate gitlab-rails database" do
 50:   code <<-EOH
 51:     set -e
 52:     log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
 53:     umask 077
 54:     /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
 55:     STATUS=${PIPESTATUS[0]}
 56:     echo $STATUS > #{db_migrate_status_file}
 57:     exit $STATUS
 58:   EOH
 59:   environment env_variables unless env_variables.empty?
 60:   notifies :run, "execute[clear the gitlab-rails cache]", :immediately
 61:   dependent_services.each do |svc|
 62:     notifies :restart, svc, :immediately
 63:   end
 64:   not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
 65:   only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
 66: end

Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:49:in `from_file'

bash("migrate gitlab-rails database") do
  action [:run]
  default_guard_interpreter :default
  command nil
  backup 5
  returns 0
  user nil
  interpreter "bash"
  declared_type :bash
  cookbook_name "gitlab"
  recipe_name "database_migrations"
  code "    set -e\n    log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n    umask 077\n    /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n    STATUS=${PIPESTATUS[0]}\n    echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-9f879dbff923f9680a162c461ef78ed2-b84bfb5\n    exit $STATUS\n"
  domain nil
  not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-9f879dbff923f9680a162c461ef78ed2-b84bfb5) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-9f879dbff923f9680a162c461ef78ed2-b84bfb5 | grep -Fx 0)"
  only_if { #code block }
end

System Info:
------------
chef_version=13.6.4
platform=redhat
platform_version=6.10
ruby=ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client


  Running handlers:
  There was an error running gitlab-ctl reconfigure:

  bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'

---- Begin output of "bash" "/tmp/chef-script20191021-33525-1xxvc9z" ---- STDOUT: rake aborted! StandardError: An error has occurred, all later migrations canceled:

(): control characters are not allowed at line 1 column 1 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in replicate_event' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in block in process_event' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in process_event' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in block in perform' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in perform' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in perform' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in block (2 levels) in steal' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in block in steal' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in each' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in steal' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in '

Caused by: Psych::SyntaxError: (): control characters are not allowed at line 1 column 1 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in replicate_event' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in block in process_event' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in process_event' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in block in perform' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in perform' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in perform' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in block (2 levels) in steal' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in block in steal' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in each' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in steal' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ' Tasks: TOP => db:migrate (See full trace by running task with --trace) == 20170830130119 StealRemainingEventMigrationJobs: migrating ================= STDERR: ---- End output of "bash" "/tmp/chef-script20191021-33525-1xxvc9z" ---- Ran "bash" "/tmp/chef-script20191021-33525-1xxvc9z" returned 1

Running handlers complete Chef Client failed. 4 resources updated in 43 seconds

Edited Oct 21, 2019 by Mani