Gitlab Upgrade is failing
Hi There,
Running into issue while upgrading to gitlab-ce-11.0.0
Gitlab reconfigure failing with following issue.
Recipe: gitlab::database_migrations
- bash[migrate gitlab-rails database] action run [execute] rake aborted! StandardError: An error has occurred, all later migrations canceled:
(<unknown>): control characters are not allowed at line 1 column 1
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in `replicate_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in `block in process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in `process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `block in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in `block (2 levels) in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in `block in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `steal'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Psych::SyntaxError: (<unknown>): control characters are not allowed at line 1 column 1
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in `replicate_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in `block in process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in `process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `block in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in `block (2 levels) in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in `block in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `steal'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20170830130119 StealRemainingEventMigrationJobs: migrating =================
================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20191021-33525-1xxvc9z" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
(<unknown>): control characters are not allowed at line 1 column 1
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in `replicate_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in `block in process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in `process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `block in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in `block (2 levels) in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in `block in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `steal'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
Psych::SyntaxError: (<unknown>): control characters are not allowed at line 1 column 1
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in `replicate_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in `block in process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in `process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `block in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in `block (2 levels) in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in `block in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `each'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in `steal'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20170830130119 StealRemainingEventMigrationJobs: migrating =================
STDERR:
---- End output of "bash" "/tmp/chef-script20191021-33525-1xxvc9z" ----
Ran "bash" "/tmp/chef-script20191021-33525-1xxvc9z" returned 1
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
49: bash "migrate gitlab-rails database" do
50: code <<-EOH
51: set -e
52: log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
53: umask 077
54: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
55: STATUS=${PIPESTATUS[0]}
56: echo $STATUS > #{db_migrate_status_file}
57: exit $STATUS
58: EOH
59: environment env_variables unless env_variables.empty?
60: notifies :run, "execute[clear the gitlab-rails cache]", :immediately
61: dependent_services.each do |svc|
62: notifies :restart, svc, :immediately
63: end
64: not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
65: only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
66: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:49:in `from_file'
bash("migrate gitlab-rails database") do
action [:run]
default_guard_interpreter :default
command nil
backup 5
returns 0
user nil
interpreter "bash"
declared_type :bash
cookbook_name "gitlab"
recipe_name "database_migrations"
code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-9f879dbff923f9680a162c461ef78ed2-b84bfb5\n exit $STATUS\n"
domain nil
not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-9f879dbff923f9680a162c461ef78ed2-b84bfb5) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-9f879dbff923f9680a162c461ef78ed2-b84bfb5 | grep -Fx 0)"
only_if { #code block }
end
System Info:
------------
chef_version=13.6.4
platform=redhat
platform_version=6.10
ruby=ruby 2.4.4p296 (2018-03-28 revision 63013) [x86_64-linux]
program_name=/opt/gitlab/embedded/bin/chef-client
executable=/opt/gitlab/embedded/bin/chef-client
Running handlers:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20191021-33525-1xxvc9z" ---- STDOUT: rake aborted! StandardError: An error has occurred, all later migrations canceled:
(): control characters are not allowed at line 1 column 1
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in replicate_event' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in
block in process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in process_event' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in
block in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in perform' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in
perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in block (2 levels) in steal' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in
block in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in each' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in
steal'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in
block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in
'
Caused by:
Psych::SyntaxError: (): control characters are not allowed at line 1 column 1
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:143:in replicate_event' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:132:in
block in process_event'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:131:in process_event' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in
block in perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration/migrate_events_to_push_event_payloads.rb:127:in perform' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:49:in
perform'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:29:in block (2 levels) in steal' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:22:in
block in steal'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in each' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:21:in
steal'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170830130119_steal_remaining_event_migration_jobs.rb:13:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in
block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in
'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20170830130119 StealRemainingEventMigrationJobs: migrating =================
STDERR:
---- End output of "bash" "/tmp/chef-script20191021-33525-1xxvc9z" ----
Ran "bash" "/tmp/chef-script20191021-33525-1xxvc9z" returned 1
Running handlers complete Chef Client failed. 4 resources updated in 43 seconds