Node JSON file got corrupted in upgrade
Summary
Related to issue gitlab-ce#46729:
Running version 10.8.0, I am unable to run gitlab-ctl reconfigure
because the migration step fails with the message:
PG::UndefinedColumn: ERROR: column "jid" does not exist
Further, I am unable to upgrade to 10.8.1 because the installation step fails:
Malformed configuration JSON file at /opt/gitlab/embedded/nodes/icess6a.rvx.is.json.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
A veritable catch 22.
Steps to reproduce
The details of how I got into this picle are not quite clear. Two days ago I upgraded to 10.8.0 seemingly without problems. Only when I wanted to change my gitlab.rb file did I run into the issue.
What is the current bug behavior?
- Unable to
gitlab-ctl reconfigure
because of problems in a migration script (see issue gitlab-ce#46729) - Unable to update from version 10.8.0 to 10.8.1 because
gitlab-ctl reconfigure
has not been run.
What is the expected correct behavior?
-
gitlab-ctl reconfigure
works. - I can upgrade to 10.8.1
Relevant logs and/or screenshots
Full report of failed migration from gitlab-ctl reconfigure
:
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20180524-15553-119krqq" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedColumn: ERROR: column "jid" does not exist
: CREATE INDEX CONCURRENTLY "index_project_mirror_data_on_jid" ON "project_mirror_data" ("jid" )
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "jid" does not exist
: CREATE INDEX CONCURRENTLY "index_project_mirror_data_on_jid" ON "project_mirror_data" ("jid" )
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::UndefinedColumn: ERROR: column "jid" does not exist
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20180503175054 AddIndexesToProjectMirrorData: migrating ====================
-- transaction_open?()
-> 0.0000s
-- execute("SET statement_timeout TO 0")
-> 0.0003s
-- index_exists?(:project_mirror_data, :jid, {:algorithm=>:concurrently})
-> 0.0040s
-- add_index(:project_mirror_data, :jid, {:algorithm=>:concurrently})
STDERR:
---- End output of "bash" "/tmp/chef-script20180524-15553-119krqq" ----
Ran "bash" "/tmp/chef-script20180524-15553-119krqq" returned 1
Also, when doing yum update gitlab-ce
I get
Downloading Packages:
gitlab-ce-10.8.1-ce.0.el6.x86_64.rpm | 358 MB 00:08
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Malformed configuration JSON file at /opt/gitlab/embedded/nodes/icess6a.rvx.is.json.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
error: %pre(gitlab-ce-10.8.1-ce.0.el6.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-10.8.1-ce.0.el6.x86_64
error: install: %pre scriptlet failed (2), skipping gitlab-ce-10.8.1-ce.0.el6
Verifying : gitlab-ce-10.8.1-ce.0.el6.x86_64 1/2
gitlab-ce-10.8.0-ce.0.el6.x86_64 was supposed to be removed but is not!
Verifying : gitlab-ce-10.8.0-ce.0.el6.x86_64 2/2
Failed:
gitlab-ce.x86_64 0:10.8.0-ce.0.el6 gitlab-ce.x86_64 0:10.8.1-ce.0.el6
Complete!
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: CentOS 6.9 Current User: git Using RVM: no Ruby Version: 2.3.7p456 Gem Version: 2.6.14 Bundler Version:1.13.7 Rake Version: 12.3.1 Redis Version: 3.2.11 Git Version: 2.16.3 Sidekiq Version:5.0.5 Go Version: unknownGitLab information Version: 10.8.0 Revision: 55e4a0b Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: http://gitlab.rvx.is HTTP Clone URL: http://gitlab.rvx.is/some-group/some-project.git SSH Clone URL: git@gitlab.rvx.is:some-group/some-project.git Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 7.1.2 Repository storage paths:
- default: /mnt/raid1/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 7.1.2 ? ... OK (7.1.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 1/1 ... ok 12/10 ... repository is empty 13/11 ... ok 14/14 ... ok 20/15 ... ok 13/16 ... ok 10/17 ... ok 10/18 ... ok 10/19 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... no Exception: undefined method `message' for #String:0x00007f0914218c60 Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain not verifying SSL hostname of LDAPS server 'dc01.rvx.is:636' LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) DN: (removed) Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 1/1 ... yes 12/10 ... yes 13/11 ... yes 14/14 ... yes 20/15 ... yes 13/16 ... yes 10/17 ... yes 10/18 ... yes 10/19 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.7) Git version >= 2.9.5 ? ... yes (2.16.3) Git user has default SSH configuration? ... yes Active users: ... 14
Checking GitLab ... Finished