Node JSON file got corrupted in upgrade
<!---
Please read this!
Before opening a new issue, make sure to search for keywords in the issues
filtered by the "regression" or "bug" label.
For the Community Edition issue tracker:
- https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name%5B%5D=regression
- https://gitlab.com/gitlab-org/gitlab-ce/issues?label_name%5B%5D=bug
For the Enterprise Edition issue tracker:
- https://gitlab.com/gitlab-org/gitlab-ee/issues?label_name%5B%5D=regression
- https://gitlab.com/gitlab-org/gitlab-ee/issues?label_name%5B%5D=bug
and verify the issue you're about to submit isn't a duplicate.
--->
### Summary
Related to issue gitlab-ce#46729:
Running version 10.8.0, I am unable to run `gitlab-ctl reconfigure` because the migration step fails with the message:
```
PG::UndefinedColumn: ERROR: column "jid" does not exist
```
Further, I am unable to upgrade to 10.8.1 because the installation step fails:
```
Malformed configuration JSON file at /opt/gitlab/embedded/nodes/icess6a.rvx.is.json.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
```
A veritable catch 22.
### Steps to reproduce
The details of how I got into this picle are not quite clear. Two days ago I upgraded to 10.8.0 seemingly without problems. Only when I wanted to change my gitlab.rb file did I run into the issue.
### What is the current *bug* behavior?
- Unable to `gitlab-ctl reconfigure` because of problems in a migration script (see issue gitlab-ce#46729)
- Unable to update from version 10.8.0 to 10.8.1 because `gitlab-ctl reconfigure` has not been run.
### What is the expected *correct* behavior?
- `gitlab-ctl reconfigure` works.
- I can upgrade to 10.8.1
### Relevant logs and/or screenshots
Full report of failed migration from `gitlab-ctl reconfigure`:
```
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20180524-15553-119krqq" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:
PG::UndefinedColumn: ERROR: column "jid" does not exist
: CREATE INDEX CONCURRENTLY "index_project_mirror_data_on_jid" ON "project_mirror_data" ("jid" )
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "jid" does not exist
: CREATE INDEX CONCURRENTLY "index_project_mirror_data_on_jid" ON "project_mirror_data" ("jid" )
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
PG::UndefinedColumn: ERROR: column "jid" does not exist
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20180503175054 AddIndexesToProjectMirrorData: migrating ====================
-- transaction_open?()
-> 0.0000s
-- execute("SET statement_timeout TO 0")
-> 0.0003s
-- index_exists?(:project_mirror_data, :jid, {:algorithm=>:concurrently})
-> 0.0040s
-- add_index(:project_mirror_data, :jid, {:algorithm=>:concurrently})
STDERR:
---- End output of "bash" "/tmp/chef-script20180524-15553-119krqq" ----
Ran "bash" "/tmp/chef-script20180524-15553-119krqq" returned 1
```
Also, when doing `yum update gitlab-ce` I get
```
Downloading Packages:
gitlab-ce-10.8.1-ce.0.el6.x86_64.rpm | 358 MB 00:08
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Malformed configuration JSON file at /opt/gitlab/embedded/nodes/icess6a.rvx.is.json.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
error: %pre(gitlab-ce-10.8.1-ce.0.el6.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-10.8.1-ce.0.el6.x86_64
error: install: %pre scriptlet failed (2), skipping gitlab-ce-10.8.1-ce.0.el6
Verifying : gitlab-ce-10.8.1-ce.0.el6.x86_64 1/2
gitlab-ce-10.8.0-ce.0.el6.x86_64 was supposed to be removed but is not!
Verifying : gitlab-ce-10.8.0-ce.0.el6.x86_64 2/2
Failed:
gitlab-ce.x86_64 0:10.8.0-ce.0.el6 gitlab-ce.x86_64 0:10.8.1-ce.0.el6
Complete!
```
#### Results of GitLab environment info
<details>
<summary>Expand for output related to GitLab environment info</summary>
<pre>
System information
System: CentOS 6.9
Current User: git
Using RVM: no
Ruby Version: 2.3.7p456
Gem Version: 2.6.14
Bundler Version:1.13.7
Rake Version: 12.3.1
Redis Version: 3.2.11
Git Version: 2.16.3
Sidekiq Version:5.0.5
Go Version: unknown
GitLab information
Version: 10.8.0
Revision: 55e4a0b
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: http://gitlab.rvx.is
HTTP Clone URL: http://gitlab.rvx.is/some-group/some-project.git
SSH Clone URL: git@gitlab.rvx.is:some-group/some-project.git
Using LDAP: yes
Using Omniauth: no
GitLab Shell
Version: 7.1.2
Repository storage paths:
- default: /mnt/raid1/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks
Git: /opt/gitlab/embedded/bin/git
</pre>
</details>
#### Results of GitLab application Check
<details>
<summary>Expand for output related to the GitLab application check</summary>
<pre>
Checking GitLab Shell ...
GitLab Shell version >= 7.1.2 ? ... OK (7.1.2)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
1/1 ... ok
12/10 ... repository is empty
13/11 ... ok
14/14 ... ok
20/15 ... ok
13/16 ... ok
10/17 ... ok
10/18 ... ok
10/19 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... no
Exception: undefined method `message' for #<String:0x00007f0914218c60>
Init.d configured correctly? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain
not verifying SSL hostname of LDAPS server 'dc01.rvx.is:636'
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
DN: (removed)
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/1 ... yes
12/10 ... yes
13/11 ... yes
14/14 ... yes
20/15 ... yes
13/16 ... yes
10/17 ... yes
10/18 ... yes
10/19 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.7)
Git version >= 2.9.5 ? ... yes (2.16.3)
Git user has default SSH configuration? ... yes
Active users: ... 14
Checking GitLab ... Finished
</pre>
</details>
issue