Skip to content

Node JSON file got corrupted in upgrade

Summary

Related to issue gitlab-ce#46729: Running version 10.8.0, I am unable to run gitlab-ctl reconfigure because the migration step fails with the message:

PG::UndefinedColumn: ERROR:  column "jid" does not exist

Further, I am unable to upgrade to 10.8.1 because the installation step fails:

Malformed configuration JSON file at /opt/gitlab/embedded/nodes/icess6a.rvx.is.json.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.

A veritable catch 22.

Steps to reproduce

The details of how I got into this picle are not quite clear. Two days ago I upgraded to 10.8.0 seemingly without problems. Only when I wanted to change my gitlab.rb file did I run into the issue.

What is the current bug behavior?

  • Unable to gitlab-ctl reconfigure because of problems in a migration script (see issue gitlab-ce#46729)
  • Unable to update from version 10.8.0 to 10.8.1 because gitlab-ctl reconfigure has not been run.

What is the expected correct behavior?

  • gitlab-ctl reconfigure works.
  • I can upgrade to 10.8.1

Relevant logs and/or screenshots

Full report of failed migration from gitlab-ctl reconfigure:

There was an error running gitlab-ctl reconfigure:

bash[migrate gitlab-rails database] (gitlab::database_migrations line 49) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash"  "/tmp/chef-script20180524-15553-119krqq" ----
STDOUT: rake aborted!
StandardError: An error has occurred, all later migrations canceled:

PG::UndefinedColumn: ERROR:  column "jid" does not exist
: CREATE  INDEX CONCURRENTLY "index_project_mirror_data_on_jid" ON "project_mirror_data"  ("jid" )
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR:  column "jid" does not exist
: CREATE  INDEX CONCURRENTLY "index_project_mirror_data_on_jid" ON "project_mirror_data"  ("jid" )
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Caused by:
PG::UndefinedColumn: ERROR:  column "jid" does not exist
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20180503175054 AddIndexesToProjectMirrorData: migrating ====================
-- transaction_open?()
   -> 0.0000s
-- execute("SET statement_timeout TO 0")
   -> 0.0003s
-- index_exists?(:project_mirror_data, :jid, {:algorithm=>:concurrently})
   -> 0.0040s
-- add_index(:project_mirror_data, :jid, {:algorithm=>:concurrently})
STDERR:
---- End output of "bash"  "/tmp/chef-script20180524-15553-119krqq" ----
Ran "bash"  "/tmp/chef-script20180524-15553-119krqq" returned 1

Also, when doing yum update gitlab-ce I get

Downloading Packages:
gitlab-ce-10.8.1-ce.0.el6.x86_64.rpm                                                                                                                                      | 358 MB     00:08
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
Malformed configuration JSON file at /opt/gitlab/embedded/nodes/icess6a.rvx.is.json.
Please run `sudo gitlab-ctl reconfigure` to fix it and try again.
error: %pre(gitlab-ce-10.8.1-ce.0.el6.x86_64) scriptlet failed, exit status 1
Error in PREIN scriptlet in rpm package gitlab-ce-10.8.1-ce.0.el6.x86_64
error:   install: %pre scriptlet failed (2), skipping gitlab-ce-10.8.1-ce.0.el6
  Verifying  : gitlab-ce-10.8.1-ce.0.el6.x86_64                                                                                                                                              1/2
gitlab-ce-10.8.0-ce.0.el6.x86_64 was supposed to be removed but is not!
  Verifying  : gitlab-ce-10.8.0-ce.0.el6.x86_64                                                                                                                                              2/2

Failed:
  gitlab-ce.x86_64 0:10.8.0-ce.0.el6                                                              gitlab-ce.x86_64 0:10.8.1-ce.0.el6

Complete!

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:         CentOS 6.9
Current User:   git
Using RVM:      no
Ruby Version:   2.3.7p456
Gem Version:    2.6.14
Bundler Version:1.13.7
Rake Version:   12.3.1
Redis Version:  3.2.11
Git Version:    2.16.3
Sidekiq Version:5.0.5
Go Version:     unknown

GitLab information
Version:        10.8.0
Revision:       55e4a0b
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            http://gitlab.rvx.is
HTTP Clone URL: http://gitlab.rvx.is/some-group/some-project.git
SSH Clone URL:  git@gitlab.rvx.is:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no


GitLab Shell
Version:        7.1.2
Repository storage paths:
  • default: /mnt/raid1/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab Shell ...

GitLab Shell version >= 7.1.2 ? ... OK (7.1.2)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
1/1 ... ok
12/10 ... repository is empty
13/11 ... ok
14/14 ... ok
20/15 ... ok
13/16 ... ok
10/17 ... ok
10/18 ... ok
10/19 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Reply by email ...


IMAP server credentials are correct? ... no
Exception: undefined method `message' for #String:0x00007f0914218c60
Init.d configured correctly? ... skipped
MailRoom running? ... skipped


Checking Reply by email ... Finished


Checking LDAP ...


Server: ldapmain
not verifying SSL hostname of LDAPS server 'dc01.rvx.is:636'
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
DN: (removed)
Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
1/1 ... yes
12/10 ... yes
13/11 ... yes
14/14 ... yes
20/15 ... yes
13/16 ... yes
10/17 ... yes
10/18 ... yes
10/19 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.3.7)
Git version >= 2.9.5 ? ... yes (2.16.3)
Git user has default SSH configuration? ... yes
Active users: ... 14


Checking GitLab ... Finished
Edited by Stan Hu