Cannot upgrade from 10.7.4 : PG::UndefinedColumn: ERROR: column "jid" does not exist
Summary
Upgrade from 10.7.4 to 10.8.0 fail at "gitlab::database_migrations" because of a missing column.
Steps to reproduce
Upgrade from 10.7.4 to 10.8.0 (docker omnibus container). May or may not happen, one of my two GitLba instance showed this error, but not the other. The one which showed the bug ran for a long time (first started with a 8.x version), but not the one that did not showed this bug was first started with a much newer version (10.x)
What is the current bug behavior?
GitLab Upgrade fail at "gitlab::database_migrations".
What is the expected correct behavior?
GitLab should be upgraded to the 10.8.0 version.
Relevant logs and/or screenshots
24/05/2018 09:51:02Recipe: gitlab::database_migrations
24/05/2018 09:51:28 * bash[migrate gitlab-rails database] action run
24/05/2018 09:51:28 [execute] rake aborted!
24/05/2018 09:51:28 StandardError: An error has occurred, all later migrations canceled:
24/05/2018 09:51:28
24/05/2018 09:51:28 PG::UndefinedColumn: ERROR: column "jid" does not exist
24/05/2018 09:51:28 : CREATE INDEX CONCURRENTLY "index_project_mirror_data_on_jid" ON "project_mirror_data" ("jid" )
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `load'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `<main>'
24/05/2018 09:51:28
24/05/2018 09:51:28 Caused by:
24/05/2018 09:51:28 ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "jid" does not exist
24/05/2018 09:51:28 : CREATE INDEX CONCURRENTLY "index_project_mirror_data_on_jid" ON "project_mirror_data" ("jid" )
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `load'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `<main>'
24/05/2018 09:51:28
24/05/2018 09:51:28 Caused by:
24/05/2018 09:51:28 PG::UndefinedColumn: ERROR: column "jid" does not exist
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `load'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `<main>'
24/05/2018 09:51:28 Tasks: TOP => db:migrate
24/05/2018 09:51:28 (See full trace by running task with --trace)
24/05/2018 09:51:28 == 20180503175054 AddIndexesToProjectMirrorData: migrating ====================
24/05/2018 09:51:28 -- transaction_open?()
24/05/2018 09:51:28 -> 0.0000s
24/05/2018 09:51:28 -- execute("SET statement_timeout TO 0")
24/05/2018 09:51:28 -> 0.0003s
24/05/2018 09:51:28 -- index_exists?(:project_mirror_data, :jid, {:algorithm=>:concurrently})
24/05/2018 09:51:28 -> 0.0036s
24/05/2018 09:51:28 -- add_index(:project_mirror_data, :jid, {:algorithm=>:concurrently})
24/05/2018 09:51:28
24/05/2018 09:51:28 ================================================================================
24/05/2018 09:51:28 Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
24/05/2018 09:51:28 ================================================================================
24/05/2018 09:51:28
24/05/2018 09:51:28 Mixlib::ShellOut::ShellCommandFailed
24/05/2018 09:51:28 ------------------------------------
24/05/2018 09:51:28 Expected process to exit with [0], but received '1'
24/05/2018 09:51:28 ---- Begin output of "bash" "/tmp/chef-script20180524-54-pyt4tp" ----
24/05/2018 09:51:28 STDOUT: rake aborted!
24/05/2018 09:51:28 StandardError: An error has occurred, all later migrations canceled:
24/05/2018 09:51:28
24/05/2018 09:51:28 PG::UndefinedColumn: ERROR: column "jid" does not exist
24/05/2018 09:51:28 : CREATE INDEX CONCURRENTLY "index_project_mirror_data_on_jid" ON "project_mirror_data" ("jid" )
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `load'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `<main>'
24/05/2018 09:51:28
24/05/2018 09:51:28 Caused by:
24/05/2018 09:51:28 ActiveRecord::StatementInvalid: PG::UndefinedColumn: ERROR: column "jid" does not exist
24/05/2018 09:51:28 : CREATE INDEX CONCURRENTLY "index_project_mirror_data_on_jid" ON "project_mirror_data" ("jid" )
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `load'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `<main>'
24/05/2018 09:51:28
24/05/2018 09:51:28 Caused by:
24/05/2018 09:51:28 PG::UndefinedColumn: ERROR: column "jid" does not exist
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_opclasses_support.rb:142:in `add_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:69:in `add_concurrent_index'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180503175054_add_indexes_to_project_mirror_data.rb:9:in `up'
24/05/2018 09:51:28 /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `load'
24/05/2018 09:51:28 /opt/gitlab/embedded/bin/bundle:23:in `<main>'
24/05/2018 09:51:28 Tasks: TOP => db:migrate
24/05/2018 09:51:28 (See full trace by running task with --trace)
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
# gitlab-rake gitlab:env:infoSystem information System: Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.7.4 Revision: ca648fc Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://lab.dremor.info HTTP Clone URL: https://lab.dremor.info/some-group/some-project.git SSH Clone URL: git@lab.dremor.info:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 7.1.2 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
# gitlab-rake gitlab:check SANITIZE=true Checking GitLab Shell ...GitLab Shell version >= 7.1.2 ? ... OK (7.1.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 2/1 ... ok 1/2 ... ok 1/3 ... ok 1/4 ... repository is empty 1/5 ... ok 10/6 ... ok 1/8 ... ok 1/9 ... ok 1/10 ... ok 1/11 ... ok 1/12 ... ok 11/13 ... ok 11/14 ... ok 11/15 ... repository is empty 11/16 ... ok 1/17 ... ok 1/18 ... ok 11/19 ... ok 1/20 ... ok 10/21 ... ok 1/22 ... ok 1/23 ... repository is empty 11/24 ... ok 18/25 ... ok 18/26 ... ok 18/27 ... ok 18/28 ... repository is empty 18/29 ... ok 18/30 ... repository is empty 21/31 ... ok 21/32 ... ok 1/33 ... ok 23/34 ... ok 1/35 ... ok 23/36 ... ok 1/37 ... ok 10/38 ... ok 1/39 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 2/1 ... yes 1/2 ... yes 1/3 ... yes 1/4 ... yes 1/5 ... yes 10/6 ... yes 1/8 ... yes 1/9 ... yes 1/10 ... yes 1/11 ... yes 1/12 ... yes 11/13 ... yes 11/14 ... yes 11/15 ... yes 11/16 ... yes 1/17 ... yes 1/18 ... yes 11/19 ... yes 1/20 ... yes 10/21 ... yes 1/22 ... yes 1/23 ... yes 11/24 ... yes 18/25 ... yes 18/26 ... yes 18/27 ... yes 18/28 ... yes 18/29 ... yes 18/30 ... yes 21/31 ... yes 21/32 ... yes 1/33 ... yes 23/34 ... yes 1/35 ... yes 23/36 ... yes 1/37 ... yes 10/38 ... yes 1/39 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.9.5 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 1
Checking GitLab ... Finished