Database Migration Failing When Upgrading Source Install
Summary
When attempting to upgrade the version for a gitlabe-ce source build, the database migration task will fail. The errors seem to indicate that the migration is trying to create tables and indexes that already exist in the database. For example, when attempting to upgrade from 8-11 to 8.12 I get the error:
Index name 'index_issue_metrics' on table 'issue_metrics' already exists/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:954:in `add_index_options'
I can work around the issue by manually adding the relevant value to SCHEMA_MIGRATIONS but I have to do this for multiple tables and indexes. In addition, when I go to do further upgrades (say from 8-12 to 8-13) I will have to go through the process all over again.
Steps to reproduce
Follow the upgrade instructions for the target version of gitlab. Run the
bundle exec rake db:migrate RAILS_ENV=production
task when instructed to do so.
What is the current bug behavior?
The migration fails with an error indicating that a table or index already exists
What is the expected correct behavior?
The migration task should handle cases when the table or index exists by either skipping that query or by dropping and recreating.
Relevant logs and/or screenshots
== 20160824103857 DropUnusedCiTables: migrating ===============================
-- drop_table(:ci_services)
-> 0.0158s
-- drop_table(:ci_web_hooks)
-> 0.0105s
== 20160824103857 DropUnusedCiTables: migrated (0.0265s) ======================
== 20160824124900 AddTableIssueMetrics: migrating =============================
-- create_table(:issue_metrics)
rake aborted!
StandardError: An error has occurred, all later migrations canceled:
Index name 'index_issue_metrics' on table 'issue_metrics' already exists/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:954:in `add_index_options'
/Users/git/gitlab/config/initializers/mysql_ignore_postgresql_options.rb:44:in `add_index_options'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:52:in `index_in_create'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `block in visit_TableDefinition'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `each'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `map'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `visit_TableDefinition'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_creation.rb:14:in `accept'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:222:in `create_table'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:474:in `create_table'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:665:in `block in method_missing'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:634:in `block in say_with_time'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:634:in `say_with_time'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:654:in `method_missing'
/Users/git/gitlab/db/migrate/20160824124900_add_table_issue_metrics.rb:27:in `change'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:608:in `exec_migration'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:591:in `block in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:590:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:768:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:1046:in `ddl_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:959:in `block in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:955:in `each'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:955:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:823:in `up'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:801:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/tasks/database_tasks.rb:137:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'
/Users/git/.rbenv/versions/2.3.1/bin/bundle:23:in `load'
/Users/git/.rbenv/versions/2.3.1/bin/bundle:23:in `<main>'
ArgumentError: Index name 'index_issue_metrics' on table 'issue_metrics' already exists
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:954:in `add_index_options'
/Users/git/gitlab/config/initializers/mysql_ignore_postgresql_options.rb:44:in `add_index_options'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:52:in `index_in_create'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `block in visit_TableDefinition'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `each'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `map'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `visit_TableDefinition'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_creation.rb:14:in `accept'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:222:in `create_table'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:474:in `create_table'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:665:in `block in method_missing'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:634:in `block in say_with_time'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:634:in `say_with_time'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:654:in `method_missing'
/Users/git/gitlab/db/migrate/20160824124900_add_table_issue_metrics.rb:27:in `change'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:608:in `exec_migration'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:591:in `block in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:590:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:768:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:1046:in `ddl_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:959:in `block in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:955:in `each'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:955:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:823:in `up'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:801:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/tasks/database_tasks.rb:137:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'
/Users/git/.rbenv/versions/2.3.1/bin/bundle:23:in `load'
/Users/git/.rbenv/versions/2.3.1/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Output of checks
(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Current User: git Using RVM: no Ruby Version: 2.3.1p112 Gem Version: 2.5.1 Bundler Version:2.0.2 Rake Version: 10.5.0 Sidekiq Version:4.1.4
GitLab information Version: 8.12.13 Revision: 06c1f945da6 Directory: /Users/git/gitlab DB Adapter: mysql2 URL: http://my.server.com/git HTTP Clone URL: http://my.server.com/git/some-group/some-project.git SSH Clone URL: git@my.server.com:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 3.6.1 Repository storage paths:
- default: /Users/git/repositories/ Hooks: /Users/git/gitlab-shell/hooks/ Git: /usr/local/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ... GitLab Shell version >= 3.6.1 ? ... OK (3.6.1) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 7/2 ... ok 7/5 ... ok 7/6 ... ok 7/7 ... ok 7/8 ... ok 7/10 ... ok 7/11 ... ok 7/12 ... ok 7/13 ... ok 7/14 ... ok 7/15 ... ok 7/16 ... ok 7/17 ... ok 7/19 ... ok 7/20 ... ok 7/21 ... ok 7/22 ... ok 1/26 ... ok 7/27 ... ok 5/28 ... ok 1/29 ... ok 1/30 ... ok 5/31 ... ok 5/32 ... ok 5/33 ... ok 6/36 ... ok 1/37 ... ok 4/40 ... ok 4/41 ... ok 4/42 ... ok 5/43 ... ok 5/44 ... ok 5/45 ... ok 5/46 ... ok 4/47 ... ok 5/48 ... ok 4/49 ... ok 4/50 ... ok 5/51 ... ok 5/52 ... ok 4/53 ... ok 5/54 ... ok 9/55 ... ok 5/56 ... ok 1/57 ... ok 24/58 ... ok 27/59 ... ok 5/60 ... ok 1/61 ... ok 19/63 ... ok 19/64 ... ok 19/65 ... ok 19/66 ... ok 19/67 ... ok 19/68 ... ok 19/69 ... ok 19/70 ... ok 19/71 ... ok 19/72 ... ok 19/73 ... ok 19/74 ... ok 19/75 ... ok 19/76 ... ok 19/77 ... ok 19/78 ... ok 19/79 ... ok 19/80 ... ok 19/81 ... ok 19/85 ... ok 19/86 ... ok 19/87 ... ok 19/88 ... ok 19/89 ... ok 4/92 ... ok 1/93 ... ok 1/94 ... ok 2/95 ... ok 27/96 ... ok 2/97 ... ok 25/98 ... ok 19/99 ... ok 19/100 ... ok 19/101 ... ok 19/102 ... ok 24/103 ... repository is empty 25/104 ... ok 2/105 ... ok 7/106 ... ok 27/107 ... ok 27/108 ... ok 27/109 ... ok 27/110 ... ok 19/111 ... ok 5/112 ... ok Running /Users/git/gitlab-shell/bin/check Check GitLab API access: OK Access to /Users/git/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful Checking GitLab Shell ... Finished Checking Sidekiq ... Running? ... yes Number of Sidekiq processes ... 1 Checking Sidekiq ... Finished Checking Reply by email ... Reply by email is disabled in config/gitlab.yml Checking Reply by email ... Finished Checking LDAP ... LDAP is disabled in config/gitlab.yml Checking LDAP ... Finished Checking GitLab ... Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... yes Init script exists? ... yes Init script up-to-date? ... yes projects have namespace: ... 7/2 ... yes 7/5 ... yes 7/6 ... yes 7/7 ... yes 7/8 ... yes 7/10 ... yes 7/11 ... yes 7/12 ... yes 7/13 ... yes 7/14 ... yes 7/15 ... yes 7/16 ... yes 7/17 ... yes 7/19 ... yes 7/20 ... yes 7/21 ... yes 7/22 ... yes 1/26 ... yes 7/27 ... yes 5/28 ... yes 1/29 ... yes 1/30 ... yes 5/31 ... yes 5/32 ... yes 5/33 ... yes 6/36 ... yes 1/37 ... yes 4/40 ... yes 4/41 ... yes 4/42 ... yes 5/43 ... yes 5/44 ... yes 5/45 ... yes 5/46 ... yes 4/47 ... yes 5/48 ... yes 4/49 ... yes 4/50 ... yes 5/51 ... yes 5/52 ... yes 4/53 ... yes 5/54 ... yes 9/55 ... yes 5/56 ... yes 1/57 ... yes 24/58 ... yes 27/59 ... yes 5/60 ... yes 1/61 ... yes 19/63 ... yes 19/64 ... yes 19/65 ... yes 19/66 ... yes 19/67 ... yes 19/68 ... yes 19/69 ... yes 19/70 ... yes 19/71 ... yes 19/72 ... yes 19/73 ... yes 19/74 ... yes 19/75 ... yes 19/76 ... yes 19/77 ... yes 19/78 ... yes 19/79 ... yes 19/80 ... yes 19/81 ... yes 19/85 ... yes 19/86 ... yes 19/87 ... yes 19/88 ... yes 19/89 ... yes 4/92 ... yes 1/93 ... yes 1/94 ... yes 2/95 ... yes 27/96 ... yes 2/97 ... yes 25/98 ... yes 19/99 ... yes 19/100 ... yes 19/101 ... yes 19/102 ... yes 24/103 ... yes 25/104 ... yes 2/105 ... yes 7/106 ... yes 27/107 ... yes 27/108 ... yes 27/109 ... yes 27/110 ... yes 19/111 ... yes 5/112 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.1) Your git bin path is "/usr/local/bin/git" Git version >= 2.7.3 ? ... yes (2.17.1) Active users: 8 Checking GitLab ... Finished
Possible fixes
The only fix I have found so far is to manually skip the failing migration stages, but the issue will come back the next time I try to upgrade.