Database Migration Failing When Upgrading Source Install

Summary

When attempting to upgrade the version for a gitlabe-ce source build, the database migration task will fail. The errors seem to indicate that the migration is trying to create tables and indexes that already exist in the database. For example, when attempting to upgrade from 8-11 to 8.12 I get the error:

Index name 'index_issue_metrics' on table 'issue_metrics' already exists/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:954:in `add_index_options'

I can work around the issue by manually adding the relevant value to SCHEMA_MIGRATIONS but I have to do this for multiple tables and indexes. In addition, when I go to do further upgrades (say from 8-12 to 8-13) I will have to go through the process all over again.

Steps to reproduce

Follow the upgrade instructions for the target version of gitlab. Run the

bundle exec rake db:migrate RAILS_ENV=production

task when instructed to do so.

What is the current bug behavior?

The migration fails with an error indicating that a table or index already exists

What is the expected correct behavior?

The migration task should handle cases when the table or index exists by either skipping that query or by dropping and recreating.

Relevant logs and/or screenshots

== 20160824103857 DropUnusedCiTables: migrating ===============================
-- drop_table(:ci_services)
   -> 0.0158s
-- drop_table(:ci_web_hooks)
   -> 0.0105s
== 20160824103857 DropUnusedCiTables: migrated (0.0265s) ======================

== 20160824124900 AddTableIssueMetrics: migrating =============================
-- create_table(:issue_metrics)
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

Index name 'index_issue_metrics' on table 'issue_metrics' already exists/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:954:in `add_index_options'
/Users/git/gitlab/config/initializers/mysql_ignore_postgresql_options.rb:44:in `add_index_options'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:52:in `index_in_create'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `block in visit_TableDefinition'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `each'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `map'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `visit_TableDefinition'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_creation.rb:14:in `accept'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:222:in `create_table'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:474:in `create_table'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:665:in `block in method_missing'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:634:in `block in say_with_time'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:634:in `say_with_time'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:654:in `method_missing'
/Users/git/gitlab/db/migrate/20160824124900_add_table_issue_metrics.rb:27:in `change'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:608:in `exec_migration'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:591:in `block in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:590:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:768:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:1046:in `ddl_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:959:in `block in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:955:in `each'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:955:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:823:in `up'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:801:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/tasks/database_tasks.rb:137:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'
/Users/git/.rbenv/versions/2.3.1/bin/bundle:23:in `load'
/Users/git/.rbenv/versions/2.3.1/bin/bundle:23:in `<main>'
ArgumentError: Index name 'index_issue_metrics' on table 'issue_metrics' already exists
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:954:in `add_index_options'
/Users/git/gitlab/config/initializers/mysql_ignore_postgresql_options.rb:44:in `add_index_options'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:52:in `index_in_create'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `block in visit_TableDefinition'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `each'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `map'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:25:in `visit_TableDefinition'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_creation.rb:14:in `accept'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/schema_statements.rb:222:in `create_table'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract_mysql_adapter.rb:474:in `create_table'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:665:in `block in method_missing'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:634:in `block in say_with_time'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:634:in `say_with_time'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:654:in `method_missing'
/Users/git/gitlab/db/migrate/20160824124900_add_table_issue_metrics.rb:27:in `change'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:608:in `exec_migration'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:592:in `block (2 levels) in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:591:in `block in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/connection_adapters/abstract/connection_pool.rb:292:in `with_connection'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:590:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:768:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:998:in `block in execute_migration_in_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:1046:in `ddl_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:997:in `execute_migration_in_transaction'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:959:in `block in migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:955:in `each'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:955:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:823:in `up'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/migration.rb:801:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/tasks/database_tasks.rb:137:in `migrate'
/Users/git/gitlab/vendor/bundle/ruby/2.3.0/gems/activerecord-4.2.7.1/lib/active_record/railties/databases.rake:44:in `block (2 levels) in <top (required)>'
/Users/git/.rbenv/versions/2.3.1/bin/bundle:23:in `load'
/Users/git/.rbenv/versions/2.3.1/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:
Current User:	git
Using RVM:	no
Ruby Version:	2.3.1p112
Gem Version:	2.5.1
Bundler Version:2.0.2
Rake Version:	10.5.0
Sidekiq Version:4.1.4
GitLab information
Version:	8.12.13
Revision:	06c1f945da6
Directory:	/Users/git/gitlab
DB Adapter:	mysql2
URL:		http://my.server.com/git
HTTP Clone URL:	http://my.server.com/git/some-group/some-project.git
SSH Clone URL:	git@my.server.com:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no
GitLab Shell
Version:	3.6.1
Repository storage paths:

• default: /Users/git/repositories/ Hooks: /Users/git/gitlab-shell/hooks/ Git: /usr/local/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab Shell ...

GitLab Shell version >= 3.6.1 ? ... OK (3.6.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ... 
7/2 ... ok
7/5 ... ok
7/6 ... ok
7/7 ... ok
7/8 ... ok
7/10 ... ok
7/11 ... ok
7/12 ... ok
7/13 ... ok
7/14 ... ok
7/15 ... ok
7/16 ... ok
7/17 ... ok
7/19 ... ok
7/20 ... ok
7/21 ... ok
7/22 ... ok
1/26 ... ok
7/27 ... ok
5/28 ... ok
1/29 ... ok
1/30 ... ok
5/31 ... ok
5/32 ... ok
5/33 ... ok
6/36 ... ok
1/37 ... ok
4/40 ... ok
4/41 ... ok
4/42 ... ok
5/43 ... ok
5/44 ... ok
5/45 ... ok
5/46 ... ok
4/47 ... ok
5/48 ... ok
4/49 ... ok
4/50 ... ok
5/51 ... ok
5/52 ... ok
4/53 ... ok
5/54 ... ok
9/55 ... ok
5/56 ... ok
1/57 ... ok
24/58 ... ok
27/59 ... ok
5/60 ... ok
1/61 ... ok
19/63 ... ok
19/64 ... ok
19/65 ... ok
19/66 ... ok
19/67 ... ok
19/68 ... ok
19/69 ... ok
19/70 ... ok
19/71 ... ok
19/72 ... ok
19/73 ... ok
19/74 ... ok
19/75 ... ok
19/76 ... ok
19/77 ... ok
19/78 ... ok
19/79 ... ok
19/80 ... ok
19/81 ... ok
19/85 ... ok
19/86 ... ok
19/87 ... ok
19/88 ... ok
19/89 ... ok
4/92 ... ok
1/93 ... ok
1/94 ... ok
2/95 ... ok
27/96 ... ok
2/97 ... ok
25/98 ... ok
19/99 ... ok
19/100 ... ok
19/101 ... ok
19/102 ... ok
24/103 ... repository is empty
25/104 ... ok
2/105 ... ok
7/106 ... ok
27/107 ... ok
27/108 ... ok
27/109 ... ok
27/110 ... ok
19/111 ... ok
5/112 ... ok
Running /Users/git/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /Users/git/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes
Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Checking Reply by email ...

Reply by email is disabled in config/gitlab.yml

Checking Reply by email ... Finished

Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... yes
Init script up-to-date? ... yes
projects have namespace: ... 
7/2 ... yes
7/5 ... yes
7/6 ... yes
7/7 ... yes
7/8 ... yes
7/10 ... yes
7/11 ... yes
7/12 ... yes
7/13 ... yes
7/14 ... yes
7/15 ... yes
7/16 ... yes
7/17 ... yes
7/19 ... yes
7/20 ... yes
7/21 ... yes
7/22 ... yes
1/26 ... yes
7/27 ... yes
5/28 ... yes
1/29 ... yes
1/30 ... yes
5/31 ... yes
5/32 ... yes
5/33 ... yes
6/36 ... yes
1/37 ... yes
4/40 ... yes
4/41 ... yes
4/42 ... yes
5/43 ... yes
5/44 ... yes
5/45 ... yes
5/46 ... yes
4/47 ... yes
5/48 ... yes
4/49 ... yes
4/50 ... yes
5/51 ... yes
5/52 ... yes
4/53 ... yes
5/54 ... yes
9/55 ... yes
5/56 ... yes
1/57 ... yes
24/58 ... yes
27/59 ... yes
5/60 ... yes
1/61 ... yes
19/63 ... yes
19/64 ... yes
19/65 ... yes
19/66 ... yes
19/67 ... yes
19/68 ... yes
19/69 ... yes
19/70 ... yes
19/71 ... yes
19/72 ... yes
19/73 ... yes
19/74 ... yes
19/75 ... yes
19/76 ... yes
19/77 ... yes
19/78 ... yes
19/79 ... yes
19/80 ... yes
19/81 ... yes
19/85 ... yes
19/86 ... yes
19/87 ... yes
19/88 ... yes
19/89 ... yes
4/92 ... yes
1/93 ... yes
1/94 ... yes
2/95 ... yes
27/96 ... yes
2/97 ... yes
25/98 ... yes
19/99 ... yes
19/100 ... yes
19/101 ... yes
19/102 ... yes
24/103 ... yes
25/104 ... yes
2/105 ... yes
7/106 ... yes
27/107 ... yes
27/108 ... yes
27/109 ... yes
27/110 ... yes
19/111 ... yes
5/112 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.1)
Your git bin path is "/usr/local/bin/git"
Git version >= 2.7.3 ? ... yes (2.17.1)
Active users: 8

Checking GitLab ... Finished

Possible fixes

The only fix I have found so far is to manually skip the failing migration stages, but the issue will come back the next time I try to upgrade.