We are running a self-hosted omnibus Gitlab, updated this morning to 10.1. Since then we cant browse any project home or settings or issue details. No problem in the other sections (issues, repo, snippets, ...)
Steps to reproduce
Browse to any project home (e.g. https:/your-gitlab/group1/project1) or settings (e.g. https:/your-gitlab/group1/project1/edit) or issue details (e.g. https:/your-gitlab/group1/project1/issues/1).
Get a 500 error
Relevant logs and/or screenshots
tarted GET "/group1/project1" for 151.65.116.107 at 2017-10-23 10:37:56 +0200Started GET "/-/metrics" for 127.0.0.1 at 2017-10-23 10:37:57 +0200Processing by MetricsController#index as HTMLFilter chain halted as :validate_prometheus_metrics rendered or redirectedCompleted 404 Not Found in 45ms (Views: 44.3ms | ActiveRecord: 0.0ms)Processing by ProjectsController#show as HTMLParameters: {"namespace_id"=>"group1", "id"=>"project1"}Completed 500 Internal Server Error in 136ms (ActiveRecord: 9.5ms)ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "fork_network_members" does not existLINE 5: WHERE a.attrelid = '"fork_network_members"'::...^: SELECT a.attname, format_type(a.atttypid, a.atttypmod),pg_get_expr(d.adbin, d.adrelid), a.attnotnull, a.atttypid, a.atttypmodFROM pg_attribute a LEFT JOIN pg_attrdef dON a.attrelid = d.adrelid AND a.attnum = d.adnumWHERE a.attrelid = '"fork_network_members"'::regclassAND a.attnum > 0 AND NOT a.attisdroppedORDER BY a.attnum):app/models/project.rb:1012:in `forked?'app/models/project.rb:634:in `import?'app/models/project.rb:654:in `import_started?'app/models/project.rb:650:in `import_in_progress?'app/controllers/projects_controller.rb:102:in `show'lib/gitlab/i18n.rb:47:in `with_locale'lib/gitlab/i18n.rb:53:in `with_user_locale'app/controllers/application_controller.rb:337:in `set_locale'lib/gitlab/middleware/multipart.rb:93:in `call'lib/gitlab/request_profiler/middleware.rb:14:in `call'lib/gitlab/middleware/go.rb:17:in `call'lib/gitlab/etag_caching/middleware.rb:11:in `call'lib/gitlab/middleware/read_only.rb:30:in `call'lib/gitlab/request_context.rb:18:in `call'lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
I'm also having the same problem after upgrading from 8.9 version, but i have a different error at migrations:
[root@nilo gitlab-rails]# gitlab-ctl pg-upgradeChecking for an omnibus managed postgresql: NOT OKNo currently installed postgresql in the omnibus instance found.[root@nilo gitlab-rails]# gitlab-rake db:migrate --verbose== 20170825104051 MigrateIssuesToGhostUser: migrating =========================== 20170825104051 MigrateIssuesToGhostUser: migrated (0.3187s)================== 20170825154015 ResolveOutdatedDiffDiscussions: migrating ===================-- add_column(:projects, :resolve_outdated_diff_discussions, :boolean) -> 0.0079s== 20170825154015 ResolveOutdatedDiffDiscussions: migrated (0.0080s)============ 20170828093725 CreateProjectAutoDevOps: migrating ==========================-- create_table(:project_auto_devops) -> 0.0471s== 20170828093725 CreateProjectAutoDevOps: migrated (0.0472s)=================== 20170828135939 MigrateUserExternalMailData: migrating ======================-- execute(" INSERT INTO user_synced_attributes_metadata (user_id, provider, email_synced)\n SELECT id, email_provider, external_email\n FROM users\n WHERE external_email = TRUE\n AND NOT EXISTS (\n SELECT true\n FROM user_synced_attributes_metadata\n WHERE user_id = users.id\n AND (provider = users.email_provider OR (provider IS NULL AND users.email_provider IS NULL))\n )\n AND id BETWEEN 1 AND 34\n")rake aborted!StandardError: An error has occurred, all later migrations canceled:PG::UniqueViolation: ERRO: duplicar valor da chave viola a restrição de unicidade "index_user_synced_attributes_metadata_on_user_id"DETAIL: Chave (user_id)=(2) já existe.: INSERT INTO user_synced_attributes_metadata (user_id, provider, email_synced) SELECT id, email_provider, external_email FROM usersWHERE external_email = TRUE AND NOT EXISTS ( SELECT trueFROM user_synced_attributes_metadata WHERE user_id = users.id AND (provider = users.email_provider OR (provider IS NULL AND users.email_provider IS NULL))) AND id BETWEEN 1 AND 34/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170828135939_migrate_user_external_mail_data.rb:27:in `block in up'/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:75:in `block in each_batch'/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:56:in `step'/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:56:in `each_batch'/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170828135939_migrate_user_external_mail_data.rb:24:in `up'/opt/gitlab/embedded/bin/bundle:23:in `load'/opt/gitlab/embedded/bin/bundle:23:in `<main>'ActiveRecord::RecordNotUnique: PG::UniqueViolation: ERRO: duplicar valor da chave viola a restrição de unicidade "index_user_synced_attributes_metadata_on_user_id"DETAIL: Chave (user_id)=(2) já existe.: INSERT INTO user_synced_attributes_metadata (user_id, provider, email_synced) SELECT id, email_provider, external_email FROM users WHERE external_email = TRUE AND NOT EXISTS ( SELECT true FROM user_synced_attributes_metadata WHERE user_id = users.id AND (provider = users.email_provider OR (provider IS NULL AND users.email_provider IS NULL)) ) AND id BETWEEN 1 AND 34/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170828135939_migrate_user_external_mail_data.rb:27:in `block in up'/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:75:in `block in each_batch'/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:56:in `step'/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:56:in `each_batch'/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170828135939_migrate_user_external_mail_data.rb:24:in `up'/opt/gitlab/embedded/bin/bundle:23:in `load'/opt/gitlab/embedded/bin/bundle:23:in `<main>'PG::UniqueViolation: ERRO: duplicar valor da chave viola a restrição de unicidade "index_user_synced_attributes_metadata_on_user_id"DETAIL: Chave (user_id)=(2) já existe./opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170828135939_migrate_user_external_mail_data.rb:27:in `block in up'/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:75:in `block in each_batch'/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:56:in `step'/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:56:in `each_batch'/opt/gitlab/embedded/service/gitlab-rails/db/migrate/20170828135939_migrate_user_external_mail_data.rb:24:in `up'/opt/gitlab/embedded/bin/bundle:23:in `load'/opt/gitlab/embedded/bin/bundle:23:in `<main>'Tasks: TOP => db:migrate(See full trace by running task with --trace)
I'm assuming that was safe to do as if it's a duplicate column then it must have been added by a prior migration. Presumably this same one seeing as it dates to 2015. Seems odd.
Thanks @bencromwell for your support. Anyway I'm bothered by the error that brought me here in the first place.
Are ActiveRecord::StatementInvalid (PG::UndefinedTable: ERROR: relation "fork_network_members" does not exist and the fast forward option migration related? The failure of the latter, compromised the former migration?
I wonder if something was changed that simply detects past failed migrations, or migrations that ran but didn't mark as complete, and tries to re-run them.
the same error brought me here. Thanks to the tip of @bencromwell I could resolve the problem with commenting out the following line in /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb
:
add_column_with_default(:projects, :merge_requests_ff_only_enabled, :boolean, default: false)
Ditto - thanks @bencromwell. I had the same issue following the recent update. Commented out the add_column in 20150827121444_add_fast_forward_option_to_project.rb and all is good after the reconfigure. I'm hoping it's just a defect with the migration script.
Also solved it for me - now the question is: Leave the line commented out in 20150827121444_add_fast_forward_option_to_project.rb or do something else for the long run?
Added the same fix @bencromwell mentioned, I am using the omnibus package and upgrading from 9.1.x to 10.1.1. Curious to know if this will hack will be ok long term.
@stanhu Yeah, we probably should. I'm not a big fan of fixes like this because we don't understand how it got added but I see about 10 people who fixed their instance using this workaround.
I suspect that some time ago we released CE with mistakenly added merge_requests_ff_only_enabled to schema.rb and lots of people who installed CE from that version is affected now. But it's only an assumption...
OK, I was right, I wrote a little ruby script that iterates over 600 tags (starting from v5.* releases) and searches for merge_requests_ff_only_enabled in db/schema.rb. I found that we mistakenly added it to 8.2.* releases and then removed in 8.3.* and higher. So if you installed GitLab since 8.2 you will be affected by this issue. The fix is to add a condition to migration!