An aborted rename column migration requires manual intervention
This happened today with @mayra-cabrera when she attempted to run 20180523042841_rename_merge_requests_allow_maintainer_to_push.rb, which did:
rename_column_concurrently :merge_requests, :allow_maintainer_to_push, :allow_collaboration
Because there were multiple base backups running from the primary, we hit a statement timeout and the migration aborted with https://sentry.gitlap.com/gitlab/staginggitlabcom/issues/218852/.
After aborting, this left the new column allow_collaboration and the trigger trigger_dbcff282ad7b:
This left the new column around and rename trigger:
gitlabhq_production=# \df+ trigger_dbcff282ad7b
List of functions
-[ RECORD 1 ]-------+---------------------------------------------------------------
Schema | public
Name | trigger_dbcff282ad7b
Result data type | trigger
Argument data types |
Type | trigger
Volatility | volatile
Parallel | unsafe
Owner | gitlab
Security | invoker
Access privileges |
Language | plpgsql
Source code | +
| BEGIN +
| NEW."allow_collaboration" := NEW."allow_maintainer_to_push";+
| RETURN NEW; +
| END; +
|
Description |
Rerunning the migrations resulted in:
✖ Running first batch of migrations from the blessed node (1 minute)
52.177.186.117 == 20180523042841 RenameMergeRequestsAllowMaintainerToPush: migrating =========
52.177.186.117 -- transaction_open?()
52.177.186.117 -> 0.0000s
52.177.186.117 -- columns(:merge_requests)
52.177.186.117 -> 0.0021s
52.177.186.117 -- add_column(:merge_requests, :allow_collaboration, :boolean, {:limit=>nil, :precision=>nil, :scale=>nil})
52.177.186.117 rake aborted!
52.177.186.117 StandardError: An error has occurred, all later migrations canceled:
52.177.186.117
52.177.186.117 PG::DuplicateColumn: ERROR: column "allow_collaboration" of relation "merge_requests" already exists
52.177.186.117 : ALTER TABLE "merge_requests" ADD "allow_collaboration" boolean
52.177.186.117 /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in `add_column'
52.177.186.117 /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:420:in `rename_column_concurrently'
52.177.186.117 /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20180523042841_rename_merge_requests_allow_maintainer_to_push.rb:9:in `up'
52.177.186.117 /opt/gitlab/embedded/bin/bundle:23:in `load'
52.177.186.117 /opt/gitlab/embedded/bin/bundle:23:in `<main>'
52.177.186.117
To recover, @eReGeBe had to run:
ALTER TABLE merge_requests DROP COLUMN allow_collaboration
DROP TRIGGER trigger_dbcff282ad7b ON merge_requests
Perhaps we should modify the migration helper to:
- Check if the column exists. If it does, check the type. If it is different from the expected type, fail.
- Otherwise move along.
- Check if the trigger exists. Drop and recreate.
/cc: @yorickpeterse, @abrandl
Edited by 🤖 GitLab Bot 🤖