Post deployment migration "20181204040404_migrate_project_approvers.rb" fails on GitLab.com
The migration 20181204040404_migrate_project_approvers.rb
fails on GitLab.com with the following error:
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
1000000000 is out of range for ActiveModel::Type::Integer with limit 2
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_locking.rb:14:in `_create_record'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/background_migration/migrate_approver_to_approval_rules.rb:166:in `first_or_initialize'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/background_migration/migrate_approver_to_approval_rules.rb:144:in `sync_rule'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/background_migration/migrate_approver_to_approval_rules.rb:135:in `handle_project'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/background_migration/migrate_approver_to_approval_rules.rb:119:in `block in perform'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/background_migration/migrate_approver_to_approval_rules.rb:114:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/ee/db/post_migrate/20181204040404_migrate_project_approvers.rb:24:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/ee/db/post_migrate/20181204040404_migrate_project_approvers.rb:23:in `each'
/opt/gitlab/embedded/service/gitlab-rails/ee/db/post_migrate/20181204040404_migrate_project_approvers.rb:23:in `up'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Caused by:
ActiveModel::RangeError: 1000000000 is out of range for ActiveModel::Type::Integer with limit 2
/opt/gitlab/embedded/service/gitlab-rails/config/initializers/active_record_locking.rb:14:in `_create_record'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/background_migration/migrate_approver_to_approval_rules.rb:166:in `first_or_initialize'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/background_migration/migrate_approver_to_approval_rules.rb:144:in `sync_rule'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/background_migration/migrate_approver_to_approval_rules.rb:135:in `handle_project'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/background_migration/migrate_approver_to_approval_rules.rb:119:in `block in perform'
/opt/gitlab/embedded/service/gitlab-rails/ee/lib/gitlab/background_migration/migrate_approver_to_approval_rules.rb:114:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/ee/db/post_migrate/20181204040404_migrate_project_approvers.rb:24:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/ee/db/post_migrate/20181204040404_migrate_project_approvers.rb:23:in `each'
/opt/gitlab/embedded/service/gitlab-rails/ee/db/post_migrate/20181204040404_migrate_project_approvers.rb:23:in `up'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
This is currently preventing us from publishing the RC6 package and performing further deploys without running into this error again. This migration was introduced in MR https://gitlab.com/gitlab-org/gitlab-ee/merge_requests/8669/diffs#f52ed8b2cc41680ef4442eafa8dd577008c3505e, which was approved in exception request gitlab-org/release/tasks#633 (closed). Per the RC6 QA issue it appears no testing was done on staging for this feature, though it might have been possible for staging to not run into this error as it appears to be caused by the volume of data on GitLab.com.
Looking at the merge request approval tables, the only column with this limit appears to be approval_merge_request_rules.approvals_required
, which is defined as follows:
t.integer "approvals_required", limit: 2, default: 0, null: false
Here limit: 2
means the column is a 2 byte (or 16 bits) integer, allowing for a maximum value of 65 535. The Rails code does not appear to have any validations for this column in the models, and I believe our previous limit was effectively infinite. Based on this I think we need to do two things:
- Modify the models to limit the maximum value to something reasonable, e.g. 100 or 1000 (anything more than that would just be insane).
- Modify the background migration to fall back to this limit when the input limit is too great. This would be a matter of something along the lines of
[100, old_value].min