Migration failing with missing class Gitlab::BackgroundMigration::PopulateAnyApprovalRuleForProjects
Summary
Migrations failing for Gitlab-FOSS:
- 20190905091812_schedule_project_any_approval_rule_migration.rb
- 20190905091831_schedule_merge_request_any_approval_rule_migration.rb
Steps to reproduce
- Upgrade Gitlab community edition from 12.2.x to 12.3.1
- Check for pending Sidekiq jobs:
background_migration BackgroundMigrationWorker "PopulateAnyApprovalRuleForProjects", [5437, 5437]
- Job fails:
background_migration BackgroundMigrationWorker "PopulateAnyApprovalRuleForProjects", [5437, 5437] NameError: uninitialized constant Gitlab::BackgroundMigration::PopulateAnyApprovalRuleForProjec
The project was an export from an EE Gitlab installation with approvals_before_merge
in projects
table set to a non 0
value. The import in the Gitlab CE stored the non-zero value which enabled the migration.
What is the current bug behavior?
The background migration are failing for:
- 20190905091812_schedule_project_any_approval_rule_migration.rb
- 20190905091831_schedule_merge_request_any_approval_rule_migration.rb
What is the expected correct behavior?
The background migration should finish successful.
Relevant logs and/or screenshots
2019-09-25T18:46:14.819Z 8 TID-ov14xpbqc BackgroundMigrationWorker JID-7bc954fc5bba5e52108de8f2 INFO: start
2019-09-25T18:46:14.931Z 8 TID-ov14xpbqc BackgroundMigrationWorker JID-7bc954fc5bba5e52108de8f2 INFO: fail: 0.112 sec
2019-09-25T18:46:14.931Z 8 TID-ov14xpbqc WARN: {"context":"Job raised exception","job":{"class":"BackgroundMigrationWorker","args":["PopulateAnyApprovalRuleForProjects",[5437,5437]],"retry":3,"queue":"background_migration","jid":"7bc954fc5bba5e52108de8f2","created_at":1569437091.5091581,"correlation_id":"29ac84a2-27bb-4a0d-81a5-deb28fdb8ddc","enqueued_at":1569437174.8184438},"jobstr":"{\"class\":\"BackgroundMigrationWorker\",\"args\":[\"PopulateAnyApprovalRuleForProjects\",[5437,5437]],\"retry\":3,\"queue\":\"background_migration\",\"jid\":\"7bc954fc5bba5e52108de8f2\",\"created_at\":1569437091.5091581,\"correlation_id\":\"29ac84a2-27bb-4a0d-81a5-deb28fdb8ddc\",\"enqueued_at\":1569437174.8184438}"}
2019-09-25T18:46:14.931Z 8 TID-ov14xpbqc WARN: NameError: uninitialized constant Gitlab::BackgroundMigration::PopulateAnyApprovalRuleForProjects
2019-09-25T18:46:14.931Z 8 TID-ov14xpbqc WARN: /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:81:in `const_get'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:81:in `migration_class_for'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/background_migration.rb:58:in `perform'
/opt/gitlab/embedded/service/gitlab-rails/app/workers/background_migration_worker.rb:26:in `perform'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:192:in `execute_job'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:165:in `block (2 levels) in process'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:128:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:15:in `block in call'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/transaction.rb:57:in `run'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/metrics/sidekiq_middleware.rb:15:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_status/server_middleware.rb:7:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/instrumentation_logger.rb:7:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/correlation_logger.rb:10:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/gitlab-labkit-0.5.2/lib/labkit/correlation/correlation_id.rb:18:in `use_id'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/correlation_logger.rb:9:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/batch_loader.rb:7:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/request_store_middleware.rb:8:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/memory_killer.rb:20:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/metrics.rb:23:in `block in call'
/opt/gitlab/embedded/lib/ruby/2.6.0/benchmark.rb:308:in `realtime'
/opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/sidekiq_middleware/metrics.rb:22:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sentry-raven-2.9.0/lib/raven/integrations/sidekiq.rb:9:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:130:in `block in invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/middleware/chain.rb:133:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:164:in `block in process'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:137:in `block (6 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:109:in `local'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:136:in `block (5 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:43:in `block in call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/reloader.rb:73:in `block in wrap'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/execution_wrapper.rb:87:in `wrap'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/activesupport-5.2.3/lib/active_support/reloader.rb:72:in `wrap'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/rails.rb:42:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:132:in `block (4 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:250:in `stats'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:127:in `block (3 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_logger.rb:8:in `call'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:126:in `block (2 levels) in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/job_retry.rb:74:in `global'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:125:in `block in dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:48:in `with_context'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/logging.rb:42:in `with_job_hash_context'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:124:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:163:in `process'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:83:in `process_one'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/processor.rb:71:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:16:in `watchdog'
/opt/gitlab/embedded/lib/ruby/gems/2.6.0/gems/sidekiq-5.2.7/lib/sidekiq/util.rb:25:in `block in safe_thread'
Possible fixes
The migration files are copied in the gitlab-foss repository:
- 20190905091812_schedule_project_any_approval_rule_migration.rb
- 20190905091831_schedule_merge_request_any_approval_rule_migration.rb
The class implementation referenced in the migration files are in the ee
directory and are removed from the FOSS mirror:
- ee/lib/gitlab/background_migration/populate_any_approval_rule_for_merge_requests.rb
- ee/lib/gitlab/background_migration/populate_any_approval_rule_for_projects.rb
The fix is to either make the migration rules also part of the EE or move the implementation out of ee
directory.
Edited by Horatiu Eugen Vlad