Gitpod DB migrations fail: NameError: undefined local variable or method `pause_ms'
Summary
Starting GitLab in Gitpod fails at the point where database migrations are being run. This renders an error with pause_ms
.
The DB migrations being run seem to be related to int => bigint
changes.
Steps to reproduce
- Navigate into the repository view and open Gitpod
- Wait until the workspace is created
- Investigate the terminal log & web view
Example Project
What is the current bug behavior?
DB migrations fail in the Gitpod workspace.
What is the expected correct behavior?
DB migrations continue and the development environment comes up in Gitpod.
Relevant logs and/or screenshots
== 20210311045138 SetTraversalIdsForGitlabOrgGroupStaging: migrating ==========
== 20210311045138 SetTraversalIdsForGitlabOrgGroupStaging: migrated (0.0000s) =
== 20210311045139 SetTraversalIdsForGitlabOrgGroupCom: migrating ==============
== 20210311045139 SetTraversalIdsForGitlabOrgGroupCom: migrated (0.0000s) =====
== 20210311093723 AddPartialIndexOnCiPipelinesByCancelableStatusAndUsers: migrating
-- transaction_open?()
-> 0.0000s
-- index_exists?(:ci_pipelines, [:user_id, :id], {:where=>"((status)::text = ANY (\n ARRAY[\n ('running'::character varying)::text,\n ('waiting_for_resource'::character varying)::text,\n ('preparing'::character varying)::text,\n ('pending'::character varying)::text,\n ('created'::character varying)::text,\n ('scheduled'::character varying)::text\n ]\n))\n", :name=>"index_ci_pipelines_on_user_id_and_id_and_cancelable_status", :algorithm=>:concurrently})
-> 0.0081s
-- add_index(:ci_pipelines, [:user_id, :id], {:where=>"((status)::text = ANY (\n ARRAY[\n ('running'::character varying)::text,\n ('waiting_for_resource'::character varying)::text,\n ('preparing'::character varying)::text,\n ('pending'::character varying)::text,\n ('created'::character varying)::text,\n ('scheduled'::character varying)::text\n ]\n))\n", :name=>"index_ci_pipelines_on_user_id_and_id_and_cancelable_status", :algorithm=>:concurrently})
-> 0.0292s
== 20210311093723 AddPartialIndexOnCiPipelinesByCancelableStatusAndUsers: migrated (0.0387s)
== 20210311120152 AddMetricsToBatchedBackgroundMigrationJobs: migrating =======
-- add_column(:batched_background_migration_jobs, :metrics, :jsonb, {:null=>false, :default=>{}})
-> 0.0232s
== 20210311120152 AddMetricsToBatchedBackgroundMigrationJobs: migrated (0.0232s)
== 20210311120153 InitializeConversionOfEventsIdToBigint: migrating ===========
-- table_exists?(:events)
-> 0.0017s
-- column_exists?(:events, :id)
-> 0.0014s
-- column_exists?(:events, :id)
-> 0.0011s
-- columns(:events)
-> 0.0012s
-- add_column(:events, "id_convert_to_bigint", :bigint, {:default=>0, :null=>false})
-> 0.0016s
NOTICE: trigger "trigger_69523443cc10" for relation "events" does not exist, skipping
== 20210311120153 InitializeConversionOfEventsIdToBigint: migrated (0.0165s) ==
== 20210311120154 InitializeConversionOfPushEventPayloadsEventIdToBigint: migrating
-- table_exists?(:push_event_payloads)
-> 0.0017s
-- column_exists?(:push_event_payloads, :event_id)
-> 0.0013s
-- column_exists?(:push_event_payloads, :event_id)
-> 0.0011s
-- columns(:push_event_payloads)
-> 0.0011s
-- add_column(:push_event_payloads, "event_id_convert_to_bigint", :bigint, {:default=>0, :null=>false})
-> 0.0015s
NOTICE: trigger "trigger_07c94931164e" for relation "push_event_payloads" does not exist, skipping
== 20210311120154 InitializeConversionOfPushEventPayloadsEventIdToBigint: migrated (0.0140s)
== 20210311120155 BackfillEventsIdForBigintConversion: migrating ==============
-- table_exists?(:events)
-> 0.0017s
-- column_exists?(:events, :id)
-> 0.0013s
-- column_exists?(:events, :id)
-> 0.0012s
-- column_exists?(:events, "id_convert_to_bigint")
-> 0.0011s
then triggers
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
undefined local variable or method `pause_ms' for #<Gitlab::Database::BackgroundMigration::BatchedMigration:0x00005648eac02518>
Did you mean? paused!
/workspace/.rvm/ruby-2.7.2/gems/activemodel-6.0.3.6/lib/active_model/attribute_methods.rb:432:in `method_missing'
/workspace/gitlab/lib/gitlab/database/background_migration/batched_migration.rb:43:in `create_batched_job!'
/workspace/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:54:in `create_next_batched_job!'
/workspace/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:22:in `run_migration_job'
/workspace/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:39:in `run_entire_migration'
/workspace/gitlab/lib/gitlab/database/migration_helpers.rb:1047:in `backfill_conversion_of_integer_to_bigint'
/workspace/gitlab/db/post_migrate/20210311120155_backfill_events_id_for_bigint_conversion.rb:11:in `up'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:831:in `exec_migration'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'
/workspace/.rvm/ruby-2.7.2/gems/benchmark-0.1.1/lib/benchmark.rb:293:in `measure'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:811:in `block in migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:810:in `migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1002:in `migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1361:in `block in ddl_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/transactions.rb:212:in `transaction'
/workspace/gitlab/lib/gitlab/database.rb:342:in `block in transaction'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/notifications.rb:182:in `instrument'
/workspace/gitlab/lib/gitlab/database.rb:341:in `transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1361:in `ddl_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1280:in `each'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1280:in `migrate_without_lock'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1229:in `block in migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1382:in `with_advisory_lock'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1229:in `migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1061:in `up'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1036:in `migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/tasks/database_tasks.rb:238:in `migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:86:in `block (3 levels) in <main>'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:84:in `each'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:84:in `block (2 levels) in <main>'
Caused by:
NameError: undefined local variable or method `pause_ms' for #<Gitlab::Database::BackgroundMigration::BatchedMigration:0x00005648eac02518>
Did you mean? paused!
/workspace/.rvm/ruby-2.7.2/gems/activemodel-6.0.3.6/lib/active_model/attribute_methods.rb:432:in `method_missing'
/workspace/gitlab/lib/gitlab/database/background_migration/batched_migration.rb:43:in `create_batched_job!'
/workspace/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:54:in `create_next_batched_job!'
/workspace/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:22:in `run_migration_job'
/workspace/gitlab/lib/gitlab/database/background_migration/batched_migration_runner.rb:39:in `run_entire_migration'
/workspace/gitlab/lib/gitlab/database/migration_helpers.rb:1047:in `backfill_conversion_of_integer_to_bigint'
/workspace/gitlab/db/post_migrate/20210311120155_backfill_events_id_for_bigint_conversion.rb:11:in `up'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:831:in `exec_migration'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:812:in `block (2 levels) in migrate'
/workspace/.rvm/ruby-2.7.2/gems/benchmark-0.1.1/lib/benchmark.rb:293:in `measure'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:811:in `block in migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/connection_pool.rb:471:in `with_connection'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:810:in `migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1002:in `migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1310:in `block in execute_migration_in_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1361:in `block in ddl_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `block in transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/transaction.rb:280:in `block in within_new_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:26:in `block (2 levels) in synchronize'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `handle_interrupt'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:25:in `block in synchronize'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `handle_interrupt'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/concurrency/load_interlock_aware_monitor.rb:21:in `synchronize'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/transaction.rb:278:in `within_new_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/connection_adapters/abstract/database_statements.rb:280:in `transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/transactions.rb:212:in `transaction'
/workspace/gitlab/lib/gitlab/database.rb:342:in `block in transaction'
/workspace/.rvm/ruby-2.7.2/gems/activesupport-6.0.3.6/lib/active_support/notifications.rb:182:in `instrument'
/workspace/gitlab/lib/gitlab/database.rb:341:in `transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1361:in `ddl_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1309:in `execute_migration_in_transaction'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1281:in `block in migrate_without_lock'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1280:in `each'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1280:in `migrate_without_lock'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1229:in `block in migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1382:in `with_advisory_lock'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1229:in `migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1061:in `up'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/migration.rb:1036:in `migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/tasks/database_tasks.rb:238:in `migrate'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:86:in `block (3 levels) in <main>'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:84:in `each'
/workspace/.rvm/ruby-2.7.2/gems/activerecord-6.0.3.6/lib/active_record/railties/databases.rake:84:in `block (2 levels) in <main>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
Fri 23 Apr 2021 01:28:22 PM UTC – Stopping GDK
ℹ️ gdk stop: before hook -> cd gitlab && bin/spring stop || true
Spring is not running
=> Shutting down all services:
ok: down: ./services/gitlab-pages: 46s
ok: down: ./services/gitlab-workhorse: 46s
ok: down: ./services/jaeger: 46s
ok: down: ./services/rails-background-jobs: 46s
ok: down: ./services/rails-web: 46s
ok: down: ./services/webpack: 46s
ok: down: ./services/praefect: 1s
ok: down: ./services/praefect-gitaly-0: 1s
ok: down: ./services/redis: 1s
ok: down: ./services/postgresql: 0s
=> Shutting down runsvdir (pid 17083)
✅️ All services have been shut down!
Fri 23 Apr 2021 01:28:24 PM UTC – GDK stopped
Fri 23 Apr 2021 01:28:24 PM UTC – Starting GDK
ok: run: ./services/gitlab-pages: (pid 18259) 1s, normally down
ok: run: ./services/gitlab-workhorse: (pid 18258) 1s, normally down
ok: run: ./services/jaeger: (pid 18257) 1s, normally down
ok: run: ./services/postgresql: (pid 18260) 1s, normally down
ok: run: ./services/praefect: (pid 18261) 1s, normally down
ok: run: ./services/praefect-gitaly-0: (pid 18263) 1s, normally down
ok: run: ./services/rails-background-jobs: (pid 18262) 1s, normally down
ok: run: ./services/rails-web: (pid 18264) 1s, normally down
ok: run: ./services/redis: (pid 18265) 1s, normally down
ok: run: ./services/webpack: (pid 18266) 1s, normally down
=> GitLab will be available at http://127.0.0.1:3000 shortly.
/workspace/.rvm/ruby-2.7.2/gems/lefthook-0.7.2/libexec/lefthook-linux install
SYNCING lefthook.yml
SERVED HOOKS: pre-push, prepare-commit-msg
Updated 1 path from the index
Awaiting port 3000... ok
Waiting for GitLab at https://3000-scarlet-jellyfish-gvow53rq.ws-eu03.gitpod.io ...
Fri 23 Apr 2021 01:28:33 PM UTC – GitLab is up (took ~0.1 minutes)
Output of checks
Results of GitLab environment info
gitlab-rake
does not exist in the Gitpod dev environment.
gitpod /workspace/gitlab $ git show HEAD
commit 4a8b7d098dae09952b2788e56686cf1acd760945 (HEAD -> master, origin/master, origin/HEAD)
Merge: 7adf65f7087 588d5630688
Author: Matthias Käppler <mkaeppler@gitlab.com>
Date: Fri Apr 23 13:08:45 2021 +0000
Merge branch 'remove-tempfile-unlink-env-var' into 'master'
Remove GITLAB_TEMPFILE_IMMEDIATE_UNLINK env var
See merge request gitlab-org/gitlab!60110
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) (For installations from source run and paste the output of: `sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production`)
Results of GitLab application Check
Does not exist in Gitpod.
gitpod /workspace/gitlab-development-kit $ gdk version
GitLab Development Kit gem version 0.2.10
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of:
sudo gitlab-rake gitlab:check SANITIZE=true
)(For installations from source run and paste the output of:
sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true
)(we will only investigate if the tests are passing)