DB Migration failing 20181121111200_schedule_runners_token_encryption with missing column

Summary

DB Migration failing 20181121111200_schedule_runners_token_encryption with missing column

Steps to reproduce

Update Gitlab 11.5.3 to 11.6.3 with ci runners registered and call db migrate task

Example Project

No

What is the current bug behavior?

The DB migration fails for running post migrate tasks

What is the expected correct behavior?

All DB migration tasks work fine

Relevant logs and/or screenshots

== 20181121111200 ScheduleRunnersTokenEncryption: migrating ===================
rake aborted!
StandardError: An error has occurred, all later migrations canceled:

the column: argument must be set to a column name to use for ordering rows
/opt/gitlab/embedded/service/gitlab-rails/app/models/concerns/each_batch.rb:52:in `each_batch'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20181121111200_schedule_runners_token_encryption.rb:22:in `block in up'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20181121111200_schedule_runners_token_encryption.rb:21:in `each'
/opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20181121111200_schedule_runners_token_encryption.rb:21:in `up'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:792:in `exec_migration'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:773:in `block (2 levels) in migrate'
/opt/gitlab/embedded/lib/ruby/2.5.0/benchmark.rb:293:in `measure'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:772:in `block in migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/connection_adapters/abstract/connection_pool.rb:398:in `with_connection'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:771:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration/compatibility.rb:121:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:951:in `migrate'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:1232:in `block in execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:1302:in `ddl_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:1231:in `execute_migration_in_transaction'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:1190:in `run_without_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:1142:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:1319:in `with_advisory_lock'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:1142:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/migration.rb:1017:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/railties/databases.rake:99:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/activerecord-5.0.7/lib/active_record/railties/databases.rake:84:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `block in execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:271:in `execute'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:213:in `block in invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/2.5.0/monitor.rb:226:in `mon_synchronize'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:193:in `invoke_with_call_chain'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/task.rb:182:in `invoke'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:160:in `invoke_task'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block (2 levels) in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `each'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:116:in `block in top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:125:in `run_with_threads'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:110:in `top_level'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:83:in `block in run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:186:in `standard_exception_handling'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/lib/rake/application.rb:80:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/rake-12.3.1/exe/rake:27:in `<top (required)>'
/opt/gitlab/embedded/bin/rake:23:in `load'
/opt/gitlab/embedded/bin/rake:23:in `<top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in `load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:74:in `kernel_load'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli/exec.rb:28:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:424:in `exec'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/command.rb:27:in `run'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/invocation.rb:126:in `invoke_command'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor.rb:387:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:27:in `dispatch'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/vendor/thor/lib/thor/base.rb:466:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/cli.rb:18:in `start'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:30:in `block in <top (required)>'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/lib/bundler/friendly_errors.rb:124:in `with_friendly_errors'
/opt/gitlab/embedded/lib/ruby/gems/2.5.0/gems/bundler-1.16.6/exe/bundle:22:in `<top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'

Output of checks

(If you are reporting a bug on GitLab.com, write: This bug happens on GitLab.com)

Results of GitLab environment info

Expand for output related to GitLab environment info 
Info RAILS_ENV=production
System information
System:
Current User:   whoami: cannot find name for user ID 1000220000
Using RVM:      no
Ruby Version:   2.5.3p105
Gem Version:    2.7.6
Bundler Version:1.16.6
Rake Version:   12.3.1
Redis Version:  3.2.12
Git Version:    2.18.1
Sidekiq Version:5.2.3
Go Version:     unknown

GitLab information
Version:        11.6.3
Revision:       bc9391c
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://gitlab.example.com
HTTP Clone URL: https://gitlab.example.com/some-group/some-project.git
SSH Clone URL:  git@gitlab.example.com:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers: openid_connect


GitLab Shell
Version:        unknown
Repository storage paths:
default:      /opt/dol/data/gitlab/repositories
Hooks:          /opt/dol/gitlab/var/gitlab/hooks
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 8.4.3 ? ... OK (8.4.3)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /opt/dol/shared/gitlab/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... OK


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
...
Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes


Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.5.3)
Git version >= 2.18.0 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 1924


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

The function EachBatch::each_batch expects a column and :primary_key is not being found. The fix for me was to add :id as column in the post migration file /opt/gitlab/embedded/service/gitlab-rails/db/post_migrate/20181121111200_schedule_runners_token_encryption.rb:

# frozen_string_literal: true

class ScheduleRunnersTokenEncryption < ActiveRecord::Migration
  include Gitlab::Database::MigrationHelpers

  DOWNTIME = false
  BATCH_SIZE = 10000
  RANGE_SIZE = 2000
  MIGRATION = 'EncryptRunnersTokens'

  MODELS = [
    ::Gitlab::BackgroundMigration::Models::EncryptColumns::Settings,
    ::Gitlab::BackgroundMigration::Models::EncryptColumns::Namespace,
    ::Gitlab::BackgroundMigration::Models::EncryptColumns::Project,
    ::Gitlab::BackgroundMigration::Models::EncryptColumns::Runner
  ].freeze

  disable_ddl_transaction!

  def up
    MODELS.each do |model|
      model.each_batch(of: BATCH_SIZE, column: :id) do |relation, index|
        delay = index * 4.minutes

        relation.each_batch(of: RANGE_SIZE, column: :id) do |relation|
          range = relation.pluck('MIN(id)', 'MAX(id)').first
          args = [model.name.demodulize.downcase, *range]

          BackgroundMigrationWorker.perform_in(delay, MIGRATION, args)
        end
      end
    end
  end

  def down
    # no-op
  end
end
Edited by 🤖 GitLab Bot 🤖