Migration job fails after chart upgrade
Summary
I've tried to upgrade GitLab chart and faced an issue with migration-db
job, it failed with the following error:
Object Storage is not enabled
And after that sidekiq
and unicorn
failed to start because of database version mismatch:
Database Schema - current: 20190815093949, codebase: 20190828083843
I started investigation the issue and it turned out that now migration-db
uses uploads
field in gitlab.yml
config file. But GitLab chart generates the config file without this field. So I decided to do it manually.
I updated the config file with the following command:
cat <<EOT >> gitlab.yml
uploads:
enabled: true
object_store:
enabled: true
remote_directory: gitlab-uploads
direct_upload: true
background_upload: false
proxy_download: true
connection:
provider: AWS
region: us-east-1
#aws_access_key_id: "<%= File.read('/etc/gitlab/minio/accesskey').strip.dump[1..-2] %>"
#aws_secret_access_key: "<%= File.read('/etc/gitlab/minio/secretkey').strip.dump[1..-2] %>"
aws_access_key_id: "<%= File.read('/srv/gitlab/minio/accesskey').strip.dump[1..-2] %>"
aws_secret_access_key: "<%= File.read('/srv/gitlab/minio/secretkey').strip.dump[1..-2] %>"
host: minio.gitlab.plana.dev
endpoint: http://gitlab-minio-svc:9000
path_style: true
EOT
added the following files with my keys for minio:
/srv/gitlab/minio/accesskey
/srv/gitlab/minio/secretkey
and started migration script manually:
/scripts/db-migrate
After that migration script passed without any issues.
I've also noticed that sidekiq
, unicorn
and migration-db
pods uses latest
as image tag. And I think this was the main reason why I faced this issue. This tags must be associated with version for GilLab in chart.
Steps to reproduce
helm upgrade gitlab ...
Relevant logs
+ /scripts/set-config /var/opt/gitlab/templates /srv/gitlab/config
Begin parsing .erb files from /var/opt/gitlab/templates
Writing /srv/gitlab/config/gitlab.yml
Writing /srv/gitlab/config/database.yml
Writing /srv/gitlab/config/resque.yml
Copying other config files found in /var/opt/gitlab/templates
+ exec /scripts/wait-for-deps /scripts/db-migrate
Checking database connection and schema version
WARNING: This version of GitLab depends on gitlab-shell 9.4.2, but you're running Unknown. Please update gitlab-shell.
Database Schema - current: 20190815093949, codebase: 20190828083843
Checking database migrations are up-to-date
Checking for new migrations
WARNING: This version of GitLab depends on gitlab-shell 9.4.2, but you're running Unknown. Please update gitlab-shell.
rake aborted!
StandardError: An error has occurred, this and all later migrations canceled:
Object Storage is not enabled
/srv/gitlab/app/uploaders/object_storage.rb:394:in `storage_for'
/srv/gitlab/app/uploaders/object_storage.rb:229:in `object_store='
/srv/gitlab/app/uploaders/object_storage.rb:64:in `upload='
/srv/gitlab/app/uploaders/object_storage.rb:49:in `retrieve_from_store!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:35:in `block in uploaders'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:33:in `map'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:33:in `uploaders'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mount.rb:142:in `avatar'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mount.rb:366:in `avatar'
/srv/gitlab/app/models/concerns/avatarable.rb:11:in `block (2 levels) in <module:Avatarable>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:426:in `instance_exec'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:426:in `block in make_lambda'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:178:in `block (2 levels) in halting_and_conditional'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:178:in `all?'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:178:in `block in halting_and_conditional'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `block in invoke_before'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `invoke_before'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:131:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:816:in `_run_validate_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activemodel-5.2.3/lib/active_model/validations.rb:409:in `run_validations!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activemodel-5.2.3/lib/active_model/validations/callbacks.rb:118:in `block in run_validations!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activemodel-0.5.1/lib/state_machines/integrations/active_model.rb:400:in `block in around_validation'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in `block in run_actions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:186:in `within_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activemodel-0.5.1/lib/state_machines/integrations/active_model.rb:400:in `around_validation'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:136:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:816:in `_run_validation_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activemodel-5.2.3/lib/active_model/validations/callbacks.rb:118:in `run_validations!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activemodel-5.2.3/lib/active_model/validations.rb:339:in `valid?'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/validations.rb:67:in `valid?'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/validations.rb:84:in `perform_validations'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/validations.rb:46:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block (2 levels) in save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block in save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:325:in `rollback_active_record_state!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:309:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/suppressor.rb:44:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `block in save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `block in around_save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in `block in run_actions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:186:in `within_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `around_save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `save'
/srv/gitlab/app/models/user.rb:1430:in `update_two_factor_requirement'
/srv/gitlab/app/models/members/group_member.rb:10:in `update_two_factor_requirement'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:426:in `block in make_lambda'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:236:in `block in halting_and_conditional'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:517:in `block in invoke_after'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:517:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:517:in `invoke_after'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:133:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:816:in `_run_create_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/callbacks.rb:346:in `_create_record'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/timestamp.rb:102:in `_create_record'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/persistence.rb:705:in `create_or_update'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/callbacks.rb:342:in `block in create_or_update'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:132:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:816:in `_run_save_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/callbacks.rb:342:in `create_or_update'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/persistence.rb:275:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/validations.rb:46:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block (2 levels) in save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block in save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:325:in `rollback_active_record_state!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:309:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/suppressor.rb:44:in `save'
/srv/gitlab/app/models/member.rb:184:in `add_user'
/srv/gitlab/app/models/member.rb:210:in `block (2 levels) in add_users'
/srv/gitlab/app/models/member.rb:209:in `map!'
/srv/gitlab/app/models/member.rb:209:in `block in add_users'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'
/srv/gitlab/app/models/member.rb:208:in `add_users'
/srv/gitlab/app/models/group.rb:200:in `add_users'
/srv/gitlab/lib/gitlab/database_importers/self_monitoring/project/create_service.rb:115:in `add_group_members'
/srv/gitlab/app/models/concerns/stepable.rb:14:in `call'
/srv/gitlab/app/models/concerns/stepable.rb:14:in `block in execute_steps'
/srv/gitlab/app/models/concerns/stepable.rb:13:in `each'
/srv/gitlab/app/models/concerns/stepable.rb:13:in `inject'
/srv/gitlab/app/models/concerns/stepable.rb:13:in `execute_steps'
/srv/gitlab/lib/gitlab/database_importers/self_monitoring/project/create_service.rb:32:in `execute!'
/srv/gitlab/db/post_migrate/20190801072937_add_gitlab_instance_administration_project.rb:7:in `up'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:817:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:798:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:797:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:796:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:977:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1292:in `block in execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1343:in `block in ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `block in transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1343:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1291:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1263:in `block in migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1262:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1262:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1210:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1363:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1210:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1036:in `up'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1011:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/tasks/database_tasks.rb:172:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/railties/databases.rake:60:in `block (2 levels) in <top (required)>'
/srv/gitlab/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
Caused by:
Object Storage is not enabled
/srv/gitlab/app/uploaders/object_storage.rb:394:in `storage_for'
/srv/gitlab/app/uploaders/object_storage.rb:229:in `object_store='
/srv/gitlab/app/uploaders/object_storage.rb:64:in `upload='
/srv/gitlab/app/uploaders/object_storage.rb:49:in `retrieve_from_store!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:35:in `block in uploaders'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:33:in `map'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mounter.rb:33:in `uploaders'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mount.rb:142:in `avatar'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/carrierwave-1.3.1/lib/carrierwave/mount.rb:366:in `avatar'
/srv/gitlab/app/models/concerns/avatarable.rb:11:in `block (2 levels) in <module:Avatarable>'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:426:in `instance_exec'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:426:in `block in make_lambda'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:178:in `block (2 levels) in halting_and_conditional'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:178:in `all?'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:178:in `block in halting_and_conditional'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `block in invoke_before'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:513:in `invoke_before'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:131:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:816:in `_run_validate_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activemodel-5.2.3/lib/active_model/validations.rb:409:in `run_validations!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activemodel-5.2.3/lib/active_model/validations/callbacks.rb:118:in `block in run_validations!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:109:in `block in run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activemodel-0.5.1/lib/state_machines/integrations/active_model.rb:400:in `block in around_validation'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in `block in run_actions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:186:in `within_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activemodel-0.5.1/lib/state_machines/integrations/active_model.rb:400:in `around_validation'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:118:in `block in run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:136:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:816:in `_run_validation_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activemodel-5.2.3/lib/active_model/validations/callbacks.rb:118:in `run_validations!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activemodel-5.2.3/lib/active_model/validations.rb:339:in `valid?'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/validations.rb:67:in `valid?'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/validations.rb:84:in `perform_validations'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/validations.rb:46:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block (2 levels) in save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block in save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:325:in `rollback_active_record_state!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:309:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/suppressor.rb:44:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `block in save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `block in around_save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:150:in `block in run_actions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:170:in `catch_exceptions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:148:in `run_actions'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:133:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:212:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block (2 levels) in perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `catch'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:63:in `block in perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:186:in `within_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-0.5.0/lib/state_machines/transition_collection.rb:62:in `perform'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:528:in `around_save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/state_machines-activerecord-0.5.1/lib/state_machines/integrations/active_record.rb:509:in `save'
/srv/gitlab/app/models/user.rb:1430:in `update_two_factor_requirement'
/srv/gitlab/app/models/members/group_member.rb:10:in `update_two_factor_requirement'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:426:in `block in make_lambda'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:236:in `block in halting_and_conditional'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:517:in `block in invoke_after'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:517:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:517:in `invoke_after'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:133:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:816:in `_run_create_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/callbacks.rb:346:in `_create_record'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/timestamp.rb:102:in `_create_record'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/persistence.rb:705:in `create_or_update'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/callbacks.rb:342:in `block in create_or_update'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:132:in `run_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activesupport-5.2.3/lib/active_support/callbacks.rb:816:in `_run_save_callbacks'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/callbacks.rb:342:in `create_or_update'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/persistence.rb:275:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/validations.rb:46:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block (2 levels) in save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:387:in `block in with_transaction_returning_status'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:385:in `with_transaction_returning_status'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:310:in `block in save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:325:in `rollback_active_record_state!'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:309:in `save'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/suppressor.rb:44:in `save'
/srv/gitlab/app/models/member.rb:184:in `add_user'
/srv/gitlab/app/models/member.rb:210:in `block (2 levels) in add_users'
/srv/gitlab/app/models/member.rb:209:in `map!'
/srv/gitlab/app/models/member.rb:209:in `block in add_users'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:265:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'
/srv/gitlab/app/models/member.rb:208:in `add_users'
/srv/gitlab/app/models/group.rb:200:in `add_users'
/srv/gitlab/lib/gitlab/database_importers/self_monitoring/project/create_service.rb:115:in `add_group_members'
/srv/gitlab/app/models/concerns/stepable.rb:14:in `call'
/srv/gitlab/app/models/concerns/stepable.rb:14:in `block in execute_steps'
/srv/gitlab/app/models/concerns/stepable.rb:13:in `each'
/srv/gitlab/app/models/concerns/stepable.rb:13:in `inject'
/srv/gitlab/app/models/concerns/stepable.rb:13:in `execute_steps'
/srv/gitlab/lib/gitlab/database_importers/self_monitoring/project/create_service.rb:32:in `execute!'
/srv/gitlab/db/post_migrate/20190801072937_add_gitlab_instance_administration_project.rb:7:in `up'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:817:in `exec_migration'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:798:in `block (2 levels) in migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:797:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/connection_pool.rb:416:in `with_connection'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:796:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:977:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1292:in `block in execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1343:in `block in ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `block in transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:239:in `block in within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/transaction.rb:236:in `within_new_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/connection_adapters/abstract/database_statements.rb:267:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/transactions.rb:212:in `transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1343:in `ddl_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1291:in `execute_migration_in_transaction'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1263:in `block in migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1262:in `each'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1262:in `migrate_without_lock'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1210:in `block in migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1363:in `with_advisory_lock'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1210:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1036:in `up'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/migration.rb:1011:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/tasks/database_tasks.rb:172:in `migrate'
/srv/gitlab/vendor/bundle/ruby/2.6.0/gems/activerecord-5.2.3/lib/active_record/railties/databases.rake:60:in `block (2 levels) in <top (required)>'
/srv/gitlab/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
Tasks: TOP => db:migrate
(See full trace by running task with --trace)
== 20190801072937 AddGitlabInstanceAdministrationProject: migrating ===========
Storing Migration Status