Investigate infrequent "advisory lock" upgrade issue on environment builds
Sometimes environment builds fail with the below error when migrating db data. Likely a timing issue in the builder. Task is to investigate and fix.
fatal: [gitlab-qa-10k-gitlab-rails-1]: FAILED! => changed=true
cmd: gitlab-ctl reconfigure
delta: '0:00:26.691282'
end: '2020-02-21 04:27:23.773153'
msg: non-zero return code
rc: 1
start: '2020-02-21 04:26:57.081871'
stderr: |-
There was an error running gitlab-ctl reconfigure:
bash[migrate gitlab-rails database] (gitlab::database_migrations line 54) had an error: Mixlib::ShellOut::ShellCommandFailed: Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20200221-18276-1lmrlc6" ----
STDOUT: WARNING: you don't own a lock of type ExclusiveLock
rake aborted!
ActiveRecord::ConcurrentMigrationError:
Failed to release advisory lock
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:23:in `load'
/opt/gitlab/embedded/bin/bundle:23:in `<main>'
Tasks: TOP => db:migrate