DB migration failure during upgrade to 10.1

Summary

Upgrading from CE 10.0 to CE 10.1 results in a PG migration error.

Steps to reproduce

Server: Debian 9.2
Installation: Omnibus GitLab CE

Update from CE 10.0 to CE 10.1 (apt-get update && apt-get upgrade)

Running gitlab-ctl reconfigure after upgrade also fails when migrating database.

What is the current bug behavior?

Recipe gitlab:database_migrations fails

Relevant logs and/or screenshots

Update console output

Preparing to unpack .../gitlab-ce_10.1.0-ce.0_amd64.deb ... gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!) Dumping database ... Dumping PostgreSQL database gitlabhq_production ... [DONE] done Dumping repositories ... [SKIPPED] Dumping uploads ... [SKIPPED] Dumping builds ... [SKIPPED] Dumping artifacts ... [SKIPPED] Dumping pages ... [SKIPPED] Dumping lfs objects ... [SKIPPED] Dumping container registry images ... [SKIPPED] Creating backup archive: 1508658964_2017_10_22_10.0.4_gitlab_backup.tar ... done Uploading backup archive to remote storage ... skipped Deleting tmp directories ... done done Deleting old backups ... skipping Unpacking gitlab-ce (10.1.0-ce.0) over (10.0.4-ce.0) ... Setting up gitlab-ce (10.1.0-ce.0) ... Checking PostgreSQL executables:Starting Chef Client, version 12.12.15 resolving cookbooks for run list: ["gitlab::postgresql-bin"] Synchronizing Cookbooks: - gitlab (0.0.1) - registry (0.1.0) - package (0.1.0) - runit (0.14.2) - consul (0.0.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 1 resources Recipe: gitlab::postgresql-bin * ruby_block[Link postgresql bin files to the correct version] action run (skipped due to only_if)

Running handlers: Running handlers complete Chef Client finished, 0/1 resources updated in 02 seconds Checking PostgreSQL executables: OK Shutting down all GitLab services except those needed for migrations ok: down: gitaly: 0s, normally up ok: down: gitlab-monitor: 0s, normally up ok: down: gitlab-workhorse: 1s, normally up ok: down: logrotate: 0s, normally up ok: down: nginx: 1s, normally up ok: down: node-exporter: 0s, normally up ok: down: postgres-exporter: 0s, normally up ok: down: postgresql: 0s, normally up ok: down: prometheus: 0s, normally up ok: down: redis: 1s, normally up ok: down: redis-exporter: 0s, normally up ok: down: registry: 1s, normally up ok: down: sidekiq: 0s, normally up ok: down: unicorn: 0s, normally up ok: run: postgresql: (pid 19494) 1s ok: run: redis: (pid 19502) 0s run: postgresql: (pid 19494) 1s; run: log: (pid 247) 75247s run: redis: (pid 19502) 0s; run: log: (pid 243) 75247s Reconfiguring GitLab to apply migrations Starting Chef Client, version 12.12.15 resolving cookbooks for run list: ["gitlab"] Synchronizing Cookbooks:

gitlab (0.0.1)
package (0.1.0)
registry (0.1.0)
consul (0.0.0)
runit (0.14.2) Installing Cookbook Gems: Compiling Cookbooks... Recipe: gitlab::default

directory[/etc/gitlab] action create (up to date) Converging 459 resources
directory[/etc/gitlab] action create (up to date)
directory[Create /var/opt/gitlab] action create (up to date)
directory[/opt/gitlab/embedded/etc] action create (up to date)
template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date) Recipe: gitlab::web-server
group[Webserver user and group] action create (up to date)
user[Webserver user and group] action create (up to date) Recipe: gitlab::users
directory[/var/opt/gitlab] action create (up to date)
group[GitLab user and group] action create (up to date)
user[GitLab user and group] action create (up to date)
template[/var/opt/gitlab/.gitconfig] action create (up to date) Recipe: gitlab::gitlab-shell
ruby_block[directory resource: /var/opt/gitlab/git-data] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/.ssh] action run (skipped due to not_if)
directory[/var/log/gitlab/gitlab-shell/] action create (up to date)
directory[/var/opt/gitlab/gitlab-shell] action create (up to date)
templatesymlink[Create a config.yml and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml] action create (up to date) (up to date)
link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create (up to date)
execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] action run
- execute /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions
bash[Set proper security context on ssh files for selinux] action run (skipped due to only_if) Recipe: gitlab::gitlab-rails
directory[/var/log/gitlab] action create (up to date)
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/artifacts] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/lfs-objects] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/uploads] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/gitlab-ci/builds] action run (skipped due to not_if)
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/pages] action run (skipped due to not_if)
directory[create /var/opt/gitlab/gitlab-rails/etc] action create (up to date)
directory[create /opt/gitlab/etc/gitlab-rails] action create (up to date)
directory[create /var/opt/gitlab/gitlab-rails/working] action create (up to date)
directory[create /var/opt/gitlab/gitlab-rails/tmp] action create (up to date)
directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] action create (up to date)
directory[create /var/log/gitlab/gitlab-rails] action create (up to date)
directory[/var/opt/gitlab/backups] action create (up to date)
directory[/var/opt/gitlab/gitlab-rails] action create (up to date)
directory[/var/opt/gitlab/gitlab-ci] action create (up to date)
file[/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key] action create (up to date)
template[/opt/gitlab/etc/gitlab-rails/gitlab-rails-rc] action create (up to date)
file[/opt/gitlab/embedded/service/gitlab-rails/.secret] action delete (up to date)
file[/var/opt/gitlab/gitlab-rails/etc/secret] action delete (up to date)
templatesymlink[Create a database.yml and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/database.yml to /var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date) (up to date)
templatesymlink[Create a secrets.yml and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create
  - update content in file /var/opt/gitlab/gitlab-rails/etc/secrets.yml from 8c4134 to a49b15
  - suppressed sensitive resource
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml to /var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
templatesymlink[Create a resque.yml and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/resque.yml to /var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date) (up to date)
templatesymlink[Create a aws.yml and create a symlink to Rails root] action delete
- template[/var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/aws.yml to /var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date) (up to date)
templatesymlink[Create a smtp_settings.rb and create a symlink to Rails root] action delete
- template[/var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb to /var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date) (up to date)
templatesymlink[Create a gitlab.yml and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create
  - update content in file /var/opt/gitlab/gitlab-rails/etc/gitlab.yml from 78284b to af6ebc --- /var/opt/gitlab/gitlab-rails/etc/gitlab.yml 2017-10-21 06:30:32.703349506 -0400 +++ /var/opt/gitlab/gitlab-rails/etc/.chef-gitlab.yml20171022-19515-1vu0ulf 2017-10-22 03:57:03.773887936 -0400 @@ -159,6 +159,7 @@ https: false external_http: null external_https: null
  - artifacts_server: true
  Gravatar
  
  For Libravatar see: https://docs.gitlab.com/ce/customization/libravatar.html
  
  @@ -350,6 +351,7 @@
  
  Eventually Gitaly use will become mandatory and
  
  this option will disappear.
  
  gitaly:
  - client_path: /opt/gitlab/embedded/bin token: ""
  @@ -398,12 +400,6 @@
  
  Use the default values unless you really know what you are doing
  
  git: bin_path: /opt/gitlab/embedded/bin/git
  - The next value is the maximum memory size grit can use
  - Given in number of bytes per git object (e.g. a commit)
  - This value can be increased if you have very large commits
  - max_size:
  - Git timeout to read a commit, in seconds
  - timeout:
  GitLab Geo settings (EE-only)
  
  geo_primary_role:
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml to /var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
templatesymlink[Create a rack_attack.rb and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/rack_attack.rb to /var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date) (up to date)
templatesymlink[Create a gitlab_workhorse_secret and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date) (up to date)
templatesymlink[Create a gitlab_shell_secret and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date) (up to date)
env_dir[/opt/gitlab/etc/gitlab-rails/env] action create
- directory[/opt/gitlab/etc/gitlab-rails/env] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/HOME] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/LD_PRELOAD] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/PATH] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/ICU_DATA] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/PYTHONPATH] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME] action create (up to date) (up to date)
link[/opt/gitlab/embedded/service/gitlab-rails/tmp] action create (up to date)
link[/opt/gitlab/embedded/service/gitlab-rails/public/uploads] action create (up to date)
link[/opt/gitlab/embedded/service/gitlab-rails/log] action create (up to date)
link[/var/log/gitlab/gitlab-rails/sidekiq.log] action create (skipped due to not_if)
file[/opt/gitlab/embedded/service/gitlab-rails/db/schema.rb] action create
- change owner from 'root' to 'git'
remote_file[/var/opt/gitlab/gitlab-rails/VERSION] action create
- update content in file /var/opt/gitlab/gitlab-rails/VERSION from de756b to 08bc77 --- /var/opt/gitlab/gitlab-rails/VERSION 2017-10-18 12:07:39.785539202 -0400 +++ /var/opt/gitlab/gitlab-rails/.chef-VERSION20171022-19515-swrrrv 2017-10-22 03:57:03.876890383 -0400 @@ -1,2 +1,2 @@ -10.0.4 +10.1.0
remote_file[/var/opt/gitlab/gitlab-rails/REVISION] action create
- update content in file /var/opt/gitlab/gitlab-rails/REVISION from 2e8a01 to faa5e7 --- /var/opt/gitlab/gitlab-rails/REVISION 2017-10-18 12:07:39.802539606 -0400 +++ /var/opt/gitlab/gitlab-rails/.chef-REVISION20171022-19515-1sv3k4p 2017-10-22 03:57:03.880890477 -0400 @@ -1,2 +1,2 @@ -8ebd2dd9 +5a695c4c
file[/var/opt/gitlab/gitlab-rails/RUBY_VERSION] action create (up to date)
execute[chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public] action run
- execute chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public
execute[clear the gitlab-rails cache] action nothing (skipped due to action :nothing)
file[/var/opt/gitlab/gitlab-rails/config.ru] action delete (up to date) Recipe: gitlab::add_trusted_certs
directory[/etc/gitlab/trusted-certs] action create (up to date)
directory[/opt/gitlab/embedded/ssl/certs] action create (up to date)
file[/opt/gitlab/embedded/ssl/certs/README] action create (up to date)
ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run
Moving existing certificates found in /opt/gitlab/embedded/ssl/certs
Symlinking existing certificates found in /etc/gitlab/trusted-certs
- execute the ruby block Move existing certs and link to /opt/gitlab/embedded/ssl/certs Recipe: gitlab::default
service[create a temporary unicorn service] action nothing (skipped due to action :nothing)
service[create a temporary sidekiq service] action nothing (skipped due to action :nothing)
service[create a temporary mailroom service] action nothing (skipped due to action :nothing) Recipe: runit::systemd
directory[/usr/lib/systemd/system] action create (up to date)
cookbook_file[/usr/lib/systemd/system/gitlab-runsvdir.service] action create (up to date)
file[/etc/systemd/system/default.target.wants/gitlab-runsvdir.service] action delete (up to date)
execute[systemctl daemon-reload] action nothing (skipped due to action :nothing)
execute[systemctl enable gitlab-runsvdir] action nothing (skipped due to action :nothing)
execute[systemctl start gitlab-runsvdir] action nothing (skipped due to action :nothing) Recipe: gitlab::redis
group[user and group for redis] action create (up to date)
user[user and group for redis] action create (up to date)
group[Socket group] action create (up to date)
directory[/var/opt/gitlab/redis] action create (up to date)
directory[/var/log/gitlab/redis] action create (up to date)
template[/var/opt/gitlab/redis/redis.conf] action create (up to date)
directory[/opt/gitlab/sv/redis] action create (up to date)
directory[/opt/gitlab/sv/redis/log] action create (up to date)
directory[/opt/gitlab/sv/redis/log/main] action create (up to date)
template[/opt/gitlab/sv/redis/run] action create (up to date)
template[/opt/gitlab/sv/redis/log/run] action create (up to date)
template[/var/log/gitlab/redis/config] action create (up to date)
ruby_block[reload redis svlogd configuration] action nothing (skipped due to action :nothing)
file[/opt/gitlab/sv/redis/down] action delete (up to date)
link[/opt/gitlab/init/redis] action create (up to date)
link[/opt/gitlab/service/redis] action create (up to date)
ruby_block[supervise_redis_sleep] action run (skipped due to not_if)
directory[/opt/gitlab/sv/redis/supervise] action create (up to date)
directory[/opt/gitlab/sv/redis/log/supervise] action create (up to date)
file[/opt/gitlab/sv/redis/supervise/ok] action touch (skipped due to only_if)
file[/opt/gitlab/sv/redis/log/supervise/ok] action touch (skipped due to only_if)
file[/opt/gitlab/sv/redis/supervise/control] action touch (skipped due to only_if)
file[/opt/gitlab/sv/redis/log/supervise/control] action touch (skipped due to only_if)
service[redis] action nothing (skipped due to action :nothing) Recipe: gitlab::postgresql_user
group[Postgresql user and group] action create (up to date)
user[Postgresql user and group] action create (up to date) Recipe: gitlab::postgresql
directory[/var/opt/gitlab/postgresql] action create (up to date)
directory[/var/opt/gitlab/postgresql/data] action create (up to date)
directory[/var/log/gitlab/postgresql] action create (up to date)
link[/var/opt/gitlab/postgresql/data] action create (skipped due to not_if)
file[/var/opt/gitlab/postgresql/.profile] action create (up to date)
directory[create /etc/sysctl.d for kernel.shmmax] action create (up to date)
file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf kernel.shmmax] action create (up to date)
link[/etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf] action create (up to date)
file[delete /etc/sysctl.d/90-postgresql.conf kernel.shmmax] action delete (skipped due to only_if)
file[delete /etc/sysctl.d/90-unicorn.conf kernel.shmmax] action delete (skipped due to only_if)
file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.shmmax] action delete (skipped due to only_if)
file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.shmmax] action delete (skipped due to only_if)
execute[load sysctl conf kernel.shmmax] action nothing (skipped due to action :nothing)
directory[create /etc/sysctl.d for kernel.shmall] action create (up to date)
file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf kernel.shmall] action create (up to date)
link[/etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf] action create (up to date)
file[delete /etc/sysctl.d/90-postgresql.conf kernel.shmall] action delete (skipped due to only_if)
file[delete /etc/sysctl.d/90-unicorn.conf kernel.shmall] action delete (skipped due to only_if)
file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.shmall] action delete (skipped due to only_if)
file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.shmall] action delete (skipped due to only_if)
execute[load sysctl conf kernel.shmall] action nothing (skipped due to action :nothing)
directory[create /etc/sysctl.d for kernel.sem] action create (up to date)
file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.sem.conf kernel.sem] action create (up to date)
link[/etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf] action create (up to date)
file[delete /etc/sysctl.d/90-postgresql.conf kernel.sem] action delete (skipped due to only_if)
file[delete /etc/sysctl.d/90-unicorn.conf kernel.sem] action delete (skipped due to only_if)
file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.sem] action delete (skipped due to only_if)
file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.sem] action delete (skipped due to only_if)
execute[load sysctl conf kernel.sem] action nothing (skipped due to action :nothing)
execute[/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8] action run (skipped due to not_if)
template[/var/opt/gitlab/postgresql/data/postgresql.conf] action create (up to date)
template[/var/opt/gitlab/postgresql/data/runtime.conf] action create
- update content in file /var/opt/gitlab/postgresql/data/runtime.conf from 65bed2 to 18dd18 --- /var/opt/gitlab/postgresql/data/runtime.conf 2017-08-24 14:51:31.000000000 -0400 +++ /var/opt/gitlab/postgresql/data/.chef-runtime.conf20171022-19515-ucb2n7 2017-10-22 03:57:04.083895296 -0400 @@ -101,6 +101,18 @@
idle_in_transaction_session_timeout = 60000

+# IO settings +effective_io_concurrency = 1 +track_io_timing = 'off' + +# Parallel worker settings +max_worker_processes = 8 +max_parallel_workers_per_gather = 0 + +# Deadlock handling and logging +deadlock_timeout = '5s' +log_lock_waits = 1 +

- Locale and Formatting -

datestyle = 'iso, mdy'
execute[reload postgresql] action run
- execute /opt/gitlab/bin/gitlab-ctl hup postgresql
execute[start postgresql] action run (skipped due to not_if)
template[/var/opt/gitlab/postgresql/data/pg_hba.conf] action create (up to date)
template[/var/opt/gitlab/postgresql/data/pg_ident.conf] action create (up to date)
directory[/opt/gitlab/sv/postgresql] action create (up to date)
directory[/opt/gitlab/sv/postgresql/log] action create (up to date)
directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date)
template[/opt/gitlab/sv/postgresql/run] action create (up to date)
template[/opt/gitlab/sv/postgresql/log/run] action create (up to date)
template[/var/log/gitlab/postgresql/config] action create (up to date)
ruby_block[reload postgresql svlogd configuration] action nothing (skipped due to action :nothing)
file[/opt/gitlab/sv/postgresql/down] action delete (up to date)
directory[/opt/gitlab/sv/postgresql/control] action create (up to date)
template[/opt/gitlab/sv/postgresql/control/t] action create (up to date)
link[/opt/gitlab/init/postgresql] action create (up to date)
link[/opt/gitlab/service/postgresql] action create (up to date)
ruby_block[supervise_postgresql_sleep] action run (skipped due to not_if)
directory[/opt/gitlab/sv/postgresql/supervise] action create (up to date)
directory[/opt/gitlab/sv/postgresql/log/supervise] action create (up to date)
file[/opt/gitlab/sv/postgresql/supervise/ok] action touch (skipped due to only_if)
file[/opt/gitlab/sv/postgresql/log/supervise/ok] action touch (skipped due to only_if)
file[/opt/gitlab/sv/postgresql/supervise/control] action touch (skipped due to only_if)
file[/opt/gitlab/sv/postgresql/log/supervise/control] action touch (skipped due to only_if)
service[postgresql] action nothing (skipped due to action :nothing) Recipe: gitlab::postgresql-bin
ruby_block[Link postgresql bin files to the correct version] action run (skipped due to only_if) Recipe: gitlab::postgresql
template[/opt/gitlab/etc/gitlab-psql-rc] action create (up to date)
postgresql_user[gitlab] action create
- execute[create gitlab postgresql user] action run (skipped due to not_if) (up to date)
execute[create gitlabhq_production database] action run (skipped due to not_if)
postgresql_user[gitlab_replicator] action create
- execute[create gitlab_replicator postgresql user] action run (skipped due to not_if) (up to date)
execute[enable pg_trgm extension] action nothing (skipped due to action :nothing)
execute[reload postgresql] action nothing (skipped due to action :nothing)
execute[start postgresql] action nothing (skipped due to action :nothing) Recipe: gitlab::database_migrations
bash[migrate gitlab-rails database] action run
- Would execute "bash" "/tmp/chef-script20171022-19515-uc00fh" Recipe: gitlab::postgresql
execute[enable pg_trgm extension] action run (skipped due to not_if) Recipe: gitlab::database_migrations
bash[migrate gitlab-rails database] action run [execute] rake aborted! StandardError: An error has occurred, all later migrations canceled:
```
      PG::DuplicateColumn: ERROR:  column "merge_requests_ff_only_enabled" of relation "projects" already exists
      : ALTER TABLE "projects" ADD "merge_requests_ff_only_enabled" boolean
      /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in `add_column'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in `block in add_column_with_default'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:in `add_column_with_default'
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      ActiveRecord::StatementInvalid: PG::DuplicateColumn: ERROR:  column "merge_requests_ff_only_enabled" of relation "projects" already exists
      : ALTER TABLE "projects" ADD "merge_requests_ff_only_enabled" boolean
      /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in `add_column'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in `block in add_column_with_default'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:in `add_column_with_default'
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      PG::DuplicateColumn: ERROR:  column "merge_requests_ff_only_enabled" of relation "projects" already exists
      /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in `add_column'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in `block in add_column_with_default'
      /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:in `add_column_with_default'
      /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in `up'
      /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>'
      /opt/gitlab/embedded/bin/bundle:23:in `load'
      /opt/gitlab/embedded/bin/bundle:23:in `<main>'
      Tasks: TOP => db:migrate
      (See full trace by running task with --trace)
      == 20141126120926 AddMergeRequestRebaseEnabledToProjects: migrating ===========
      -- transaction_open?()
         -> 0.0000s
      -- execute("SET statement_timeout TO 0")
         -> 0.0004s
      -- transaction()
      -- add_column(:projects, :merge_requests_rebase_enabled, :boolean, {:default=>nil})
         -> 0.0020s
      -- change_column_default(:projects, :merge_requests_rebase_enabled, false)
         -> 0.0037s
         -> 0.0065s
      -- transaction_open?()
         -> 0.0000s
      -- exec_query("SELECT COUNT(*) AS count FROM \"projects\"")
         -> 0.0021s
      -- exec_query("SELECT  \"projects\".\"id\" FROM \"projects\"  ORDER BY \"projects\".\"id\" ASC LIMIT 1")
         -> 0.0005s
      -- exec_query("SELECT  \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 1  ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1")
         -> 0.0004s
      -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 1 AND \"projects\".\"id\" < 2")
         -> 0.0010s
      -- exec_query("SELECT  \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 2  ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1")
         -> 0.0003s
      -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 2 AND \"projects\".\"id\" < 3")
         -> 0.0008s
      -- exec_query("SELECT  \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 3  ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1")
         -> 0.0003s
      -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 3 AND \"projects\".\"id\" < 4")
         -> 0.0004s
      -- exec_query("SELECT  \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 4  ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1")
         -> 0.0003s
      -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 4 AND \"projects\".\"id\" < 5")
         -> 0.0004s
      -- exec_query("SELECT  \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 5  ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1")
         -> 0.0003s
      -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 5 AND \"projects\".\"id\" < 6")
         -> 0.0004s
      -- exec_query("SELECT  \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 6  ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1")
         -> 0.0003s
      -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 6 AND \"projects\".\"id\" < 7")
         -> 0.0004s
      -- exec_query("SELECT  \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 7  ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1")
         -> 0.0003s
      -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 7 AND \"projects\".\"id\" < 8")
         -> 0.0005s
      -- exec_query("SELECT  \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 8  ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1")
         -> 0.0003s
      -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 8")
         -> 0.0004s
      -- change_column_null(:projects, :merge_requests_rebase_enabled, false)
         -> 0.0006s
      == 20141126120926 AddMergeRequestRebaseEnabledToProjects: migrated (0.0243s) ==
      
      == 20150827121444 AddFastForwardOptionToProject: migrating ====================
      -- transaction_open?()
         -> 0.0000s
      -- execute("SET statement_timeout TO 0")
         -> 0.0002s
      -- transaction()
      -- add_column(:projects, :merge_requests_ff_only_enabled, :boolean, {:default=>nil})
```
================================================================================ Error executing action run on resource 'bash[migrate gitlab-rails database]'

Mixlib::ShellOut::ShellCommandFailed

Expected process to exit with [0], but received '1' ---- Begin output of "bash" "/tmp/chef-script20171022-19515-ef47vt" ---- STDOUT: rake aborted! StandardError: An error has occurred, all later migrations canceled:

PG::DuplicateColumn: ERROR: column "merge_requests_ff_only_enabled" of relation "projects" already exists : ALTER TABLE "projects" ADD "merge_requests_ff_only_enabled" boolean /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in add_column' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in block in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in <main>' ActiveRecord::StatementInvalid: PG::DuplicateColumn: ERROR: column "merge_requests_ff_only_enabled" of relation "projects" already exists : ALTER TABLE "projects" ADD "merge_requests_ff_only_enabled" boolean /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in add_column' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in block in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in ' PG::DuplicateColumn: ERROR: column "merge_requests_ff_only_enabled" of relation "projects" already exists /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in add_column' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in block in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in load' /opt/gitlab/embedded/bin/bundle:23:in `' Tasks: TOP => db:migrate (See full trace by running task with --trace) == 20141126120926 AddMergeRequestRebaseEnabledToProjects: migrating =========== -- transaction_open?() -> 0.0000s -- execute("SET statement_timeout TO 0") -> 0.0004s -- transaction() -- add_column(:projects, :merge_requests_rebase_enabled, :boolean, {:default=>nil}) -> 0.0020s -- change_column_default(:projects, :merge_requests_rebase_enabled, false) -> 0.0037s -> 0.0065s -- transaction_open?() -> 0.0000s -- exec_query("SELECT COUNT(*) AS count FROM "projects"") -> 0.0021s -- exec_query("SELECT "projects"."id" FROM "projects" ORDER BY "projects"."id" ASC LIMIT 1") -> 0.0005s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 1 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0004s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 1 AND "projects"."id" < 2") -> 0.0010s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 2 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 2 AND "projects"."id" < 3") -> 0.0008s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 3 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 3 AND "projects"."id" < 4") -> 0.0004s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 4 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 4 AND "projects"."id" < 5") -> 0.0004s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 5 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 5 AND "projects"."id" < 6") -> 0.0004s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 6 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 6 AND "projects"."id" < 7") -> 0.0004s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 7 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 7 AND "projects"."id" < 8") -> 0.0005s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 8 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 8") -> 0.0004s -- change_column_null(:projects, :merge_requests_rebase_enabled, false) -> 0.0006s == 20141126120926 AddMergeRequestRebaseEnabledToProjects: migrated (0.0243s) ==

== 20150827121444 AddFastForwardOptionToProject: migrating ==================== -- transaction_open?() -> 0.0000s -- execute("SET statement_timeout TO 0") -> 0.0002s -- transaction() -- add_column(:projects, :merge_requests_ff_only_enabled, :boolean, {:default=>nil}) STDERR: ---- End output of "bash" "/tmp/chef-script20171022-19515-ef47vt" ---- Ran "bash" "/tmp/chef-script20171022-19515-ef47vt" returned 1

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

51: bash "migrate gitlab-rails database" do 52: code <<-EOH 53: set -e 54: log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log" 55: umask 077 56: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file} 57: STATUS=${PIPESTATUS[0]} 58: echo $STATUS > #{db_migrate_status_file} 59: exit $STATUS 60: EOH 61: environment env_variables unless env_variables.empty? 62: notifies :run, 'execute[enable pg_trgm extension]', :before if omnibus_helper.service_enabled?('postgresql') 63: notifies :run, "execute[clear the gitlab-rails cache]", :immediately 64: dependent_services.each do |svc| 65: notifies :restart, svc, :immediately 66: end 67: not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)" 68: only_if { node['gitlab']['gitlab-rails']['auto_migrate'] } 69: end

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'

bash("migrate gitlab-rails database") do action [:run] updated true retries 0 retry_delay 2 default_guard_interpreter :default command "migrate gitlab-rails database" backup 5 returns 0 code " set -e\n log_file="/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee {log_file}\n STATUS={PIPESTATUS[0]}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-5a695c4c\n exit $STATUS\n" interpreter "bash" declared_type :bash cookbook_name "gitlab" recipe_name "database_migrations" not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-5a695c4c) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-5a695c4c | grep -Fx 0)" only_if { #code block } end

Platform:

x86_64-linux

Recipe: gitlab::gitlab-rails

execute[clear the gitlab-rails cache] action run
- execute /opt/gitlab/bin/gitlab-rake cache:clear

Running handlers: Running handlers complete Chef Client failed. 14 resources updated in 35 seconds Checking for an omnibus managed postgresql: OK Checking for a newer version of PostgreSQL to install No new version of PostgreSQL installed, nothing to upgrade to Ensuring PostgreSQL is updated: OK Restarting previously running GitLab services ok: run: gitaly: (pid 20049) 1s ok: run: gitlab-monitor: (pid 20060) 0s ok: run: gitlab-workhorse: (pid 20071) 1s ok: run: logrotate: (pid 20095) 0s ok: run: nginx: (pid 20120) 0s ok: run: node-exporter: (pid 20128) 1s ok: run: postgres-exporter: (pid 20177) 0s ok: run: postgresql: (pid 19494) 47s ok: run: prometheus: (pid 20185) 0s ok: run: redis: (pid 19502) 46s ok: run: redis-exporter: (pid 20188) 1s ok: run: sidekiq: (pid 20202) 0s ok: run: unicorn: (pid 20209) 0s

 _______ __  __          __
/ ____(_) /_/ /   ____ _/ /_

/ / / / / / / __ `/ __
/ // / / // // // / // / ___//_/__/__,/./

Upgrade complete! If your GitLab server is misbehaving try running sudo gitlab-ctl restart before anything else. If you need to roll back to the previous version you can use the database backup made during the upgrade (scroll up for the filename).

Results of GitLab environment info

Expand for output related to GitLab environment info

System information System: Debian 9.2 Current User: git Using RVM: no Ruby Version: 2.3.5p376 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.1.0 Redis Version: 3.2.5 Git Version: 2.13.5 Sidekiq Version:5.0.4 Go Version: unknown

GitLab information Version: 10.1.0 Revision: 5a695c4 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://develop.briankintz.com HTTP Clone URL: https://develop.briankintz.com/some-group/some-project.git SSH Clone URL: git@develop.briankintz.com:some-group/some-project.git Using LDAP: no Using Omniauth: no

GitLab Shell Version: 5.9.3 Repository storage paths:

default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab Shell ...

GitLab Shell version >= 5.9.3 ? ... OK (5.9.3) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 5/1 ... ok 4/2 ... ok 4/3 ... ok 6/4 ... ok 5/5 ... ok 6/6 ... ok 6/7 ... ok 6/8 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK

Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful

Checking GitLab Shell ... Finished

Checking Sidekiq ...

Running? ... yes Number of Sidekiq processes ... 1

Checking Sidekiq ... Finished

Reply by email is disabled in config/gitlab.yml Checking LDAP ...

LDAP is disabled in config/gitlab.yml

Checking LDAP ... Finished

Checking GitLab ...

Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/1 ... yes 4/2 ... yes 4/3 ... yes 6/4 ... yes 5/5 ... yes 6/6 ... yes 6/7 ... yes 6/8 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.5) Git version >= 2.7.3 ? ... yes (2.13.5) Git user has default SSH configuration? ... yes Active users: ... 2

Checking GitLab ... Finished

Edited Oct 22, 2017 by Brian Kintz

DB migration failure during upgrade to 10.1

Summary

Steps to reproduce

What is the current bug behavior?

Relevant logs and/or screenshots

Gravatar

For Libravatar see: https://docs.gitlab.com/ce/customization/libravatar.html

Eventually Gitaly use will become mandatory and

this option will disappear.

Use the default values unless you really know what you are doing

The next value is the maximum memory size grit can use

Given in number of bytes per git object (e.g. a commit)

This value can be increased if you have very large commits

Git timeout to read a commit, in seconds

GitLab Geo settings (EE-only)

- Locale and Formatting -

================================================================================ Error executing action run on resource 'bash[migrate gitlab-rails database]'

Mixlib::ShellOut::ShellCommandFailed

Resource Declaration:

In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb

Compiled Resource:

Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'

Platform:

Results of GitLab environment info

Results of GitLab application Check

================================================================================ Error executing action `run` on resource 'bash[migrate gitlab-rails database]'