DB migration failure during upgrade to 10.1
Summary
Upgrading from CE 10.0 to CE 10.1 results in a PG migration error.
Steps to reproduce
Server: Debian 9.2
Installation: Omnibus GitLab CE
- Update from CE 10.0 to CE 10.1 (
apt-get update && apt-get upgrade
)
Running gitlab-ctl reconfigure
after upgrade also fails when migrating database.
What is the current bug behavior?
Recipe gitlab:database_migrations
fails
Relevant logs and/or screenshots
Update console output
Preparing to unpack .../gitlab-ce_10.1.0-ce.0_amd64.deb ... gitlab preinstall: Automatically backing up only the GitLab SQL database (excluding everything else!) Dumping database ... Dumping PostgreSQL database gitlabhq_production ... [DONE] done Dumping repositories ... [SKIPPED] Dumping uploads ... [SKIPPED] Dumping builds ... [SKIPPED] Dumping artifacts ... [SKIPPED] Dumping pages ... [SKIPPED] Dumping lfs objects ... [SKIPPED] Dumping container registry images ... [SKIPPED] Creating backup archive: 1508658964_2017_10_22_10.0.4_gitlab_backup.tar ... done Uploading backup archive to remote storage ... skipped Deleting tmp directories ... done done Deleting old backups ... skipping Unpacking gitlab-ce (10.1.0-ce.0) over (10.0.4-ce.0) ... Setting up gitlab-ce (10.1.0-ce.0) ... Checking PostgreSQL executables:Starting Chef Client, version 12.12.15 resolving cookbooks for run list: ["gitlab::postgresql-bin"] Synchronizing Cookbooks: - gitlab (0.0.1) - registry (0.1.0) - package (0.1.0) - runit (0.14.2) - consul (0.0.0) Installing Cookbook Gems: Compiling Cookbooks... Converging 1 resources Recipe: gitlab::postgresql-bin * ruby_block[Link postgresql bin files to the correct version] action run (skipped due to only_if)Running handlers: Running handlers complete Chef Client finished, 0/1 resources updated in 02 seconds Checking PostgreSQL executables: OK Shutting down all GitLab services except those needed for migrations ok: down: gitaly: 0s, normally up ok: down: gitlab-monitor: 0s, normally up ok: down: gitlab-workhorse: 1s, normally up ok: down: logrotate: 0s, normally up ok: down: nginx: 1s, normally up ok: down: node-exporter: 0s, normally up ok: down: postgres-exporter: 0s, normally up ok: down: postgresql: 0s, normally up ok: down: prometheus: 0s, normally up ok: down: redis: 1s, normally up ok: down: redis-exporter: 0s, normally up ok: down: registry: 1s, normally up ok: down: sidekiq: 0s, normally up ok: down: unicorn: 0s, normally up ok: run: postgresql: (pid 19494) 1s ok: run: redis: (pid 19502) 0s run: postgresql: (pid 19494) 1s; run: log: (pid 247) 75247s run: redis: (pid 19502) 0s; run: log: (pid 243) 75247s Reconfiguring GitLab to apply migrations Starting Chef Client, version 12.12.15 resolving cookbooks for run list: ["gitlab"] Synchronizing Cookbooks:
- gitlab (0.0.1)
- package (0.1.0)
- registry (0.1.0)
- consul (0.0.0)
- runit (0.14.2) Installing Cookbook Gems: Compiling Cookbooks... Recipe: gitlab::default
-
directory[/etc/gitlab] action create (up to date) Converging 459 resources
-
directory[/etc/gitlab] action create (up to date)
-
directory[Create /var/opt/gitlab] action create (up to date)
-
directory[/opt/gitlab/embedded/etc] action create (up to date)
-
template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date) Recipe: gitlab::web-server
-
group[Webserver user and group] action create (up to date)
-
user[Webserver user and group] action create (up to date) Recipe: gitlab::users
-
directory[/var/opt/gitlab] action create (up to date)
-
group[GitLab user and group] action create (up to date)
-
user[GitLab user and group] action create (up to date)
-
template[/var/opt/gitlab/.gitconfig] action create (up to date) Recipe: gitlab::gitlab-shell
-
ruby_block[directory resource: /var/opt/gitlab/git-data] action run (skipped due to not_if)
-
ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] action run (skipped due to not_if)
-
ruby_block[directory resource: /var/opt/gitlab/.ssh] action run (skipped due to not_if)
-
directory[/var/log/gitlab/gitlab-shell/] action create (up to date)
-
directory[/var/opt/gitlab/gitlab-shell] action create (up to date)
-
templatesymlink[Create a config.yml and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml] action create (up to date) (up to date)
-
link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create (up to date)
-
execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] action run
- execute /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions
-
bash[Set proper security context on ssh files for selinux] action run (skipped due to only_if) Recipe: gitlab::gitlab-rails
-
directory[/var/log/gitlab] action create (up to date)
-
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared] action run (skipped due to not_if)
-
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/artifacts] action run (skipped due to not_if)
-
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/lfs-objects] action run (skipped due to not_if)
-
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/uploads] action run (skipped due to not_if)
-
ruby_block[directory resource: /var/opt/gitlab/gitlab-ci/builds] action run (skipped due to not_if)
-
ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/pages] action run (skipped due to not_if)
-
directory[create /var/opt/gitlab/gitlab-rails/etc] action create (up to date)
-
directory[create /opt/gitlab/etc/gitlab-rails] action create (up to date)
-
directory[create /var/opt/gitlab/gitlab-rails/working] action create (up to date)
-
directory[create /var/opt/gitlab/gitlab-rails/tmp] action create (up to date)
-
directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] action create (up to date)
-
directory[create /var/log/gitlab/gitlab-rails] action create (up to date)
-
directory[/var/opt/gitlab/backups] action create (up to date)
-
directory[/var/opt/gitlab/gitlab-rails] action create (up to date)
-
directory[/var/opt/gitlab/gitlab-ci] action create (up to date)
-
file[/var/opt/gitlab/gitlab-rails/etc/gitlab-registry.key] action create (up to date)
-
template[/opt/gitlab/etc/gitlab-rails/gitlab-rails-rc] action create (up to date)
-
file[/opt/gitlab/embedded/service/gitlab-rails/.secret] action delete (up to date)
-
file[/var/opt/gitlab/gitlab-rails/etc/secret] action delete (up to date)
-
templatesymlink[Create a database.yml and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/database.yml to /var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date) (up to date)
-
templatesymlink[Create a secrets.yml and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create
- update content in file /var/opt/gitlab/gitlab-rails/etc/secrets.yml from 8c4134 to a49b15
- suppressed sensitive resource
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml to /var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
- template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create
-
templatesymlink[Create a resque.yml and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/resque.yml to /var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date) (up to date)
-
templatesymlink[Create a aws.yml and create a symlink to Rails root] action delete
- template[/var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/aws.yml to /var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date) (up to date)
-
templatesymlink[Create a smtp_settings.rb and create a symlink to Rails root] action delete
- template[/var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb to /var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date) (up to date)
-
templatesymlink[Create a gitlab.yml and create a symlink to Rails root] action create
-
template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create
- update content in file /var/opt/gitlab/gitlab-rails/etc/gitlab.yml from 78284b to af6ebc --- /var/opt/gitlab/gitlab-rails/etc/gitlab.yml 2017-10-21 06:30:32.703349506 -0400 +++ /var/opt/gitlab/gitlab-rails/etc/.chef-gitlab.yml20171022-19515-1vu0ulf 2017-10-22 03:57:03.773887936 -0400 @@ -159,6 +159,7 @@ https: false external_http: null external_https: null
- artifacts_server: true
https://docs.gitlab.com/ce/customization/libravatar.html
For Libravatar see:@@ -350,6 +351,7 @@
gitaly:
- client_path: /opt/gitlab/embedded/bin token: ""
@@ -398,12 +400,6 @@
git: bin_path: /opt/gitlab/embedded/bin/git
-
-
-
- max_size:
-
- timeout:
geo_primary_role:
-
link[Link /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml to /var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
-
-
templatesymlink[Create a rack_attack.rb and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/rack_attack.rb to /var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date) (up to date)
-
templatesymlink[Create a gitlab_workhorse_secret and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date) (up to date)
-
templatesymlink[Create a gitlab_shell_secret and create a symlink to Rails root] action create
- template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date)
- link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date) (up to date)
-
env_dir[/opt/gitlab/etc/gitlab-rails/env] action create
- directory[/opt/gitlab/etc/gitlab-rails/env] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/HOME] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/LD_PRELOAD] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/PATH] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/ICU_DATA] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/PYTHONPATH] action create (up to date)
- file[/opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME] action create (up to date) (up to date)
-
link[/opt/gitlab/embedded/service/gitlab-rails/tmp] action create (up to date)
-
link[/opt/gitlab/embedded/service/gitlab-rails/public/uploads] action create (up to date)
-
link[/opt/gitlab/embedded/service/gitlab-rails/log] action create (up to date)
-
link[/var/log/gitlab/gitlab-rails/sidekiq.log] action create (skipped due to not_if)
-
file[/opt/gitlab/embedded/service/gitlab-rails/db/schema.rb] action create
- change owner from 'root' to 'git'
-
remote_file[/var/opt/gitlab/gitlab-rails/VERSION] action create
- update content in file /var/opt/gitlab/gitlab-rails/VERSION from de756b to 08bc77 --- /var/opt/gitlab/gitlab-rails/VERSION 2017-10-18 12:07:39.785539202 -0400 +++ /var/opt/gitlab/gitlab-rails/.chef-VERSION20171022-19515-swrrrv 2017-10-22 03:57:03.876890383 -0400 @@ -1,2 +1,2 @@ -10.0.4 +10.1.0
-
remote_file[/var/opt/gitlab/gitlab-rails/REVISION] action create
- update content in file /var/opt/gitlab/gitlab-rails/REVISION from 2e8a01 to faa5e7 --- /var/opt/gitlab/gitlab-rails/REVISION 2017-10-18 12:07:39.802539606 -0400 +++ /var/opt/gitlab/gitlab-rails/.chef-REVISION20171022-19515-1sv3k4p 2017-10-22 03:57:03.880890477 -0400 @@ -1,2 +1,2 @@ -8ebd2dd9 +5a695c4c
-
file[/var/opt/gitlab/gitlab-rails/RUBY_VERSION] action create (up to date)
-
execute[chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public] action run
- execute chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public
-
execute[clear the gitlab-rails cache] action nothing (skipped due to action :nothing)
-
file[/var/opt/gitlab/gitlab-rails/config.ru] action delete (up to date) Recipe: gitlab::add_trusted_certs
-
directory[/etc/gitlab/trusted-certs] action create (up to date)
-
directory[/opt/gitlab/embedded/ssl/certs] action create (up to date)
-
file[/opt/gitlab/embedded/ssl/certs/README] action create (up to date)
-
ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run
-
Moving existing certificates found in /opt/gitlab/embedded/ssl/certs
-
Symlinking existing certificates found in /etc/gitlab/trusted-certs
- execute the ruby block Move existing certs and link to /opt/gitlab/embedded/ssl/certs Recipe: gitlab::default
-
service[create a temporary unicorn service] action nothing (skipped due to action :nothing)
-
service[create a temporary sidekiq service] action nothing (skipped due to action :nothing)
-
service[create a temporary mailroom service] action nothing (skipped due to action :nothing) Recipe: runit::systemd
-
directory[/usr/lib/systemd/system] action create (up to date)
-
cookbook_file[/usr/lib/systemd/system/gitlab-runsvdir.service] action create (up to date)
-
file[/etc/systemd/system/default.target.wants/gitlab-runsvdir.service] action delete (up to date)
-
execute[systemctl daemon-reload] action nothing (skipped due to action :nothing)
-
execute[systemctl enable gitlab-runsvdir] action nothing (skipped due to action :nothing)
-
execute[systemctl start gitlab-runsvdir] action nothing (skipped due to action :nothing) Recipe: gitlab::redis
-
group[user and group for redis] action create (up to date)
-
user[user and group for redis] action create (up to date)
-
group[Socket group] action create (up to date)
-
directory[/var/opt/gitlab/redis] action create (up to date)
-
directory[/var/log/gitlab/redis] action create (up to date)
-
template[/var/opt/gitlab/redis/redis.conf] action create (up to date)
-
directory[/opt/gitlab/sv/redis] action create (up to date)
-
directory[/opt/gitlab/sv/redis/log] action create (up to date)
-
directory[/opt/gitlab/sv/redis/log/main] action create (up to date)
-
template[/opt/gitlab/sv/redis/run] action create (up to date)
-
template[/opt/gitlab/sv/redis/log/run] action create (up to date)
-
template[/var/log/gitlab/redis/config] action create (up to date)
-
ruby_block[reload redis svlogd configuration] action nothing (skipped due to action :nothing)
-
file[/opt/gitlab/sv/redis/down] action delete (up to date)
-
link[/opt/gitlab/init/redis] action create (up to date)
-
link[/opt/gitlab/service/redis] action create (up to date)
-
ruby_block[supervise_redis_sleep] action run (skipped due to not_if)
-
directory[/opt/gitlab/sv/redis/supervise] action create (up to date)
-
directory[/opt/gitlab/sv/redis/log/supervise] action create (up to date)
-
file[/opt/gitlab/sv/redis/supervise/ok] action touch (skipped due to only_if)
-
file[/opt/gitlab/sv/redis/log/supervise/ok] action touch (skipped due to only_if)
-
file[/opt/gitlab/sv/redis/supervise/control] action touch (skipped due to only_if)
-
file[/opt/gitlab/sv/redis/log/supervise/control] action touch (skipped due to only_if)
-
service[redis] action nothing (skipped due to action :nothing) Recipe: gitlab::postgresql_user
-
group[Postgresql user and group] action create (up to date)
-
user[Postgresql user and group] action create (up to date) Recipe: gitlab::postgresql
-
directory[/var/opt/gitlab/postgresql] action create (up to date)
-
directory[/var/opt/gitlab/postgresql/data] action create (up to date)
-
directory[/var/log/gitlab/postgresql] action create (up to date)
-
link[/var/opt/gitlab/postgresql/data] action create (skipped due to not_if)
-
file[/var/opt/gitlab/postgresql/.profile] action create (up to date)
-
directory[create /etc/sysctl.d for kernel.shmmax] action create (up to date)
-
file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf kernel.shmmax] action create (up to date)
-
link[/etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf] action create (up to date)
-
file[delete /etc/sysctl.d/90-postgresql.conf kernel.shmmax] action delete (skipped due to only_if)
-
file[delete /etc/sysctl.d/90-unicorn.conf kernel.shmmax] action delete (skipped due to only_if)
-
file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.shmmax] action delete (skipped due to only_if)
-
file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.shmmax] action delete (skipped due to only_if)
-
execute[load sysctl conf kernel.shmmax] action nothing (skipped due to action :nothing)
-
directory[create /etc/sysctl.d for kernel.shmall] action create (up to date)
-
file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf kernel.shmall] action create (up to date)
-
link[/etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf] action create (up to date)
-
file[delete /etc/sysctl.d/90-postgresql.conf kernel.shmall] action delete (skipped due to only_if)
-
file[delete /etc/sysctl.d/90-unicorn.conf kernel.shmall] action delete (skipped due to only_if)
-
file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.shmall] action delete (skipped due to only_if)
-
file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.shmall] action delete (skipped due to only_if)
-
execute[load sysctl conf kernel.shmall] action nothing (skipped due to action :nothing)
-
directory[create /etc/sysctl.d for kernel.sem] action create (up to date)
-
file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.sem.conf kernel.sem] action create (up to date)
-
link[/etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf] action create (up to date)
-
file[delete /etc/sysctl.d/90-postgresql.conf kernel.sem] action delete (skipped due to only_if)
-
file[delete /etc/sysctl.d/90-unicorn.conf kernel.sem] action delete (skipped due to only_if)
-
file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.sem] action delete (skipped due to only_if)
-
file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.sem] action delete (skipped due to only_if)
-
execute[load sysctl conf kernel.sem] action nothing (skipped due to action :nothing)
-
execute[/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8] action run (skipped due to not_if)
-
template[/var/opt/gitlab/postgresql/data/postgresql.conf] action create (up to date)
-
template[/var/opt/gitlab/postgresql/data/runtime.conf] action create
- update content in file /var/opt/gitlab/postgresql/data/runtime.conf from 65bed2 to 18dd18 --- /var/opt/gitlab/postgresql/data/runtime.conf 2017-08-24 14:51:31.000000000 -0400 +++ /var/opt/gitlab/postgresql/data/.chef-runtime.conf20171022-19515-ucb2n7 2017-10-22 03:57:04.083895296 -0400 @@ -101,6 +101,18 @@
idle_in_transaction_session_timeout = 60000
+# IO settings +effective_io_concurrency = 1 +track_io_timing = 'off' + +# Parallel worker settings +max_worker_processes = 8 +max_parallel_workers_per_gather = 0 + +# Deadlock handling and logging +deadlock_timeout = '5s' +log_lock_waits = 1 +
datestyle = 'iso, mdy'
-
execute[reload postgresql] action run
- execute /opt/gitlab/bin/gitlab-ctl hup postgresql
-
execute[start postgresql] action run (skipped due to not_if)
-
template[/var/opt/gitlab/postgresql/data/pg_hba.conf] action create (up to date)
-
template[/var/opt/gitlab/postgresql/data/pg_ident.conf] action create (up to date)
-
directory[/opt/gitlab/sv/postgresql] action create (up to date)
-
directory[/opt/gitlab/sv/postgresql/log] action create (up to date)
-
directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date)
-
template[/opt/gitlab/sv/postgresql/run] action create (up to date)
-
template[/opt/gitlab/sv/postgresql/log/run] action create (up to date)
-
template[/var/log/gitlab/postgresql/config] action create (up to date)
-
ruby_block[reload postgresql svlogd configuration] action nothing (skipped due to action :nothing)
-
file[/opt/gitlab/sv/postgresql/down] action delete (up to date)
-
directory[/opt/gitlab/sv/postgresql/control] action create (up to date)
-
template[/opt/gitlab/sv/postgresql/control/t] action create (up to date)
-
link[/opt/gitlab/init/postgresql] action create (up to date)
-
link[/opt/gitlab/service/postgresql] action create (up to date)
-
ruby_block[supervise_postgresql_sleep] action run (skipped due to not_if)
-
directory[/opt/gitlab/sv/postgresql/supervise] action create (up to date)
-
directory[/opt/gitlab/sv/postgresql/log/supervise] action create (up to date)
-
file[/opt/gitlab/sv/postgresql/supervise/ok] action touch (skipped due to only_if)
-
file[/opt/gitlab/sv/postgresql/log/supervise/ok] action touch (skipped due to only_if)
-
file[/opt/gitlab/sv/postgresql/supervise/control] action touch (skipped due to only_if)
-
file[/opt/gitlab/sv/postgresql/log/supervise/control] action touch (skipped due to only_if)
-
service[postgresql] action nothing (skipped due to action :nothing) Recipe: gitlab::postgresql-bin
-
ruby_block[Link postgresql bin files to the correct version] action run (skipped due to only_if) Recipe: gitlab::postgresql
-
template[/opt/gitlab/etc/gitlab-psql-rc] action create (up to date)
-
postgresql_user[gitlab] action create
- execute[create gitlab postgresql user] action run (skipped due to not_if) (up to date)
-
execute[create gitlabhq_production database] action run (skipped due to not_if)
-
postgresql_user[gitlab_replicator] action create
- execute[create gitlab_replicator postgresql user] action run (skipped due to not_if) (up to date)
-
execute[enable pg_trgm extension] action nothing (skipped due to action :nothing)
-
execute[reload postgresql] action nothing (skipped due to action :nothing)
-
execute[start postgresql] action nothing (skipped due to action :nothing) Recipe: gitlab::database_migrations
-
bash[migrate gitlab-rails database] action run
- Would execute "bash" "/tmp/chef-script20171022-19515-uc00fh" Recipe: gitlab::postgresql
-
execute[enable pg_trgm extension] action run (skipped due to not_if) Recipe: gitlab::database_migrations
-
bash[migrate gitlab-rails database] action run [execute] rake aborted! StandardError: An error has occurred, all later migrations canceled:
PG::DuplicateColumn: ERROR: column "merge_requests_ff_only_enabled" of relation "projects" already exists : ALTER TABLE "projects" ADD "merge_requests_ff_only_enabled" boolean /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in `add_column' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in `block in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:in `add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in `up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' ActiveRecord::StatementInvalid: PG::DuplicateColumn: ERROR: column "merge_requests_ff_only_enabled" of relation "projects" already exists : ALTER TABLE "projects" ADD "merge_requests_ff_only_enabled" boolean /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in `add_column' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in `block in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:in `add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in `up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' PG::DuplicateColumn: ERROR: column "merge_requests_ff_only_enabled" of relation "projects" already exists /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in `add_column' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in `block in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:in `add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in `up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in `block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in `load' /opt/gitlab/embedded/bin/bundle:23:in `<main>' Tasks: TOP => db:migrate (See full trace by running task with --trace) == 20141126120926 AddMergeRequestRebaseEnabledToProjects: migrating =========== -- transaction_open?() -> 0.0000s -- execute("SET statement_timeout TO 0") -> 0.0004s -- transaction() -- add_column(:projects, :merge_requests_rebase_enabled, :boolean, {:default=>nil}) -> 0.0020s -- change_column_default(:projects, :merge_requests_rebase_enabled, false) -> 0.0037s -> 0.0065s -- transaction_open?() -> 0.0000s -- exec_query("SELECT COUNT(*) AS count FROM \"projects\"") -> 0.0021s -- exec_query("SELECT \"projects\".\"id\" FROM \"projects\" ORDER BY \"projects\".\"id\" ASC LIMIT 1") -> 0.0005s -- exec_query("SELECT \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 1 ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1") -> 0.0004s -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 1 AND \"projects\".\"id\" < 2") -> 0.0010s -- exec_query("SELECT \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 2 ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 2 AND \"projects\".\"id\" < 3") -> 0.0008s -- exec_query("SELECT \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 3 ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 3 AND \"projects\".\"id\" < 4") -> 0.0004s -- exec_query("SELECT \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 4 ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 4 AND \"projects\".\"id\" < 5") -> 0.0004s -- exec_query("SELECT \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 5 ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 5 AND \"projects\".\"id\" < 6") -> 0.0004s -- exec_query("SELECT \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 6 ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 6 AND \"projects\".\"id\" < 7") -> 0.0004s -- exec_query("SELECT \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 7 ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 7 AND \"projects\".\"id\" < 8") -> 0.0005s -- exec_query("SELECT \"projects\".\"id\" FROM \"projects\" WHERE \"projects\".\"id\" >= 8 ORDER BY \"projects\".\"id\" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE \"projects\" SET \"merge_requests_rebase_enabled\" = 'f' WHERE \"projects\".\"id\" >= 8") -> 0.0004s -- change_column_null(:projects, :merge_requests_rebase_enabled, false) -> 0.0006s == 20141126120926 AddMergeRequestRebaseEnabledToProjects: migrated (0.0243s) == == 20150827121444 AddFastForwardOptionToProject: migrating ==================== -- transaction_open?() -> 0.0000s -- execute("SET statement_timeout TO 0") -> 0.0002s -- transaction() -- add_column(:projects, :merge_requests_ff_only_enabled, :boolean, {:default=>nil})
run
on resource 'bash[migrate gitlab-rails database]'Expected process to exit with [0], but received '1' ---- Begin output of "bash" "/tmp/chef-script20171022-19515-ef47vt" ---- STDOUT: rake aborted! StandardError: An error has occurred, all later migrations canceled:
PG::DuplicateColumn: ERROR: column "merge_requests_ff_only_enabled" of relation "projects" already exists : ALTER TABLE "projects" ADD "merge_requests_ff_only_enabled" boolean /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in
add_column' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in
block in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:inadd_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in
up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:inblock (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in
load' /opt/gitlab/embedded/bin/bundle:23:in<main>' ActiveRecord::StatementInvalid: PG::DuplicateColumn: ERROR: column "merge_requests_ff_only_enabled" of relation "projects" already exists : ALTER TABLE "projects" ADD "merge_requests_ff_only_enabled" boolean /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:in
add_column' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:inblock in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:in
add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:inup' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:in
block (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:inload' /opt/gitlab/embedded/bin/bundle:23:in
' PG::DuplicateColumn: ERROR: column "merge_requests_ff_only_enabled" of relation "projects" already exists /opt/gitlab/embedded/service/gitlab-rails/config/initializers/postgresql_limit_fix.rb:6:inadd_column' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:325:in
block in add_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/lib/gitlab/database/migration_helpers.rb:321:inadd_column_with_default' /opt/gitlab/embedded/service/gitlab-rails/db/migrate/20150827121444_add_fast_forward_option_to_project.rb:11:in
up' /opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:50:inblock (3 levels) in <top (required)>' /opt/gitlab/embedded/bin/bundle:23:in
load' /opt/gitlab/embedded/bin/bundle:23:in `' Tasks: TOP => db:migrate (See full trace by running task with --trace) == 20141126120926 AddMergeRequestRebaseEnabledToProjects: migrating =========== -- transaction_open?() -> 0.0000s -- execute("SET statement_timeout TO 0") -> 0.0004s -- transaction() -- add_column(:projects, :merge_requests_rebase_enabled, :boolean, {:default=>nil}) -> 0.0020s -- change_column_default(:projects, :merge_requests_rebase_enabled, false) -> 0.0037s -> 0.0065s -- transaction_open?() -> 0.0000s -- exec_query("SELECT COUNT(*) AS count FROM "projects"") -> 0.0021s -- exec_query("SELECT "projects"."id" FROM "projects" ORDER BY "projects"."id" ASC LIMIT 1") -> 0.0005s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 1 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0004s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 1 AND "projects"."id" < 2") -> 0.0010s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 2 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 2 AND "projects"."id" < 3") -> 0.0008s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 3 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 3 AND "projects"."id" < 4") -> 0.0004s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 4 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 4 AND "projects"."id" < 5") -> 0.0004s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 5 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 5 AND "projects"."id" < 6") -> 0.0004s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 6 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 6 AND "projects"."id" < 7") -> 0.0004s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 7 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 7 AND "projects"."id" < 8") -> 0.0005s -- exec_query("SELECT "projects"."id" FROM "projects" WHERE "projects"."id" >= 8 ORDER BY "projects"."id" ASC LIMIT 1 OFFSET 1") -> 0.0003s -- execute("UPDATE "projects" SET "merge_requests_rebase_enabled" = 'f' WHERE "projects"."id" >= 8") -> 0.0004s -- change_column_null(:projects, :merge_requests_rebase_enabled, false) -> 0.0006s == 20141126120926 AddMergeRequestRebaseEnabledToProjects: migrated (0.0243s) ==== 20150827121444 AddFastForwardOptionToProject: migrating ==================== -- transaction_open?() -> 0.0000s -- execute("SET statement_timeout TO 0") -> 0.0002s -- transaction() -- add_column(:projects, :merge_requests_ff_only_enabled, :boolean, {:default=>nil}) STDERR: ---- End output of "bash" "/tmp/chef-script20171022-19515-ef47vt" ---- Ran "bash" "/tmp/chef-script20171022-19515-ef47vt" returned 1
51: bash "migrate gitlab-rails database" do 52: code <<-EOH 53: set -e 54: log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log" 55: umask 077 56: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file} 57: STATUS=${PIPESTATUS[0]} 58: echo $STATUS > #{db_migrate_status_file} 59: exit $STATUS 60: EOH 61: environment env_variables unless env_variables.empty? 62: notifies :run, 'execute[enable pg_trgm extension]', :before if omnibus_helper.service_enabled?('postgresql') 63: notifies :run, "execute[clear the gitlab-rails cache]", :immediately 64: dependent_services.each do |svc| 65: notifies :restart, svc, :immediately 66: end 67: not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)" 68: only_if { node['gitlab']['gitlab-rails']['auto_migrate'] } 69: end
bash("migrate gitlab-rails database") do action [:run] updated true retries 0 retry_delay 2 default_guard_interpreter :default command "migrate gitlab-rails database" backup 5 returns 0 code " set -e\n log_file="/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee
{log_file}\n STATUS=
{PIPESTATUS[0]}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-5a695c4c\n exit $STATUS\n" interpreter "bash" declared_type :bash cookbook_name "gitlab" recipe_name "database_migrations" not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-5a695c4c) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-5a695c4c | grep -Fx 0)" only_if { #code block } endx86_64-linux
Recipe: gitlab::gitlab-rails
- execute[clear the gitlab-rails cache] action run
- execute /opt/gitlab/bin/gitlab-rake cache:clear
Running handlers: Running handlers complete Chef Client failed. 14 resources updated in 35 seconds Checking for an omnibus managed postgresql: OK Checking for a newer version of PostgreSQL to install No new version of PostgreSQL installed, nothing to upgrade to Ensuring PostgreSQL is updated: OK Restarting previously running GitLab services ok: run: gitaly: (pid 20049) 1s ok: run: gitlab-monitor: (pid 20060) 0s ok: run: gitlab-workhorse: (pid 20071) 1s ok: run: logrotate: (pid 20095) 0s ok: run: nginx: (pid 20120) 0s ok: run: node-exporter: (pid 20128) 1s ok: run: postgres-exporter: (pid 20177) 0s ok: run: postgresql: (pid 19494) 47s ok: run: prometheus: (pid 20185) 0s ok: run: redis: (pid 19502) 46s ok: run: redis-exporter: (pid 20188) 1s ok: run: sidekiq: (pid 20202) 0s ok: run: unicorn: (pid 20209) 0s
_______ __ __ __
/ ____(_) /_/ / ____ _/ /_
/ / / / / / / __ `/ __
/ // / / // // // / // /
___//_/__/__,/./
Upgrade complete! If your GitLab server is misbehaving try running sudo gitlab-ctl restart before anything else. If you need to roll back to the previous version you can use the database backup made during the upgrade (scroll up for the filename).
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Debian 9.2 Current User: git Using RVM: no Ruby Version: 2.3.5p376 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.1.0 Redis Version: 3.2.5 Git Version: 2.13.5 Sidekiq Version:5.0.4 Go Version: unknownGitLab information Version: 10.1.0 Revision: 5a695c4 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://develop.briankintz.com HTTP Clone URL: https://develop.briankintz.com/some-group/some-project.git SSH Clone URL: git@develop.briankintz.com:some-group/some-project.git Using LDAP: no Using Omniauth: no
GitLab Shell Version: 5.9.3 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...GitLab Shell version >= 5.9.3 ? ... OK (5.9.3) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 5/1 ... ok 4/2 ... ok 4/3 ... ok 6/4 ... ok 5/5 ... ok 6/6 ... ok 6/7 ... ok 6/8 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 5/1 ... yes 4/2 ... yes 4/3 ... yes 6/4 ... yes 5/5 ... yes 6/6 ... yes 6/7 ... yes 6/8 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.3 ? ... yes (2.3.5) Git version >= 2.7.3 ? ... yes (2.13.5) Git user has default SSH configuration? ... yes Active users: ... 2
Checking GitLab ... Finished