Omnibus did not start up PostgreSQL in time to do migration
Here's what I did yesterday:
- I installed 9.0.5 CE on a Ubuntu 16.04 machine. All services were
up. - Then I attempted to install the latest CE nightly build. It shut down all services but did not start up PostgreSQL:
Running handlers complete
Chef Client finished, 0/1 resources updated in 02 seconds
Checking PostgreSQL executables: OK
Shutting down all GitLab services except those needed for migrations
ok: down: gitaly: 1s, normally up
ok: down: gitlab-monitor: 0s, normally up
ok: down: gitlab-workhorse: 0s, normally up
ok: down: logrotate: 1s, normally up
ok: down: nginx: 0s, normally up
ok: down: node-exporter: 1s, normally up
ok: down: postgres-exporter: 0s, normally up
ok: down: postgresql: 1s, normally up
ok: down: prometheus: 0s, normally up
ok: down: redis: 0s, normally up
ok: down: redis-exporter: 1s, normally up
ok: down: sidekiq: 0s, normally up
ok: down: unicorn: 0s, normally up
Reconfiguring GitLab to apply migrations
Starting Chef Client, version 12.12.15
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
- runit (0.14.2)
- package (0.0.0)
- gitlab (0.0.1)
Installing Cookbook Gems:
Compiling Cookbooks...
Recipe: gitlab::default
* directory[/etc/gitlab] action create (up to date)
/sbin/init: unrecognized option '--version'
-.mount loaded active mounted /
Converging 367 resources
* directory[/etc/gitlab] action create (up to date)
* directory[Create /var/opt/gitlab] action create (up to date)
* directory[/opt/gitlab/embedded/etc] action create (up to date)
* template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
Recipe: gitlab::web-server
* group[Webserver user and group] action create (up to date)
* user[Webserver user and group] action create (up to date)
Recipe: gitlab::users
* directory[/var/opt/gitlab] action create (up to date)
* group[GitLab user and group] action create (up to date)
* user[GitLab user and group] action create (up to date)
* template[/var/opt/gitlab/.gitconfig] action create (up to date)
Recipe: gitlab::gitlab-shell
* ruby_block[directory resource: /var/opt/gitlab/git-data] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/.ssh] action run (skipped due to not_if)
* directory[/var/log/gitlab/gitlab-shell/] action create (up to date)
* directory[/var/opt/gitlab/gitlab-shell] action create (up to date)
* templatesymlink[Create a config.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
(up to date)
* link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create (up to date)
* execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] action run
- execute /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions
* bash[Set proper security context on ssh files for selinux] action run (skipped due to only_if)
Recipe: gitlab::gitlab-rails
* directory[/var/log/gitlab] action create (up to date)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/artifacts] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/lfs-objects] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/uploads] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-ci/builds] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/pages] action run (skipped due to not_if)
* directory[create /var/opt/gitlab/gitlab-rails/etc] action create (up to date)
* directory[create /opt/gitlab/etc/gitlab-rails] action create (up to date)
* directory[create /var/opt/gitlab/gitlab-rails/working] action create (up to date)
* directory[create /var/opt/gitlab/gitlab-rails/tmp] action create (up to date)
* directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] action create (up to date)
* directory[create /var/log/gitlab/gitlab-rails] action create (up to date)
* directory[/var/opt/gitlab/backups] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails] action create (up to date)
* directory[/var/opt/gitlab/gitlab-ci] action create (up to date)
* template[/opt/gitlab/etc/gitlab-rails/gitlab-rails-rc] action create (up to date)
* file[/opt/gitlab/embedded/service/gitlab-rails/.secret] action delete (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/secret] action delete (up to date)
* templatesymlink[Create a database.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/database.yml to /var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
(up to date)
* templatesymlink[Create a secrets.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml to /var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
(up to date)
* templatesymlink[Create a resque.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/resque.yml to /var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
(up to date)
* templatesymlink[Create a aws.yml and create a symlink to Rails root] action delete
* template[/var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/aws.yml to /var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date)
(up to date)
* templatesymlink[Create a smtp_settings.rb and create a symlink to Rails root] action delete
* template[/var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb to /var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action delete (up to date)
(up to date)
* templatesymlink[Create a relative_url.rb and create a symlink to Rails root] action delete
* template[/var/opt/gitlab/gitlab-rails/etc/relative_url.rb] action delete (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/relative_url.rb to /var/opt/gitlab/gitlab-rails/etc/relative_url.rb] action delete (up to date)
(up to date)
* templatesymlink[Create a gitlab.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml to /var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
(up to date)
* templatesymlink[Create a rack_attack.rb and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/rack_attack.rb to /var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date)
(up to date)
* templatesymlink[Create a gitlab_workhorse_secret and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date)
(up to date)
* templatesymlink[Create a gitlab_shell_secret and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date)
(up to date)
* directory[/opt/gitlab/etc/gitlab-rails/env] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/HOME] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/LD_PRELOAD] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/PATH] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/ICU_DATA] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/PYTHONPATH] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/tmp] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/public/uploads] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/log] action create (up to date)
* link[/var/log/gitlab/gitlab-rails/sidekiq.log] action create (skipped due to not_if)
* file[/opt/gitlab/embedded/service/gitlab-rails/db/schema.rb] action create
- change owner from 'root' to 'git'
* remote_file[/var/opt/gitlab/gitlab-rails/VERSION] action create
- update content in file /var/opt/gitlab/gitlab-rails/VERSION from e5a9f0 to 3efcd6
--- /var/opt/gitlab/gitlab-rails/VERSION 2017-04-14 07:26:20.484576974 +0000
+++ /var/opt/gitlab/gitlab-rails/.chef-VERSION20170414-21191-1kx9o8o 2017-04-14 07:34:30.325391910 +0000
@@ -1,2 +1,2 @@
-9.1.0-rc1
+9.1.0-pre
* remote_file[/var/opt/gitlab/gitlab-rails/REVISION] action create
- update content in file /var/opt/gitlab/gitlab-rails/REVISION from 3dae8c to aa2116
--- /var/opt/gitlab/gitlab-rails/REVISION 2017-04-14 07:26:20.488576974 +0000
+++ /var/opt/gitlab/gitlab-rails/.chef-REVISION20170414-21191-mac70h 2017-04-14 07:34:30.329391915 +0000
@@ -1,2 +1,2 @@
-467576d
+75af8d9
* file[/var/opt/gitlab/gitlab-rails/RUBY_VERSION] action create (up to date)
* execute[chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public] action run
- execute chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public
* execute[clear the gitlab-rails cache] action nothing (skipped due to action :nothing)
* file[/var/opt/gitlab/gitlab-rails/config.ru] action delete (up to date)
Recipe: gitlab::add_trusted_certs
* directory[/etc/gitlab/trusted-certs] action create (up to date)
* directory[/opt/gitlab/embedded/ssl/certs] action create (up to date)
* file[/opt/gitlab/embedded/ssl/certs/README] action create (up to date)
* ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run
* Moving existing certificates found in /opt/gitlab/embedded/ssl/certs
* Symlinking existing certificates found in /etc/gitlab/trusted-certs
- execute the ruby block Move existing certs and link to /opt/gitlab/embedded/ssl/certs
Recipe: gitlab::default
* service[create a temporary unicorn service] action nothing (skipped due to action :nothing)
* service[create a temporary sidekiq service] action nothing (skipped due to action :nothing)
* service[create a temporary mailroom service] action nothing (skipped due to action :nothing)
Recipe: runit::systemd
* directory[/usr/lib/systemd/system] action create (up to date)
* cookbook_file[/usr/lib/systemd/system/gitlab-runsvdir.service] action create (up to date)
* file[/etc/systemd/system/default.target.wants/gitlab-runsvdir.service] action delete (up to date)
* execute[systemctl daemon-reload] action nothing (skipped due to action :nothing)
* execute[systemctl enable gitlab-runsvdir] action nothing (skipped due to action :nothing)
* execute[systemctl start gitlab-runsvdir] action nothing (skipped due to action :nothing)
Recipe: gitlab::redis
* group[user and group for redis] action create (up to date)
* user[user and group for redis] action create (up to date)
* group[Socket group] action create (up to date)
* directory[/var/opt/gitlab/redis] action create (up to date)
* directory[/var/log/gitlab/redis] action create (up to date)
* template[/var/opt/gitlab/redis/redis.conf] action create (up to date)
* directory[/opt/gitlab/sv/redis] action create (up to date)
* directory[/opt/gitlab/sv/redis/log] action create (up to date)
* directory[/opt/gitlab/sv/redis/log/main] action create (up to date)
* template[/opt/gitlab/sv/redis/run] action create (up to date)
* template[/opt/gitlab/sv/redis/log/run] action create (up to date)
* template[/var/log/gitlab/redis/config] action create (up to date)
* ruby_block[reload redis svlogd configuration] action nothing (skipped due to action :nothing)
* file[/opt/gitlab/sv/redis/down] action delete (up to date)
* link[/opt/gitlab/init/redis] action create (up to date)
* link[/opt/gitlab/service/redis] action create (up to date)
* ruby_block[supervise_redis_sleep] action run (skipped due to not_if)
* service[redis] action nothing (skipped due to action :nothing)
Recipe: gitlab::postgresql
* group[Postgresql user and group] action create (up to date)
* user[Postgresql user and group] action create (up to date)
* directory[/var/opt/gitlab/postgresql] action create (up to date)
* directory[/var/opt/gitlab/postgresql/data] action create (up to date)
* directory[/var/log/gitlab/postgresql] action create (up to date)
* link[/var/opt/gitlab/postgresql/data] action create (skipped due to not_if)
* file[/var/opt/gitlab/postgresql/.profile] action create (up to date)
* directory[create /etc/sysctl.d for kernel.shmmax] action create (up to date)
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf kernel.shmmax] action create (up to date)
* link[/etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf] action create (up to date)
* file[delete /etc/sysctl.d/90-postgresql.conf kernel.shmmax] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-unicorn.conf kernel.shmmax] action delete (skipped due to only_if)
* file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.shmmax] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.shmmax] action delete (skipped due to only_if)
* execute[load sysctl conf kernel.shmmax] action nothing (skipped due to action :nothing)
* directory[create /etc/sysctl.d for kernel.shmall] action create (up to date)
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf kernel.shmall] action create (up to date)
* link[/etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf] action create (up to date)
* file[delete /etc/sysctl.d/90-postgresql.conf kernel.shmall] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-unicorn.conf kernel.shmall] action delete (skipped due to only_if)
* file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.shmall] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.shmall] action delete (skipped due to only_if)
* execute[load sysctl conf kernel.shmall] action nothing (skipped due to action :nothing)
* directory[create /etc/sysctl.d for kernel.sem] action create (up to date)
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.sem.conf kernel.sem] action create (up to date)
* link[/etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf] action create (up to date)
* file[delete /etc/sysctl.d/90-postgresql.conf kernel.sem] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-unicorn.conf kernel.sem] action delete (skipped due to only_if)
* file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.sem] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.sem] action delete (skipped due to only_if)
* execute[load sysctl conf kernel.sem] action nothing (skipped due to action :nothing)
* execute[/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8] action run (skipped due to not_if)
* template[/var/opt/gitlab/postgresql/data/postgresql.conf] action create (up to date)
* template[/var/opt/gitlab/postgresql/data/pg_hba.conf] action create (up to date)
* template[/var/opt/gitlab/postgresql/data/pg_ident.conf] action create (up to date)
* directory[/opt/gitlab/sv/postgresql] action create (up to date)
* directory[/opt/gitlab/sv/postgresql/log] action create (up to date)
* directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date)
* template[/opt/gitlab/sv/postgresql/run] action create (up to date)
* template[/opt/gitlab/sv/postgresql/log/run] action create (up to date)
* template[/var/log/gitlab/postgresql/config] action create (up to date)
* ruby_block[reload postgresql svlogd configuration] action nothing (skipped due to action :nothing)
* file[/opt/gitlab/sv/postgresql/down] action delete (up to date)
* directory[/opt/gitlab/sv/postgresql/control] action create (up to date)
* template[/opt/gitlab/sv/postgresql/control/t] action create (up to date)
* link[/opt/gitlab/init/postgresql] action create (up to date)
* link[/opt/gitlab/service/postgresql] action create (up to date)
* ruby_block[supervise_postgresql_sleep] action run (skipped due to not_if)
* service[postgresql] action nothing (skipped due to action :nothing)
Recipe: gitlab::postgresql-bin
* ruby_block[Link postgresql bin files to the correct version] action run (skipped due to only_if)
Recipe: gitlab::postgresql
* template[/opt/gitlab/etc/gitlab-psql-rc] action create (up to date)
* execute[create gitlab database user] action run (skipped due to not_if)
* execute[create gitlabhq_production database] action run (skipped due to not_if)
* execute[create gitlab_replicator replication user] action run (skipped due to not_if)
* execute[enable pg_trgm extension] action nothing (skipped due to action :nothing)
Recipe: gitlab::database_migrations
* bash[migrate gitlab-rails database] action run
- Would execute "bash" "/tmp/chef-script20170414-21191-dqa6qt"
Recipe: gitlab::postgresql
* execute[enable pg_trgm extension] action run (skipped due to not_if)
Recipe: gitlab::database_migrations
* bash[migrate gitlab-rails database] action run
[execute] rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
================================================================================
Error executing action `run` on resource 'bash[migrate gitlab-rails database]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '1'
---- Begin output of "bash" "/tmp/chef-script20170414-21191-nx6hfn" ----
STDOUT: rake aborted!
PG::ConnectionBad: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
/opt/gitlab/embedded/service/gitlab-rails/lib/tasks/gitlab/db.rake:49:in `block (3 levels) in <top (required)>'
/opt/gitlab/embedded/bin/bundle:22:in `load'
/opt/gitlab/embedded/bin/bundle:22:in `<main>'
Tasks: TOP => gitlab:db:configure
(See full trace by running task with --trace)
STDERR:
---- End output of "bash" "/tmp/chef-script20170414-21191-nx6hfn" ----
Ran "bash" "/tmp/chef-script20170414-21191-nx6hfn" returned 1
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb
51: bash "migrate gitlab-rails database" do
52: code <<-EOH
53: set -e
54: log_file="#{node['gitlab']['gitlab-rails']['log_directory']}/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log"
55: umask 077
56: /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}
57: STATUS=${PIPESTATUS[0]}
58: echo $STATUS > #{db_migrate_status_file}
59: exit $STATUS
60: EOH
61: environment env_variables unless env_variables.empty?
62: notifies :run, 'execute[enable pg_trgm extension]', :before unless omnibus_helper.not_listening?("postgresql") || !node['gitlab']['postgresql']['enable']
63: notifies :run, "execute[clear the gitlab-rails cache]", :immediately unless omnibus_helper.not_listening?("redis") || !node['gitlab']['gitlab-rails']['rake_cache_clear']
64: dependent_services.each do |svc|
65: notifies :restart, svc, :immediately
66: end
67: not_if "(test -f #{db_migrate_status_file}) && (cat #{db_migrate_status_file} | grep -Fx 0)"
68: only_if { node['gitlab']['gitlab-rails']['auto_migrate'] }
69: end
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/database_migrations.rb:51:in `from_file'
bash("migrate gitlab-rails database") do
action [:run]
updated true
retries 0
retry_delay 2
default_guard_interpreter :default
command "migrate gitlab-rails database"
backup 5
returns 0
code " set -e\n log_file=\"/var/log/gitlab/gitlab-rails/gitlab-rails-db-migrate-$(date +%Y-%m-%d-%H-%M-%S).log\"\n umask 077\n /opt/gitlab/bin/gitlab-rake gitlab:db:configure 2>& 1 | tee ${log_file}\n STATUS=${PIPESTATUS[0]}\n echo $STATUS > /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-75af8d9\n exit $STATUS\n"
interpreter "bash"
declared_type :bash
cookbook_name "gitlab"
recipe_name "database_migrations"
not_if "(test -f /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-75af8d9) && (cat /var/opt/gitlab/gitlab-rails/upgrade-status/db-migrate-873248b1f0d3a7a5535771a3a1635803-75af8d9 | grep -Fx 0)"
only_if { #code block }
end
Platform:
---------
x86_64-linux
Running handlers:
Running handlers complete
Chef Client failed. 7 resources updated in 22 seconds
dpkg: error processing package gitlab-ce (--configure):
subprocess installed post-installation script returned error exit status 1
stanhu@stanhu-test:~$ sudo gitlab-ctl status
down: gitaly: 72s, normally up; run: log: (pid 11198) 505s
down: gitlab-monitor: 71s, normally up; run: log: (pid 11327) 479s
down: gitlab-workhorse: 71s, normally up; run: log: (pid 11210) 499s
down: logrotate: 71s, normally up; run: log: (pid 11232) 496s
down: nginx: 70s, normally up; run: log: (pid 11216) 498s
down: node-exporter: 70s, normally up; run: log: (pid 11284) 488s
down: postgres-exporter: 69s, normally up; run: log: (pid 11308) 481s
down: postgresql: 69s, normally up; run: log: (pid 11021) 535s
down: prometheus: 68s, normally up; run: log: (pid 11270) 490s
down: redis: 68s, normally up; run: log: (pid 10952) 541s
down: redis-exporter: 68s, normally up; run: log: (pid 11297) 482s
down: sidekiq: 66s, normally up; run: log: (pid 11188) 507s
down: unicorn: 66s, normally up; run: log: (pid 11157) 508sPostgreSQL logs:
2017-04-14_07:26:31.96105 LOG: database system was shut down at 2017-04-14 07:26:27 GMT
2017-04-14_07:26:31.96453 LOG: MultiXact member wraparound protections are now enabled
2017-04-14_07:26:31.96733 LOG: database system is ready to accept connections
2017-04-14_07:26:31.96776 LOG: autovacuum launcher started
2017-04-14_07:34:17.68409 received TERM from runit, sending INT instead to force quit connections
2017-04-14_07:34:17.68940 LOG: received fast shutdown request
2017-04-14_07:34:17.68942 LOG: aborting any active transactions
2017-04-14_07:34:17.68942 FATAL: terminating connection due to administrator command
2017-04-14_07:34:17.68942 FATAL: terminating connection due to administrator command
2017-04-14_07:34:17.68943 FATAL: terminating connection due to administrator command
2017-04-14_07:34:17.68943 FATAL: terminating connection due to administrator command
2017-04-14_07:34:17.68943 FATAL: terminating connection due to administrator command
2017-04-14_07:34:17.68944 LOG: autovacuum launcher shutting down
2017-04-14_07:34:17.69037 LOG: shutting down
2017-04-14_07:34:17.98183 LOG: database system is shut down
2017-04-14_07:35:39.81496 LOG: database system was shut down at 2017-04-14 07:34:17 GMT
2017-04-14_07:35:39.82010 LOG: MultiXact member wraparound protections are now enabled
2017-04-14_07:35:39.82081 LOG: database system is ready to accept connections
2017-04-14_07:35:39.82153 LOG: autovacuum launcher started
2017-04-14_08:46:47.87234 LOG: could not fork autovacuum worker process: Cannot allocate memoryIt's odd that it said Shutting down all GitLab services except those needed for migrations but somehow postgresql never started in time.