Gitlab 8.17 NOT WORKING after upgrade from 8.16.6! URGENT !
Summary
After upgrading the gitlab-ce version from 8.16.6 to 8.17 today using yum update gitlab-ce command, the gitlab UI in the browser says 500 Error.
We then stopped the gitlab using gitlab-ctl stop and gave gitlab-ctl reconfigure
The output of gitlab-ctl reconfigure is below
[root@cm ~]# gitlab-ctl reconfigure
Starting Chef Client, version 12.12.15
resolving cookbooks for run list: ["gitlab"]
Synchronizing Cookbooks:
- gitlab (0.0.1)
- package (0.0.0)
- runit (0.14.2)
Installing Cookbook Gems:
Compiling Cookbooks...
Recipe: gitlab::default
* directory[/etc/gitlab] action create (up to date)
/sbin/init: unrecognized option '--version'
-.mount loaded active mounted /
Converging 291 resources
* directory[/etc/gitlab] action create (up to date)
* directory[Create /var/opt/gitlab] action create (up to date)
* directory[/opt/gitlab/embedded/etc] action create (up to date)
* template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date)
Recipe: gitlab::web-server
* group[Webserver user and group] action create (up to date)
* user[Webserver user and group] action create (up to date)
Recipe: gitlab::users
* directory[/var/opt/gitlab] action create (up to date)
* group[GitLab user and group] action create (up to date)
* user[GitLab user and group] action create (up to date)
* template[/var/opt/gitlab/.gitconfig] action create (up to date)
Recipe: gitlab::gitlab-shell
* ruby_block[directory resource: /var/opt/gitlab/git-data] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/git-data/repositories] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/.ssh] action run (skipped due to not_if)
* directory[/opt/gitlab/embedded/service/gitlab-shell/hooks/] action create (up to date)
* directory[/var/log/gitlab/gitlab-shell/] action create (up to date)
* directory[/var/opt/gitlab/gitlab-shell] action create (up to date)
* templatesymlink[Create a config.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)
(up to date)
* link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create (up to date)
* execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] action run
- execute /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions
* bash[Set proper security context on ssh files for selinux] action run (skipped due to only_if)
Recipe: gitlab::gitlab-rails
* directory[/var/log/gitlab] action create (up to date)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/artifacts] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/lfs-objects] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/uploads] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-ci/builds] action run (skipped due to not_if)
* ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared/pages] action run (skipped due to not_if)
* directory[create /var/opt/gitlab/gitlab-rails/etc] action create (up to date)
* directory[create /opt/gitlab/etc/gitlab-rails] action create (up to date)
* directory[create /var/opt/gitlab/gitlab-rails/working] action create (up to date)
* directory[create /var/opt/gitlab/gitlab-rails/tmp] action create (up to date)
* directory[create /var/opt/gitlab/gitlab-rails/upgrade-status] action create (up to date)
* directory[create /var/log/gitlab/gitlab-rails] action create (up to date)
* directory[/var/opt/gitlab/backups] action create (up to date)
* directory[/var/opt/gitlab/gitlab-rails] action create (up to date)
* directory[/var/opt/gitlab/gitlab-ci] action create (up to date)
* template[/opt/gitlab/etc/gitlab-rails/gitlab-rails-rc] action create (up to date)
* file[/opt/gitlab/embedded/service/gitlab-rails/.secret] action delete (up to date)
* file[/var/opt/gitlab/gitlab-rails/etc/secret] action delete (up to date)
* templatesymlink[Create a database.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/database.yml to /var/opt/gitlab/gitlab-rails/etc/database.yml] action create (up to date)
(up to date)
* templatesymlink[Create a secrets.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/secrets.yml to /var/opt/gitlab/gitlab-rails/etc/secrets.yml] action create (up to date)
(up to date)
* templatesymlink[Create a resque.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/resque.yml to /var/opt/gitlab/gitlab-rails/etc/resque.yml] action create (up to date)
(up to date)
* templatesymlink[Create a aws.yml and create a symlink to Rails root] action delete
* template[/var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/aws.yml to /var/opt/gitlab/gitlab-rails/etc/aws.yml] action delete (up to date)
(up to date)
* templatesymlink[Create a smtp_settings.rb and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/smtp_settings.rb to /var/opt/gitlab/gitlab-rails/etc/smtp_settings.rb] action create (up to date)
(up to date)
* templatesymlink[Create a relative_url.rb and create a symlink to Rails root] action delete
* template[/var/opt/gitlab/gitlab-rails/etc/relative_url.rb] action delete (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/relative_url.rb to /var/opt/gitlab/gitlab-rails/etc/relative_url.rb] action delete (up to date)
(up to date)
* templatesymlink[Create a gitlab.yml and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/gitlab.yml to /var/opt/gitlab/gitlab-rails/etc/gitlab.yml] action create (up to date)
(up to date)
* templatesymlink[Create a rack_attack.rb and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/config/initializers/rack_attack.rb to /var/opt/gitlab/gitlab-rails/etc/rack_attack.rb] action create (up to date)
(up to date)
* templatesymlink[Create a gitlab_workhorse_secret and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_workhorse_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_workhorse_secret] action create (up to date)
(up to date)
* templatesymlink[Create a gitlab_shell_secret and create a symlink to Rails root] action create
* template[/var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date)
* link[Link /opt/gitlab/embedded/service/gitlab-rails/.gitlab_shell_secret to /var/opt/gitlab/gitlab-rails/etc/gitlab_shell_secret] action create (up to date)
(up to date)
* directory[/opt/gitlab/etc/gitlab-rails/env] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/HOME] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/RAILS_ENV] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/LD_PRELOAD] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/SIDEKIQ_MEMORY_KILLER_MAX_RSS] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/BUNDLE_GEMFILE] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/PATH] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/ICU_DATA] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/PYTHONPATH] action create (up to date)
* file[/opt/gitlab/etc/gitlab-rails/env/EXECJS_RUNTIME] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/tmp] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/public/uploads] action create (up to date)
* link[/opt/gitlab/embedded/service/gitlab-rails/log] action create (up to date)
* link[/var/log/gitlab/gitlab-rails/sidekiq.log] action create (skipped due to not_if)
* file[/opt/gitlab/embedded/service/gitlab-rails/db/schema.rb] action create (up to date)
* remote_file[/var/opt/gitlab/gitlab-rails/VERSION] action create (up to date)
* remote_file[/var/opt/gitlab/gitlab-rails/REVISION] action create (up to date)
* file[/var/opt/gitlab/gitlab-rails/RUBY_VERSION] action create (up to date)
* execute[chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public] action run
- execute chown -R root:root /opt/gitlab/embedded/service/gitlab-rails/public
* execute[clear the gitlab-rails cache] action nothing (skipped due to action :nothing)
* file[/var/opt/gitlab/gitlab-rails/config.ru] action delete (up to date)
Recipe: gitlab::gitlab-ci-proxying
* template[/var/opt/gitlab/nginx/conf/gitlab-ci-http.conf] action delete (up to date)
Recipe: gitlab::selinux
* execute[semodule -i /opt/gitlab/embedded/selinux/rhel/7/gitlab-7.2.0-ssh-keygen.pp] action runDisabled
(skipped due to not_if)
Recipe: gitlab::add_trusted_certs
* directory[/etc/gitlab/trusted-certs] action create (up to date)
* directory[/opt/gitlab/embedded/ssl/certs] action create (up to date)
* file[/opt/gitlab/embedded/ssl/certs/README] action create (up to date)
* ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] action run (skipped due to only_if)
Recipe: gitlab::default
* service[create a temporary unicorn service] action nothing (skipped due to action :nothing)
* service[create a temporary ci-unicorn service] action nothing (skipped due to action :nothing)
* service[create a temporary sidekiq service] action nothing (skipped due to action :nothing)
* service[create a temporary ci-sidekiq service] action nothing (skipped due to action :nothing)
* service[create a temporary mailroom service] action nothing (skipped due to action :nothing)
Recipe: runit::systemd
* directory[/usr/lib/systemd/system] action create (up to date)
* cookbook_file[/usr/lib/systemd/system/gitlab-runsvdir.service] action create (up to date)
* file[/etc/systemd/system/default.target.wants/gitlab-runsvdir.service] action delete (up to date)
* execute[systemctl daemon-reload] action nothing (skipped due to action :nothing)
* execute[systemctl enable gitlab-runsvdir] action nothing (skipped due to action :nothing)
* execute[systemctl start gitlab-runsvdir] action nothing (skipped due to action :nothing)
Recipe: gitlab::redis
* group[user and group for redis] action create (up to date)
* user[user and group for redis] action create (up to date)
* group[Socket group] action create (up to date)
* directory[/var/opt/gitlab/redis] action create (up to date)
* directory[/var/log/gitlab/redis] action create (up to date)
* template[/var/opt/gitlab/redis/redis.conf] action create (up to date)
* directory[/opt/gitlab/sv/redis] action create (up to date)
* directory[/opt/gitlab/sv/redis/log] action create (up to date)
* directory[/opt/gitlab/sv/redis/log/main] action create (up to date)
* template[/opt/gitlab/sv/redis/run] action create (up to date)
* template[/opt/gitlab/sv/redis/log/run] action create (up to date)
* template[/var/log/gitlab/redis/config] action create (up to date)
* ruby_block[reload redis svlogd configuration] action nothing (skipped due to action :nothing)
* file[/opt/gitlab/sv/redis/down] action delete (up to date)
* link[/opt/gitlab/init/redis] action create (up to date)
* link[/opt/gitlab/service/redis] action create (up to date)
* ruby_block[supervise_redis_sleep] action run (skipped due to not_if)
* service[redis] action nothing (skipped due to action :nothing)
Recipe: gitlab::postgresql
* group[Postgresql user and group] action create (up to date)
* user[Postgresql user and group] action create (up to date)
* directory[/var/opt/gitlab/postgresql] action create (up to date)
* directory[/var/opt/gitlab/postgresql/data] action create (up to date)
* directory[/var/log/gitlab/postgresql] action create (up to date)
* link[/var/opt/gitlab/postgresql/data] action create (skipped due to not_if)
* file[/var/opt/gitlab/postgresql/.profile] action create (up to date)
* directory[create /etc/sysctl.d for kernel.shmmax] action create (up to date)
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmmax.conf kernel.shmmax] action create (up to date)
* link[/etc/sysctl.d/90-omnibus-gitlab-kernel.shmmax.conf] action create (up to date)
* file[delete /etc/sysctl.d/90-postgresql.conf kernel.shmmax] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-unicorn.conf kernel.shmmax] action delete (skipped due to only_if)
* file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.shmmax] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.shmmax] action delete (skipped due to only_if)
* execute[load sysctl conf kernel.shmmax] action nothing (skipped due to action :nothing)
* directory[create /etc/sysctl.d for kernel.shmall] action create (up to date)
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.shmall.conf kernel.shmall] action create (up to date)
* link[/etc/sysctl.d/90-omnibus-gitlab-kernel.shmall.conf] action create (up to date)
* file[delete /etc/sysctl.d/90-postgresql.conf kernel.shmall] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-unicorn.conf kernel.shmall] action delete (skipped due to only_if)
* file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.shmall] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.shmall] action delete (skipped due to only_if)
* execute[load sysctl conf kernel.shmall] action nothing (skipped due to action :nothing)
* directory[create /etc/sysctl.d for kernel.sem] action create (up to date)
* file[create /opt/gitlab/embedded/etc/90-omnibus-gitlab-kernel.sem.conf kernel.sem] action create (up to date)
* link[/etc/sysctl.d/90-omnibus-gitlab-kernel.sem.conf] action create (up to date)
* file[delete /etc/sysctl.d/90-postgresql.conf kernel.sem] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-unicorn.conf kernel.sem] action delete (skipped due to only_if)
* file[delete /opt/gitlab/embedded/etc/90-omnibus-gitlab.conf kernel.sem] action delete (skipped due to only_if)
* file[delete /etc/sysctl.d/90-omnibus-gitlab.conf kernel.sem] action delete (skipped due to only_if)
* execute[load sysctl conf kernel.sem] action nothing (skipped due to action :nothing)
* execute[/opt/gitlab/embedded/bin/initdb -D /var/opt/gitlab/postgresql/data -E UTF8] action run (skipped due to not_if)
* template[/var/opt/gitlab/postgresql/data/postgresql.conf] action create (up to date)
* template[/var/opt/gitlab/postgresql/data/pg_hba.conf] action create (up to date)
* template[/var/opt/gitlab/postgresql/data/pg_ident.conf] action create (up to date)
* directory[/opt/gitlab/sv/postgresql] action create (up to date)
* directory[/opt/gitlab/sv/postgresql/log] action create (up to date)
* directory[/opt/gitlab/sv/postgresql/log/main] action create (up to date)
* template[/opt/gitlab/sv/postgresql/run] action create (up to date)
* template[/opt/gitlab/sv/postgresql/log/run] action create (up to date)
* template[/var/log/gitlab/postgresql/config] action create (up to date)
* ruby_block[reload postgresql svlogd configuration] action nothing (skipped due to action :nothing)
* file[/opt/gitlab/sv/postgresql/down] action delete (up to date)
* directory[/opt/gitlab/sv/postgresql/control] action create (up to date)
* template[/opt/gitlab/sv/postgresql/control/t] action create (up to date)
* link[/opt/gitlab/init/postgresql] action create (up to date)
* link[/opt/gitlab/service/postgresql] action create (up to date)
* ruby_block[supervise_postgresql_sleep] action run (skipped due to not_if)
* service[postgresql] action nothing (skipped due to action :nothing)
Recipe: gitlab::postgresql-bin
* ruby_block[Link postgresql bin files to the correct version] action run (skipped due to only_if)
Recipe: gitlab::postgresql
* template[/opt/gitlab/etc/gitlab-psql-rc] action create (up to date)
* execute[create gitlab database user] action run
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
[execute] psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
================================================================================
Error executing action `run` on resource 'execute[create gitlab database user]'
================================================================================
Mixlib::ShellOut::ShellCommandFailed
------------------------------------
Expected process to exit with [0], but received '2'
---- Begin output of /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER gitlab" ----
STDOUT:
STDERR: psql: could not connect to server: No such file or directory
Is the server running locally and accepting
connections on Unix domain socket "/var/opt/gitlab/postgresql/.s.PGSQL.5432"?
---- End output of /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER gitlab" ----
Ran /opt/gitlab/bin/gitlab-psql -d template1 -c "CREATE USER gitlab" returned 2
Resource Declaration:
---------------------
# In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/postgresql.rb
161: execute "create #{gitlab_sql_user} database user" do
162: command "/opt/gitlab/bin/gitlab-psql -d template1 -c \"CREATE USER #{gitlab_sql_user}\""
163: user postgresql_user
164: # Added retries to give the service time to start on slower systems
165: retries 20
166: not_if { !pg_helper.is_running? || pg_helper.user_exists?(gitlab_sql_user) }
167: end
168:
Compiled Resource:
------------------
# Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/recipes/postgresql.rb:161:in `from_file'
execute("create gitlab database user") do
action [:run]
retries 20
retry_delay 2
default_guard_interpreter :execute
command "/opt/gitlab/bin/gitlab-psql -d template1 -c \"CREATE USER gitlab\""
backup 5
returns 0
user "gitlab-psql"
declared_type :execute
cookbook_name "gitlab"
recipe_name "postgresql"
not_if { #code block }
end
Platform:
---------
x86_64-linux
Running handlers:
Running handlers complete
Chef Client failed. 2 resources updated in 44 seconds