upgrade to 8.12.3 caused gitlab to show 502 bad gateway error

Summary

I upgraded from gitlab-ce-8.12.2-ce.0.el6.x86_64 to gitlab-ce-8.12.3-ce.0.el6.x86_64 via a Yum -y update on a RedHat 6.8. After the update I was seeing the following errors in /var/log/messages

Oct 4 19:31:13 gitlab gitlab-runner[2107]: time="2016-10-04T19:31:13-06:00" level=warning msg="Checking for builds... failed" runner=99cc1404 status="502 Bad Gateway" #12 (closed) Oct 4 19:31:16 gitlab gitlab-runner[2107]: time="2016-10-04T19:31:16-06:00" level=warning msg="Checking for builds... failed" runner=99cc1404 status="502 Bad Gateway" #12 (closed) Oct 4 19:31:19 gitlab gitlab-runner[2107]: time="2016-10-04T19:31:19-06:00" level=warning msg="Checking for builds... failed" runner=99cc1404 status="502 Bad Gateway" #12 (closed) Oct 4 19:31:22 gitlab gitlab-runner[2107]: time="2016-10-04T19:31:22-06:00" level=warning msg="Checking for builds... failed" runner=99cc1404 status="502 Bad Gateway" #12 (closed)

These errors in gitlab-ctl tail ==> /var/log/gitlab/gitlab-workhorse/current <== 2016-10-05_01:12:28.91167 2016/10/04 19:12:28 error: badgateway: POST "/ci/api/v1/builds/register.json" failed after 0.000s: dial unix /var/opt/gitlab/gitlab-rails/sockets/gitlab.socket: connect: connection refused 2016-10-05_01:12:28.91173 gitlab.sandia.gov @ - - [2016-10-04 19:12:28.911332171 -0600 MDT] "POST /ci/api/v1/builds/register.json HTTP/1.1" 502 89 "" "gitlab-ci-multi-runner 1.3.2 (stable-1-3; go1.6.2; linux/amd64)" 0.000373

(Summarize the bug encountered concisely)

Steps to reproduce

Running gitlab-ctl reconfigure produced the following (Note error about /var/opt/gitlab/gitlab-rails/shared)

[root@gitlab gitlab]# gitlab-ctl reconfigure Starting Chef Client, version 12.12.15 resolving cookbooks for run list: ["gitlab"] Synchronizing Cookbooks:

• package (0.0.0)
• runit (0.14.2)
• gitlab (0.0.1) Installing Cookbook Gems: Compiling Cookbooks... Recipe: gitlab::default

• directory[/etc/gitlab] action create (up to date) init (upstart 0.6.5) Converging 282 resources

• directory[/etc/gitlab] action create (up to date)

• directory[Create /var/opt/gitlab] action create (up to date)

• directory[/opt/gitlab/embedded/etc] action create (up to date)

• template[/opt/gitlab/embedded/etc/gitconfig] action create (up to date) Recipe: gitlab::web-server

• group[gitlab-www] action create (up to date)

• user[gitlab-www] action create (up to date) Recipe: gitlab::users

• directory[/var/opt/gitlab] action create (up to date)

• group[git] action create (up to date)

• user[git] action create (up to date)

• template[/var/opt/gitlab/.gitconfig] action create (up to date) Recipe: gitlab::gitlab-shell

• ruby_block[directory resource: /srv/gitlab-data] action run (skipped due to not_if)

• ruby_block[directory resource: /srv/gitlab-data/repositories] action run (skipped due to not_if)

• ruby_block[directory resource: /var/opt/gitlab/.ssh] action run (skipped due to not_if)

• directory[/opt/gitlab/embedded/service/gitlab-shell/hooks/] action create (up to date)

• directory[/var/log/gitlab/gitlab-shell/] action create (up to date)

• directory[/var/opt/gitlab/gitlab-shell] action create (up to date)

• template[/var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)

• link[Link /opt/gitlab/embedded/service/gitlab-shell/config.yml to /var/opt/gitlab/gitlab-shell/config.yml] action create (up to date)

• link[/opt/gitlab/embedded/service/gitlab-shell/.gitlab_shell_secret] action create (up to date)

• execute[/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions] action run

  • execute /opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-keys check-permissions

• bash[Set proper security context on ssh files for selinux] action run (skipped due to only_if) Recipe: gitlab::gitlab-rails

• directory[/var/log/gitlab] action create (up to date)

• ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared] action run

  ================================================================================ Error executing action run on resource 'ruby_block[directory resource: /var/opt/gitlab/gitlab-rails/shared]'

  Mixlib::ShellOut::ShellCommandFailed

  Expected process to exit with [0], but received '1' ---- Begin output of test -d "/var/opt/gitlab/gitlab-rails/shared" -a "$(stat --printf='%U:%G %04a' /var/opt/gitlab/gitlab-rails/shared)" = 'git:gitlab-www 0751' ---- STDOUT: STDERR: ---- End output of test -d "/var/opt/gitlab/gitlab-rails/shared" -a "$(stat --printf='%U:%G %04a' /var/opt/gitlab/gitlab-rails/shared)" = 'git:gitlab-www 0751' ---- Ran test -d "/var/opt/gitlab/gitlab-rails/shared" -a "$(stat --printf='%U:%G %04a' /var/opt/gitlab/gitlab-rails/shared)" = 'git:gitlab-www 0751' returned 1

  Cookbook Trace:

  /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:35:in run_command' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:89:in validate' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/libraries/storage_directory_helper.rb:82:in validate!' /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb:35:in block (3 levels) in from_file'

  Resource Declaration:

  In /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb

  26: ruby_block "directory resource: #{params[:path]}" do 27: block do 28: # Ensure the directory exists 29: storage_helper.ensure_directory_exists(params[:path]) 30: 31: # Ensure the permissions are set 32: storage_helper.ensure_permissions_set(params[:path]) 33: 34: # Error out if we have not achieved the target permissions 35: storage_helper.validate!(params[:path]) 36: end 37: not_if { storage_helper.validate(params[:path]) } 38: end 39: end

  Compiled Resource:

  Declared in /opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb:26:in `block in from_file'

  ruby_block("directory resource: /var/opt/gitlab/gitlab-rails/shared") do params {:path=>"/var/opt/gitlab/gitlab-rails/shared", :owner=>"git", :group=>"gitlab-www", :mode=>"0751", :name=>"/var/opt/gitlab/gitlab-rails/shared"} action [:run] retries 0 retry_delay 2 default_guard_interpreter :default block_name "directory resource: /var/opt/gitlab/gitlab-rails/shared" declared_type :ruby_block cookbook_name "gitlab" recipe_name "gitlab-rails" block #Proc:0x0000000317de70@/opt/gitlab/embedded/cookbooks/cache/cookbooks/gitlab/definitions/storage_directory.rb:27 not_if { #code block } end

  Platform:

  x86_64-linux

Running handlers: Running handlers complete Chef Client failed. 1 resources updated in 13 seconds

(How one can reproduce the issue - this is very important)

Expected behavior

Reconfiguring worked before upgrading the server. (What you should see instead)

Actual behavior

After much research I discovered that the problem went away if I made /var/opt/gitlab/gitlab-rails/shared a directory instead of the link /var/opt/gitlab/gitlab-rails/shared -> /srv/gitlab/var/shared/ Even though it used to work and all of the ownerships looked correct.

(What actually happens)

Relevant logs and/or screenshots

(Paste any relevant logs - please use code blocks (```) to format console output, logs, and code as it's very hard to read otherwise.)

Output of checks

/var/log/messages gitlab-ctl tail

Results of GitLab application Check

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:check SANITIZE=true)

(For installations from source run and paste the output of: sudo -u git -H bundle exec rake gitlab:check RAILS_ENV=production SANITIZE=true)

(we will only investigate if the tests are passing)

Results of GitLab environment info

(For installations with omnibus-gitlab package run and paste the output of: sudo gitlab-rake gitlab:env:info)

[root@gitlab gitlab-rails]# gitlab-rake gitlab:env:info

System information System: RedHatEnterpriseServer 6.8 Current User: git Using RVM: no Ruby Version: 2.3.1p112 Gem Version: 2.6.6 Bundler Version:1.13.1 Rake Version: 10.5.0 Sidekiq Version:4.1.4

GitLab information Version: 8.12.3 Revision: 467e1ca4 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://gitlab.mycompany.edu HTTP Clone URL: https://gitlab.mycompany.edu/some-group/some-project.git SSH Clone URL: git@gitlab.mycompany.edu:some-group/some-project.git Using LDAP: yes Using Omniauth: yes Omniauth Providers: github

GitLab Shell Version: 3.6.1 Repository storage paths:

• default: /srv/gitlab-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/ Git: /opt/gitlab/embedded/bin/git

(For installations from source run and paste the output of: sudo -u git -H bundle exec rake gitlab:env:info RAILS_ENV=production)

Possible fixes

(If you can, link to the line of code that might be responsible for the problem)