(Ubuntu) GitLab starts services in the wrong order after upgrade

Summary

After an upgrade and when using gitlab-ctl (re)start, GitLab starts it's services in the wrong order. At least on my system this results in an infinite 502 until I start the services in the right order manually.

Steps to reproduce

Variant 1

  1. Install an older version of gitlab using https//packages.gitlab.com
  2. Upgrade GitLab on Debian or Ubuntu (apt upgrade)
  3. wild 502 appears

Variant 2

  1. Run one of the following commands as root:
    • gitlab-ctl restart
    • gitlab-ctl stop && gitlab-ctl start
  2. 502

What is the current bug behavior?

When looking at the logs of the upgrade I just did, I get the following order:

ok: run: gitlab-pages: (pid 25665) 1s
ok: run: gitlab-workhorse: (pid 25603) 2s
ok: run: logrotate: (pid 25687) 0s
ok: run: nginx: (pid 25763) 1s
ok: run: postgresql: (pid 16891) 169557s
ok: run: redis: (pid 16928) 169552s
ok: run: registry: (pid 25777) 0s
ok: run: sidekiq: (pid 25790) 0s
ok: run: unicorn: (pid 25797) 1s

What is the expected correct behavior?

I checked some logs while GitLab was in the infinite 502 state and saw errors connecting to Redis and Postgres. (I don't recall which one because this was a few months ago)

That's why I'd actually expect this order:

ok: run: postgresql: (pid 16891) 169557s
ok: run: redis: (pid 16928) 169552s
ok: run: gitlab-pages: (pid 25665) 1s
ok: run: gitlab-workhorse: (pid 25603) 2s
ok: run: logrotate: (pid 25687) 0s
ok: run: nginx: (pid 25763) 1s
ok: run: registry: (pid 25777) 0s
ok: run: sidekiq: (pid 25790) 0s
ok: run: unicorn: (pid 25797) 1s

Possible workaround

I have this command as an alias in my ~/.zshrc and run it after every GitLab update or maintenance:

gitlab-ctl stop && gitlab-ctl start postgresql && gitlab-ctl start redis && sleep 5 && gitlab-ctl start

Details of package version

I have this issue on two installations of Debian and Ubuntu (moved the server) at least since 10.8.0

Provide the package version installation details 
ii  gitlab-ce                  11.3.3-ce.0        amd64

Environment details

  • Operating System: Ubuntu (previously Debian)
  • Installation Target, remove incorrect values:
    • Bare Metal Machine
  • Installation Type, remove incorrect values:
    • Upgrade from version 11.3.1-ce.0 and previous
  • Is there any other software running on the machine: gitlab-runner
  • Single node installation
  • Resources
    • CPU: Intel(R) Core(TM) i7-2600 CPU @ 3.40GHz
    • Memory total: 16G

Configuration details

Provide the relevant sections of `/etc/gitlab/gitlab.rb` 
Nothing relevant to this issue in /etc/gitlab/gitlab.rb.
Edited by Lerk