Prometheus won't start after gitlab upgrade

Summary

After upgrading gitlab 10.7.2 -> 10.8.7 -> 11.4.0, prometheus will no longer start, and access to server results in HTTP 500.

Steps to reproduce

Starting with a gitlab 10.7.2 omnibus install, update to 10.8.7 (which appears to be successful), and then to 11.4.0.

Example Project

What is the current bug behavior?

Prometheus does not start

What is the expected correct behavior?

Prometheus starts

Relevant logs and/or screenshots

The prometheus log contains:

2018-10-23_05:59:38.93255 chpst: fatal: unable to run: /opt/gitlab/embedded/bin/prometheus: file does not exist

repeated many times. There is indeed no /opt/gitlab/embedded/bin/prometheus; rather, there are prometheus1, prometheus2, and prometheus-storage-migrators

Output of checks

This does not happen on gitlab.com.

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		Ubuntu 16.04
Current User:	git
Using RVM:	no
Ruby Version:	2.4.4p296
Gem Version:	2.7.6
Bundler Version:.
Rake Version:	12.3.1
Redis Version:	3.2.12
Git Version:	2.18.1
Sidekiq Version:5.2.1
Go Version:	unknown
GitLab information
Version:	11.4.0
Revision:	6ebbd70
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	postgresql
URL:		[REDACTED]
HTTP Clone URL:	[REDACTED]/some-group/some-project.git
SSH Clone URL:	[REDACTED]:some-group/some-project.git
Using LDAP:	no
Using Omniauth:	no
GitLab Shell
Version:	8.3.3
Repository storage paths:

• default: /mnt/bes/git/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab Shell ...
GitLab Shell version >= 8.3.3 ? ... OK (8.3.3)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
4/1 ... ok
5/2 ... ok
5/3 ... ok
6/4 ... ok
6/5 ... ok
7/6 ... ok
8/7 ... ok
9/8 ... ok
10/9 ... ok
6/10 ... ok
4/12 ... ok
4/13 ... ok
15/14 ... ok
16/15 ... ok
16/16 ... ok
16/17 ... ok
6/18 ... ok
18/19 ... ok
4/20 ... ok
22/21 ... ok
24/22 ... ok
24/23 ... ok
24/24 ... ok
18/25 ... ok
24/26 ... ok
26/28 ... ok
28/29 ... ok
26/31 ... ok
30/32 ... ok
30/33 ... ok
30/34 ... ok
30/35 ... ok
32/37 ... ok
7/39 ... ok
7/40 ... ok
19/41 ... ok
34/42 ... ok
34/43 ... ok
34/44 ... ok
34/45 ... ok
34/46 ... ok
7/47 ... ok
36/48 ... ok
37/50 ... ok
37/51 ... ok
3/52 ... ok
3/53 ... ok
37/54 ... ok
3/55 ... ok
7/56 ... ok
40/61 ... ok
41/62 ... ok
7/63 ... ok
40/64 ... ok
3/66 ... ok
39/67 ... ok
39/68 ... ok
7/69 ... ok
7/70 ... ok
3/71 ... ok
7/72 ... ok
7/73 ... ok
7/74 ... ok
40/75 ... ok
7/76 ... ok
30/77 ... ok
45/78 ... ok
7/79 ... ok
7/80 ... ok
7/82 ... ok
49/83 ... ok
7/84 ... ok
3/85 ... ok
7/87 ... ok
51/88 ... ok
19/89 ... ok
40/90 ... ok
19/91 ... ok
19/92 ... ok
39/96 ... ok
40/98 ... ok
7/100 ... ok
7/101 ... ok
7/103 ... ok
52/104 ... ok
52/105 ... ok
52/106 ... ok
19/107 ... ok
19/108 ... ok
52/109 ... ok
39/110 ... ok
53/112 ... ok
19/113 ... ok
19/116 ... ok
19/117 ... ok
41/118 ... ok
7/119 ... ok
7/120 ... ok
7/121 ... ok
55/122 ... ok
30/123 ... ok
19/124 ... ok
7/125 ... ok
7/126 ... ok
56/127 ... ok
7/128 ... ok
57/129 ... ok
19/131 ... ok
19/132 ... ok
19/133 ... ok
37/134 ... ok
45/135 ... ok
45/138 ... ok
60/139 ... ok
60/140 ... ok
53/141 ... ok
19/142 ... ok
61/143 ... ok
60/144 ... ok
62/145 ... ok
61/147 ... ok
67/148 ... ok
7/149 ... ok
60/150 ... ok
3/151 ... ok
68/152 ... ok
62/153 ... ok
60/154 ... ok
7/157 ... ok
7/158 ... ok
7/159 ... ok
71/160 ... ok
71/161 ... ok
45/162 ... ok
73/163 ... ok
67/164 ... ok
71/165 ... ok
7/166 ... ok
41/167 ... ok
76/168 ... ok
7/169 ... repository is empty
76/170 ... ok
76/171 ... ok
71/172 ... ok
71/173 ... ok
71/174 ... ok
53/175 ... ok
30/176 ... ok
79/178 ... ok
79/179 ... ok
81/180 ... ok
19/181 ... ok
26/182 ... ok
82/184 ... ok
3/185 ... ok
3/186 ... ok
19/187 ... ok
85/188 ... ok
87/189 ... ok
87/190 ... ok
87/191 ... ok
87/192 ... ok
88/193 ... ok
71/194 ... ok
19/195 ... ok
47/196 ... ok
19/197 ... ok
47/199 ... ok
47/200 ... ok
91/201 ... repository is empty
86/202 ... ok
71/203 ... repository is empty
7/204 ... ok
56/205 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Reply by email is disabled in config/gitlab.yml
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
4/1 ... yes
5/2 ... yes
5/3 ... yes
6/4 ... yes
6/5 ... yes
7/6 ... yes
8/7 ... yes
9/8 ... yes
10/9 ... yes
6/10 ... yes
4/12 ... yes
4/13 ... yes
15/14 ... yes
16/15 ... yes
16/16 ... yes
16/17 ... yes
6/18 ... yes
18/19 ... yes
4/20 ... yes
22/21 ... yes
24/22 ... yes
24/23 ... yes
24/24 ... yes
18/25 ... yes
24/26 ... yes
26/28 ... yes
28/29 ... yes
26/31 ... yes
30/32 ... yes
30/33 ... yes
30/34 ... yes
30/35 ... yes
32/37 ... yes
7/39 ... yes
7/40 ... yes
19/41 ... yes
34/42 ... yes
34/43 ... yes
34/44 ... yes
34/45 ... yes
34/46 ... yes
7/47 ... yes
36/48 ... yes
37/50 ... yes
37/51 ... yes
3/52 ... yes
3/53 ... yes
37/54 ... yes
3/55 ... yes
7/56 ... yes
40/61 ... yes
41/62 ... yes
7/63 ... yes
40/64 ... yes
3/66 ... yes
39/67 ... yes
39/68 ... yes
7/69 ... yes
7/70 ... yes
3/71 ... yes
7/72 ... yes
7/73 ... yes
7/74 ... yes
40/75 ... yes
7/76 ... yes
30/77 ... yes
45/78 ... yes
7/79 ... yes
7/80 ... yes
7/82 ... yes
49/83 ... yes
7/84 ... yes
3/85 ... yes
7/87 ... yes
51/88 ... yes
19/89 ... yes
40/90 ... yes
19/91 ... yes
19/92 ... yes
39/96 ... yes
40/98 ... yes
7/100 ... yes
7/101 ... yes
7/103 ... yes
52/104 ... yes
52/105 ... yes
52/106 ... yes
19/107 ... yes
19/108 ... yes
52/109 ... yes
39/110 ... yes
53/112 ... yes
19/113 ... yes
19/116 ... yes
19/117 ... yes
41/118 ... yes
7/119 ... yes
7/120 ... yes
7/121 ... yes
55/122 ... yes
30/123 ... yes
19/124 ... yes
7/125 ... yes
7/126 ... yes
56/127 ... yes
7/128 ... yes
57/129 ... yes
19/131 ... yes
19/132 ... yes
19/133 ... yes
37/134 ... yes
45/135 ... yes
45/138 ... yes
60/139 ... yes
60/140 ... yes
53/141 ... yes
19/142 ... yes
61/143 ... yes
60/144 ... yes
62/145 ... yes
61/147 ... yes
67/148 ... yes
7/149 ... yes
60/150 ... yes
3/151 ... yes
68/152 ... yes
62/153 ... yes
60/154 ... yes
7/157 ... yes
7/158 ... yes
7/159 ... yes
71/160 ... yes
71/161 ... yes
45/162 ... yes
73/163 ... yes
67/164 ... yes
71/165 ... yes
7/166 ... yes
41/167 ... yes
76/168 ... yes
7/169 ... yes
76/170 ... yes
76/171 ... yes
71/172 ... yes
71/173 ... yes
71/174 ... yes
53/175 ... yes
30/176 ... yes
79/178 ... yes
79/179 ... yes
81/180 ... yes
19/181 ... yes
26/182 ... yes
82/184 ... yes
3/185 ... yes
3/186 ... yes
19/187 ... yes
85/188 ... yes
87/189 ... yes
87/190 ... yes
87/191 ... yes
87/192 ... yes
88/193 ... yes
71/194 ... yes
19/195 ... yes
47/196 ... yes
19/197 ... yes
47/199 ... yes
47/200 ... yes
91/201 ... yes
86/202 ... yes
71/203 ... yes
7/204 ... yes
56/205 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.4.4)
Git version >= 2.9.5 ? ... yes (2.18.1)
Git user has default SSH configuration? ... yes
Active users: ... 26
Checking GitLab ... Finished

Possible fixes