Self hosted instance isn't loading js files after update

Summary

Our org is running a self hosted Gitlab instance which I'm updating on a weekly basis. After updating it this Tuesday (5th May) to version 12.10.1 (e658772bd63) JS files stopped loading.

I've tried gitlab-ctl reconfigure && gitlab-ctl restart, but the issue persists.

Steps to reproduce

I just ran apt-get update followed by apt-get upgrade. Since I do this every week, it was an upgrade from 12.10.1-ce to 12.10.3-ce. I was in a screen my session didn't drop mid way and I don't remember anything unusual.

Example Project

This happens on all of projects/pages of the running Gitlab instance.

What is the current bug behavior?

JS file requests are returning 404. Instance is not functional.

What is the expected correct behavior?

JS files load successfully.

Relevant logs and/or screenshots

DeepinScreenshot_select-area_20200508182233

The requested files are not present on the system. I can see other bundle files present:

/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/runtime.92169e35.bundle.js
/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/runtime.92169e35.bundle.js.gz
/opt/gitlab/embedded/service/gitlab-rails/public/assets/webpack/runtime.92169e35.bundle.js.map.gz

Requesting them works https://gitlab.cytec.bg/assets/webpack/runtime.92169e35.bundle.js

dpkg.log 
2020-05-05 09:29:23 upgrade gitlab-ce:amd64 12.10.1-ce.0 12.10.3-ce.0
2020-05-05 09:29:23 status half-configured gitlab-ce:amd64 12.10.1-ce.0
2020-05-05 09:29:23 status unpacked gitlab-ce:amd64 12.10.1-ce.0
2020-05-05 09:29:23 status half-installed gitlab-ce:amd64 12.10.1-ce.0
2020-05-05 09:30:52 status half-installed gitlab-ce:amd64 12.10.1-ce.0
2020-05-05 09:30:56 status unpacked gitlab-ce:amd64 12.10.3-ce.0
2020-05-05 09:30:58 status unpacked gitlab-ce:amd64 12.10.3-ce.0
2020-05-05 09:30:59 configure gitlab-ce:amd64 12.10.3-ce.0 
2020-05-05 09:30:59 status unpacked gitlab-ce:amd64 12.10.3-ce.0
2020-05-05 09:30:59 status half-configured gitlab-ce:amd64 12.10.3-ce.0
2020-05-05 09:32:36 status installed gitlab-ce:amd64 12.10.3-ce.0
reconfigure log from the update 
cat 1588660279.log


Logfile created on 2020-05-05 09:31:19 +0300 by logger.rb/66358


[2020-05-05T09:31:19+03:00] INFO: Started chef-zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded
One version per cookbook


[2020-05-05T09:31:20+03:00] INFO: *** Chef 14.14.29 ***
[2020-05-05T09:31:20+03:00] INFO: Platform: x86_64-linux
[2020-05-05T09:31:20+03:00] INFO: Chef-client pid: 7488
[2020-05-05T09:31:20+03:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2020-05-05T09:31:22+03:00] INFO: Setting the run_list to ["recipe[gitlab::config]", "recipe[postgresql::bin]"] from CLI options
[2020-05-05T09:31:22+03:00] INFO: Run List is [recipe[gitlab::config], recipe[postgresql::bin]]
[2020-05-05T09:31:22+03:00] INFO: Run List expands to [gitlab::config, postgresql::bin]
[2020-05-05T09:31:22+03:00] INFO: Starting Chef Run for gitlab.cytec.bg
[2020-05-05T09:31:22+03:00] INFO: Running start handlers
[2020-05-05T09:31:22+03:00] INFO: Start handlers complete.
[2020-05-05T09:31:23+03:00] INFO: Loading cookbooks [gitlab@0.0.1, postgresql@0.1.0, package@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@4.3.0, acme@4.1.1, crond@0.1.0]
[2020-05-05T09:31:28+03:00] INFO: Chef Run complete in 6.719218656 seconds
[2020-05-05T09:31:28+03:00] INFO: Running report handlers
[2020-05-05T09:31:28+03:00] INFO: Report handlers complete


cat 1588660302.log



Logfile created on 2020-05-05 09:31:42 +0300 by logger.rb/66358


[2020-05-05T09:31:42+03:00] INFO: Started chef-zero at chefzero://localhost:1 with repository at /opt/gitlab/embedded
One version per cookbook


[2020-05-05T09:31:42+03:00] INFO: *** Chef 14.14.29 ***
[2020-05-05T09:31:42+03:00] INFO: Platform: x86_64-linux
[2020-05-05T09:31:42+03:00] INFO: Chef-client pid: 7643
[2020-05-05T09:31:42+03:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2020-05-05T09:31:44+03:00] INFO: Setting the run_list to ["recipe[gitlab]"] from CLI options
[2020-05-05T09:31:44+03:00] INFO: Run List is [recipe[gitlab]]
[2020-05-05T09:31:44+03:00] INFO: Run List expands to [gitlab]
[2020-05-05T09:31:44+03:00] INFO: Starting Chef Run for gitlab.cytec.bg
[2020-05-05T09:31:44+03:00] INFO: Running start handlers
[2020-05-05T09:31:44+03:00] INFO: Start handlers complete.
[2020-05-05T09:31:45+03:00] INFO: Loading cookbooks [gitlab@0.0.1, package@0.1.0, postgresql@0.1.0, redis@0.1.0, monitoring@0.1.0, registry@0.1.0, mattermost@0.1.0, consul@0.1.0, gitaly@0.1.0, praefect@0.1.0, letsencrypt@0.1.0, nginx@0.1.0, runit@4.3.0, acme@4.1.1, crond@0.1.0]
[2020-05-05T09:31:50+03:00] WARN: Environment variable LANG specifies a non-UTF-8 locale. GitLab requires UTF-8 encoding to function properly. Please check your locale settings.
[2020-05-05T09:31:50+03:00] WARN: Selected systemd because systemctl shows .mount units
[2020-05-05T09:31:52+03:00] INFO: The plugin path /etc/chef/ohai/plugins does not exist. Skipping...
[2020-05-05T09:31:53+03:00] WARN: only_if block for templatesymlink[Create a gitlab_pages_secret and create a symlink to Rails root] returned a string, did you mean to run a command?
[2020-05-05T09:31:53+03:00] INFO: file[/opt/gitlab/embedded/service/gitlab-rails/db/structure.sql] owner changed to 998
[2020-05-05T09:31:53+03:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/VERSION] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/VERSION.chef-20200505093153.216430
[2020-05-05T09:31:53+03:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/VERSION] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/VERSION.chef-20200316101148.218236
[2020-05-05T09:31:53+03:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/VERSION] updated file contents /var/opt/gitlab/gitlab-rails/VERSION
[2020-05-05T09:31:53+03:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/REVISION] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/REVISION.chef-20200505093153.220593
[2020-05-05T09:31:53+03:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/REVISION] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-rails/REVISION.chef-20200316101148.228738
[2020-05-05T09:31:53+03:00] INFO: remote_file[/var/opt/gitlab/gitlab-rails/REVISION] updated file contents /var/opt/gitlab/gitlab-rails/REVISION
[2020-05-05T09:31:53+03:00] INFO: ruby_block[Move existing certs and link to /opt/gitlab/embedded/ssl/certs] called
[2020-05-05T09:31:53+03:00] INFO: file[/var/opt/gitlab/gitaly/VERSION] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitaly/VERSION.chef-20200505093153.506626
[2020-05-05T09:31:53+03:00] INFO: file[/var/opt/gitlab/gitaly/VERSION] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitaly/VERSION.chef-20200316101148.623873
[2020-05-05T09:31:53+03:00] INFO: file[/var/opt/gitlab/gitaly/VERSION] updated file contents /var/opt/gitlab/gitaly/VERSION
[2020-05-05T09:31:53+03:00] INFO: file[/opt/gitlab/sv/postgresql/supervise/status] owner changed to 996
[2020-05-05T09:31:53+03:00] INFO: file[/opt/gitlab/sv/postgresql/supervise/status] group changed to 997
[2020-05-05T09:31:53+03:00] INFO: file[/opt/gitlab/sv/postgresql/supervise/status] updated atime and mtime to 2020-05-05 09:31:53 +0300
[2020-05-05T09:32:13+03:00] INFO: bash[migrate gitlab-rails database] ran successfully
[2020-05-05T09:32:13+03:00] INFO: bash[migrate gitlab-rails database] sending run action to execute[clear the gitlab-rails cache] (immediate)
[2020-05-05T09:32:30+03:00] INFO: execute[clear the gitlab-rails cache] ran successfully
[2020-05-05T09:32:30+03:00] INFO: bash[set ownership of old migration log files] ran successfully
[2020-05-05T09:32:30+03:00] INFO: runit_service[unicorn] running why-run enable action to support before action
[2020-05-05T09:32:30+03:00] INFO: file[/var/opt/gitlab/gitlab-workhorse/VERSION] backed up to /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-workhorse/VERSION.chef-20200505093230.994282
[2020-05-05T09:32:30+03:00] INFO: file[/var/opt/gitlab/gitlab-workhorse/VERSION] removed backup at /opt/gitlab/embedded/cookbooks/cache/backup/var/opt/gitlab/gitlab-workhorse/VERSION.chef-20200304104340.589465
[2020-05-05T09:32:30+03:00] INFO: file[/var/opt/gitlab/gitlab-workhorse/VERSION] updated file contents /var/opt/gitlab/gitlab-workhorse/VERSION
[2020-05-05T09:32:31+03:00] INFO: ruby_block[populate Grafana configuration options] called
[2020-05-05T09:32:31+03:00] INFO: file[/var/opt/gitlab/gitaly/VERSION] sending hup action to runit_service[gitaly] (delayed)
[2020-05-05T09:32:31+03:00] INFO: runit_service[gitaly] signalled (HUP)
[2020-05-05T09:32:31+03:00] INFO: runit_service[gitaly] sent hup
[2020-05-05T09:32:31+03:00] INFO: file[/var/opt/gitlab/gitlab-workhorse/VERSION] sending restart action to runit_service[gitlab-workhorse] (delayed)
[2020-05-05T09:32:32+03:00] INFO: Chef Run complete in 47.657672045 seconds
[2020-05-05T09:32:32+03:00] INFO: Running report handlers
[2020-05-05T09:32:32+03:00] INFO: Report handlers complete

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:		Ubuntu 18.04
Current User:	git
Using RVM:	no
Ruby Version:	2.6.5p114
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.7
Git Version:	2.26.2
Sidekiq Version:5.2.7
Go Version:	unknown

GitLab information
Version:	12.10.3
Revision:	e01ceba1b49
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	11.7
URL:		https://gitlab.cytec.bg
HTTP Clone URL:	https://gitlab.cytec.bg/some-group/some-project.git
SSH Clone URL:	git@gitlab.cytec.bg:some-group/some-project.git
Using LDAP:	yes
Using Omniauth:	yes
Omniauth Providers:


GitLab Shell
Version:	12.2.0
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab subtasks ...

Checking GitLab Shell ...


GitLab Shell: ... GitLab Shell version >= 12.2.0 ? ... OK (12.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Gitaly ...


Gitaly: ... default ... OK


Checking Gitaly ... Finished


Checking Sidekiq ...


Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Checking Incoming Email ...


Incoming Email: ... Reply by email is disabled in config/gitlab.yml


Checking Incoming Email ... Finished


Checking LDAP ...


LDAP: ... Server: ldapmain
LDAP authentication... Anonymous. No bind_dn or password configured
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 17 users of 100 limit.


Checking LDAP ... Finished


Checking GitLab App ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
5/2 ... yes
5/3 ... yes
5/4 ... yes
5/5 ... yes
6/6 ... yes
6/7 ... yes
5/8 ... yes
5/9 ... yes
5/10 ... yes
5/11 ... yes
5/12 ... yes
5/13 ... yes
7/14 ... yes
6/16 ... yes
6/17 ... yes
6/18 ... yes
5/19 ... yes
6/20 ... yes
6/21 ... yes
6/22 ... yes
6/23 ... yes
6/24 ... yes
6/25 ... yes
6/26 ... yes
5/27 ... yes
7/28 ... yes
7/29 ... yes
5/30 ... yes
5/31 ... yes
6/32 ... yes
5/33 ... yes
6/34 ... yes
12/35 ... yes
7/36 ... yes
12/37 ... yes
12/38 ... yes
12/39 ... yes
7/40 ... yes
7/41 ... yes
12/42 ... yes
12/43 ... yes
12/44 ... yes
12/45 ... yes
5/46 ... yes
12/47 ... yes
7/48 ... yes
4/49 ... yes
6/51 ... yes
7/52 ... yes
12/53 ... yes
2/56 ... yes
22/58 ... yes
22/60 ... yes
22/62 ... yes
23/63 ... yes
23/64 ... yes
22/66 ... yes
2/67 ... yes
6/68 ... yes
22/69 ... yes
22/70 ... yes
7/71 ... yes
6/72 ... yes
21/73 ... yes
26/74 ... yes
28/75 ... yes
28/76 ... yes
28/77 ... yes
2/78 ... yes
28/79 ... yes
7/80 ... yes
7/81 ... yes
3/82 ... yes
7/84 ... yes
6/85 ... yes
12/86 ... yes
6/88 ... yes
6/91 ... yes
6/92 ... yes
6/93 ... yes
6/94 ... yes
2/95 ... yes
2/96 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.5)
Git version >= 2.22.0 ? ... yes (2.26.2)
Git user has default SSH configuration? ... yes
Active users: ... 16
Is authorized keys file accessible? ... yes


Checking GitLab App ... Finished


Checking GitLab subtasks ... Finished

Possible fixes

#202651 (closed) looks related, but I was instructed there to create a new issue.