System does not gracefully recover after missing secrets file

Summary

I have just restored from backup (everything was smooth, except that I did not have secrets file) and now having trouble with CI in some projects. Specifically, when I run pipeline manually or after commit, the jobs seem to start, but terminal log never shows anything. It stuck for forever.

Even worse, when I go to CI settings for a project, it shows 500 error page.

I suspect that it is because I did not have a secrets file. It looks like the affected projects are the ones with secret variables. And it looks like gitlab is trying to load them from file and file does not have them (500 is not human friendly :( )

Just in case, to have 2FA reset I used this command sudo gitlab-rails runner 'User.update_all(otp_required_for_login: false, encrypted_otp_secret: nil)'

Steps to reproduce

Restore backup without gitlab-secrets.json file.

Example Project

N/A

What is the current bug behavior?

500 page in CI settings and forever-stuck builds.

What is the expected correct behavior?

Obviously, regular settings page and running builds.

Relevant logs and/or screenshots

Output of checks

sudo gitlab-rake gitlab:check SANITIZE=true does not show any errors.

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         Ubuntu 18.04
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.4.4p296
Gem Version:    2.7.6
Bundler Version:1.16.2
Rake Version:   12.3.1
Redis Version:  3.2.11
Git Version:    2.17.1
Sidekiq Version:5.1.3
Go Version:     unknown
GitLab information
Version:        11.1.2-ee
Revision:       f5babb03520
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
DB Version:     9.6.8
URL:            https://git.dbogatov.org
HTTP Clone URL: https://git.dbogatov.org/some-group/some-project.git
SSH Clone URL:  git@git.dbogatov.org:some-group/some-project.git
Elasticsearch:  yes
Geo:            no
Using LDAP:     no
Using Omniauth: yes
Omniauth Providers: github, bitbucket, gitlab, google_oauth2, facebook, twitter
GitLab Shell
Version:        7.1.4
Repository storage paths:
default:      /var/opt/gitlab/git-data/repositories
Hooks:          /opt/gitlab/embedded/service/gitlab-shell/hooks
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab Shell ...
GitLab Shell version >= 7.1.4 ? ... OK (7.1.4)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
13/2 ... ok
2/3 ... ok
2/4 ... ok
2/5 ... ok
2/6 ... ok
2/7 ... ok
2/8 ... ok
2/9 ... ok
2/11 ... ok
2/13 ... ok
2/15 ... ok
2/16 ... ok
2/17 ... ok
2/22 ... ok
2/23 ... ok
2/24 ... repository is empty
2/25 ... ok
2/26 ... ok
2/27 ... ok
14/28 ... ok
2/29 ... ok
2/30 ... ok
2/31 ... ok
2/32 ... ok
2/33 ... ok
14/34 ... ok
2/35 ... ok
2/36 ... ok
2/37 ... ok
2/38 ... ok
2/39 ... ok
2/40 ... ok
26/41 ... ok
26/42 ... ok
2/44 ... ok
2/47 ... ok
2/49 ... ok
29/50 ... ok
29/53 ... ok
2/54 ... ok
30/55 ... ok
30/56 ... ok
2/59 ... ok
19/60 ... ok
19/61 ... ok
39/62 ... ok
30/63 ... ok
39/64 ... repository is empty
39/65 ... ok
30/66 ... ok
2/67 ... ok
42/68 ... ok
42/69 ... ok
30/70 ... ok
45/71 ... ok
26/72 ... ok
30/73 ... ok
55/74 ... ok
45/75 ... ok
30/76 ... ok
2/77 ... ok
45/78 ... ok
2/79 ... ok
2/80 ... ok
26/81 ... ok
26/82 ... ok
54/83 ... repository is empty
54/84 ... ok
55/86 ... ok
54/87 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... yes
Init.d configured correctly? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
13/2 ... yes
2/3 ... yes
2/4 ... yes
2/5 ... yes
2/6 ... yes
2/7 ... yes
2/8 ... yes
2/9 ... yes
2/11 ... yes
2/13 ... yes
2/15 ... yes
2/16 ... yes
2/17 ... yes
2/22 ... yes
2/23 ... yes
2/24 ... yes
2/25 ... yes
2/26 ... yes
2/27 ... yes
14/28 ... yes
2/29 ... yes
2/30 ... yes
2/31 ... yes
2/32 ... yes
2/33 ... yes
14/34 ... yes
2/35 ... yes
2/36 ... yes
2/37 ... yes
2/38 ... yes
2/39 ... yes
2/40 ... yes
26/41 ... yes
26/42 ... yes
2/44 ... yes
2/47 ... yes
2/49 ... yes
29/50 ... yes
29/53 ... yes
2/54 ... yes
30/55 ... yes
30/56 ... yes
2/59 ... yes
19/60 ... yes
19/61 ... yes
39/62 ... yes
30/63 ... yes
39/64 ... yes
39/65 ... yes
30/66 ... yes
2/67 ... yes
42/68 ... yes
42/69 ... yes
30/70 ... yes
45/71 ... yes
26/72 ... yes
30/73 ... yes
55/74 ... yes
45/75 ... yes
30/76 ... yes
2/77 ... yes
45/78 ... yes
2/79 ... yes
2/80 ... yes
26/81 ... yes
26/82 ... yes
54/83 ... yes
54/84 ... yes
55/86 ... yes
54/87 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.5 ? ... yes (2.4.4)
Git version >= 2.9.5 ? ... yes (2.17.1)
Git user has default SSH configuration? ... yes
Active users: ... 15
Elasticsearch version 5.1 - 5.5? ... yes (5.5.3)
Checking GitLab ... Finished

Possible fixes

Handle the case when secret variables are not in the secrets file.