System does not gracefully recover after missing secrets file
Everyone can contribute. Help move this issue forward while earning points, leveling up and collecting rewards.
Summary
I have just restored from backup (everything was smooth, except that I did not have secrets file) and now having trouble with CI in some projects. Specifically, when I run pipeline manually or after commit, the jobs seem to start, but terminal log never shows anything. It stuck for forever.
Even worse, when I go to CI settings for a project, it shows 500 error page.
I suspect that it is because I did not have a secrets file. It looks like the affected projects are the ones with secret variables. And it looks like gitlab is trying to load them from file and file does not have them (500 is not human friendly :( )
Just in case, to have 2FA reset I used this command sudo gitlab-rails runner 'User.update_all(otp_required_for_login: false, encrypted_otp_secret: nil)'
Steps to reproduce
Restore backup without gitlab-secrets.json file.
Example Project
N/A
What is the current bug behavior?
500 page in CI settings and forever-stuck builds.
What is the expected correct behavior?
Obviously, regular settings page and running builds.
Relevant logs and/or screenshots
Output of checks
sudo gitlab-rake gitlab:check SANITIZE=true does not show any errors.
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 18.04 Proxy: no Current User: git Using RVM: no Ruby Version: 2.4.4p296 Gem Version: 2.7.6 Bundler Version:1.16.2 Rake Version: 12.3.1 Redis Version: 3.2.11 Git Version: 2.17.1 Sidekiq Version:5.1.3 Go Version: unknownGitLab information Version: 11.1.2-ee Revision: f5babb03520 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.8 URL: https://git.dbogatov.org HTTP Clone URL: https://git.dbogatov.org/some-group/some-project.git SSH Clone URL: git@git.dbogatov.org:some-group/some-project.git Elasticsearch: yes Geo: no Using LDAP: no Using Omniauth: yes Omniauth Providers: github, bitbucket, gitlab, google_oauth2, facebook, twitter
GitLab Shell Version: 7.1.4 Repository storage paths: default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 7.1.4 ? ... OK (7.1.4) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 13/2 ... ok 2/3 ... ok 2/4 ... ok 2/5 ... ok 2/6 ... ok 2/7 ... ok 2/8 ... ok 2/9 ... ok 2/11 ... ok 2/13 ... ok 2/15 ... ok 2/16 ... ok 2/17 ... ok 2/22 ... ok 2/23 ... ok 2/24 ... repository is empty 2/25 ... ok 2/26 ... ok 2/27 ... ok 14/28 ... ok 2/29 ... ok 2/30 ... ok 2/31 ... ok 2/32 ... ok 2/33 ... ok 14/34 ... ok 2/35 ... ok 2/36 ... ok 2/37 ... ok 2/38 ... ok 2/39 ... ok 2/40 ... ok 26/41 ... ok 26/42 ... ok 2/44 ... ok 2/47 ... ok 2/49 ... ok 29/50 ... ok 29/53 ... ok 2/54 ... ok 30/55 ... ok 30/56 ... ok 2/59 ... ok 19/60 ... ok 19/61 ... ok 39/62 ... ok 30/63 ... ok 39/64 ... repository is empty 39/65 ... ok 30/66 ... ok 2/67 ... ok 42/68 ... ok 42/69 ... ok 30/70 ... ok 45/71 ... ok 26/72 ... ok 30/73 ... ok 55/74 ... ok 45/75 ... ok 30/76 ... ok 2/77 ... ok 45/78 ... ok 2/79 ... ok 2/80 ... ok 26/81 ... ok 26/82 ... ok 54/83 ... repository is empty 54/84 ... ok 55/86 ... ok 54/87 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... yes Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 13/2 ... yes 2/3 ... yes 2/4 ... yes 2/5 ... yes 2/6 ... yes 2/7 ... yes 2/8 ... yes 2/9 ... yes 2/11 ... yes 2/13 ... yes 2/15 ... yes 2/16 ... yes 2/17 ... yes 2/22 ... yes 2/23 ... yes 2/24 ... yes 2/25 ... yes 2/26 ... yes 2/27 ... yes 14/28 ... yes 2/29 ... yes 2/30 ... yes 2/31 ... yes 2/32 ... yes 2/33 ... yes 14/34 ... yes 2/35 ... yes 2/36 ... yes 2/37 ... yes 2/38 ... yes 2/39 ... yes 2/40 ... yes 26/41 ... yes 26/42 ... yes 2/44 ... yes 2/47 ... yes 2/49 ... yes 29/50 ... yes 29/53 ... yes 2/54 ... yes 30/55 ... yes 30/56 ... yes 2/59 ... yes 19/60 ... yes 19/61 ... yes 39/62 ... yes 30/63 ... yes 39/64 ... yes 39/65 ... yes 30/66 ... yes 2/67 ... yes 42/68 ... yes 42/69 ... yes 30/70 ... yes 45/71 ... yes 26/72 ... yes 30/73 ... yes 55/74 ... yes 45/75 ... yes 30/76 ... yes 2/77 ... yes 45/78 ... yes 2/79 ... yes 2/80 ... yes 26/81 ... yes 26/82 ... yes 54/83 ... yes 54/84 ... yes 55/86 ... yes 54/87 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.4.4) Git version >= 2.9.5 ? ... yes (2.17.1) Git user has default SSH configuration? ... yes Active users: ... 15 Elasticsearch version 5.1 - 5.5? ... yes (5.5.3)
Checking GitLab ... Finished
Possible fixes
Handle the case when secret variables are not in the secrets file.