Health check Error 500
Summary
The health_check endpoint produces a 500 error page ("Whoops, something went wrong on our end.") about half of the time it is accessed. This occurs since the update to v12.5.0, on 3 different Omnibus instances I have access to (completely different environments each time).
No problem on a 4th Omnibus instance still at v12.4.1, no problem on previous version. I have the endpoints monitored, so I would have noticed.
Steps to reproduce
Go to the instance's /health_check?token=. If no 500 error yet, refresh a couple time.
Example Project
N/A
What is the current bug behavior?
A 500 error page occurs about half of the time
What is the expected correct behavior?
Displaying "success" or some other status info
Relevant logs and/or screenshots
==> /var/log/gitlab/gitlab-rails/production.log <==
Started GET "/health_check?token=[FILTERED]" for A.B.C.D at 2019-11-24 18:36:32 +0100
Processing by HealthCheckController#index as HTML
Parameters: {"token"=>"[FILTERED]"}
health_check failed: Unable to write to cache.
Completed 500 Internal Server Error in 28ms (Views: 1.1ms | ActiveRecord: 2.4ms | Elasticsearch: 0.0ms)
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
$ sudo gitlab-rake gitlab:env:info
<snip>
GitLab information
Version: 12.5.0
Revision: 1f0ab8978ef
<snip>
GitLab Shell
Version: 10.2.0
Results of GitLab application Check
Expand for output related to the GitLab application check
$ sudo gitlab-rake gitlab:check SANITIZE=true
Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 10.2.0 ? ... OK (10.2.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found <snip> users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
<snip> (yes)
Redis version >= 2.8.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.3)
Git version >= 2.22.0 ? ... yes (2.22.0)
Git user has default SSH configuration? ... yes
Active users: ... <snip>
Is authorized keys file accessible? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
Edited by 🤖 GitLab Bot 🤖