Sporadic server Errors since upgrade to 10.4.0-ee
Summary
We're seeing sporadic HTTP 500 Server errors, since an upgrade to 10.4.0-ee this morning.
Steps to reproduce
Unfortunately reproducing seems somewhat random, but we do see this being logged in /var/log/gitlab/gitlab-rails/production.log:
NoMethodError (undefined method `downcase' for false:FalseClass):
lib/gitlab/ldap/person.rb:124:in `map'
lib/gitlab/ldap/person.rb:124:in `validate_entry'
lib/gitlab/ldap/person.rb:71:in `initialize'
lib/gitlab/ldap/adapter.rb:39:in `new'
lib/gitlab/ldap/adapter.rb:39:in `block in users'
lib/gitlab/ldap/adapter.rb:38:in `map'
lib/gitlab/ldap/adapter.rb:38:in `users'
lib/gitlab/ldap/adapter.rb:44:in `user'
lib/gitlab/ldap/person.rb:23:in `find_by_dn'
lib/gitlab/ldap/access.rb:72:in `find_ldap_user'
lib/gitlab/ldap/access.rb:81:in `ldap_user'
lib/gitlab/ldap/access.rb:40:in `allowed?'
lib/gitlab/ldap/access.rb:21:in `block in allowed?'
lib/gitlab/ldap/access.rb:13:in `block in open'
lib/gitlab/ldap/adapter.rb:14:in `block in open'
lib/gitlab/ldap/adapter.rb:13:in `open'
lib/gitlab/ldap/access.rb:12:in `open'
lib/gitlab/ldap/access.rb:18:in `allowed?'
app/controllers/application_controller.rb:220:in `ldap_security_check'
lib/gitlab/middleware/multipart.rb:93:in `call'
lib/gitlab/request_profiler/middleware.rb:14:in `call'
lib/gitlab/jira/middleware.rb:15:in `call'
lib/gitlab/middleware/go.rb:18:in `call'
lib/gitlab/etag_caching/middleware.rb:11:in `call'
lib/gitlab/middleware/read_only.rb:31:in `call'
lib/gitlab/request_context.rb:18:in `call'
lib/gitlab/metrics/requests_rack_middleware.rb:27:in `call'
What is the current bug behavior?
Visitors see "Server Error" when performing everyday-activities, although hitting reload makes the problem go away.
What is the expected correct behavior?
No server errors should be found.
Relevant logs and/or screenshots
See above.
Output of checks
Expand for output related to GitLab checks
Checking GitLab Shell ...GitLab Shell version >= 5.11.0 ? ... OK (5.11.0) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:root, or git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 42/2 ... ok 8/5 ... ok 20/6 ... ok 20/7 ... ok 20/8 ... ok 4/9 ... ok 44/10 ... ok 2/11 ... ok 41/12 ... ok 8/13 ... ok 20/15 ... ok 42/18 ... ok 4/19 ... ok 8/23 ... ok 41/25 ... ok 41/26 ... ok 2/27 ... ok 42/29 ... ok 8/30 ... ok 4/31 ... ok 44/32 ... ok 8/33 ... ok 4/34 ... ok 20/38 ... ok 15/39 ... ok 22/40 ... ok 22/41 ... ok 22/43 ... ok 4/44 ... ok 4/45 ... ok 20/46 ... ok 23/47 ... ok 24/48 ... ok 34/49 ... ok 4/52 ... ok 24/53 ... ok 24/54 ... ok 24/55 ... ok 20/56 ... ok 25/57 ... ok 6/59 ... ok 44/62 ... ok 4/63 ... ok 23/64 ... ok 42/65 ... ok 41/66 ... ok 20/68 ... ok 10/69 ... ok 6/72 ... ok 9/73 ... ok 20/74 ... ok 8/75 ... ok 16/76 ... ok 20/77 ... ok 38/78 ... ok 41/79 ... ok 51/81 ... ok 19/82 ... ok 19/83 ... ok 19/85 ... ok 4/86 ... ok 5/87 ... ok 9/88 ... ok 27/89 ... ok 14/90 ... ok 24/91 ... ok 27/93 ... ok 4/94 ... ok 4/95 ... ok 8/97 ... ok 18/98 ... ok 24/99 ... ok 34/100 ... ok 38/102 ... ok 13/103 ... ok 24/105 ... ok 34/106 ... ok 34/107 ... ok 4/108 ... ok 19/109 ... ok 43/111 ... ok 4/112 ... ok 38/114 ... ok 14/116 ... ok 42/117 ... ok 13/121 ... ok 8/122 ... ok 20/123 ... ok 20/124 ... ok 8/128 ... ok 4/129 ... ok 20/130 ... ok 27/131 ... ok 20/132 ... ok 20/133 ... ok 38/137 ... ok 8/138 ... ok 38/139 ... ok 24/140 ... ok 40/141 ... ok 44/145 ... ok 44/146 ... ok 20/147 ... ok 20/148 ... ok 6/149 ... ok 4/150 ... ok 39/151 ... ok 39/152 ... ok 39/153 ... ok 39/154 ... ok 39/155 ... ok 39/156 ... ok 39/157 ... ok 39/158 ... ok 39/160 ... ok 39/161 ... ok 39/162 ... ok 39/163 ... ok 39/164 ... ok 39/166 ... ok 39/168 ... ok 41/169 ... ok 41/171 ... ok 41/172 ... ok 41/173 ... ok 43/175 ... ok 43/177 ... ok 6/180 ... ok 38/181 ... ok 39/183 ... ok 44/186 ... ok 42/188 ... ok 63/189 ... ok 63/191 ... ok 38/192 ... ok 39/195 ... ok 39/196 ... ok 39/197 ... ok 39/198 ... ok 8/199 ... ok 39/200 ... ok 45/201 ... ok 34/202 ... ok 34/204 ... ok 42/205 ... ok 20/207 ... ok 9/210 ... ok 19/211 ... ok 4/219 ... ok 42/220 ... ok 8/221 ... ok 8/222 ... ok 9/223 ... ok 9/224 ... ok 40/225 ... ok 34/227 ... ok 39/228 ... ok 34/229 ... ok 46/230 ... ok 46/231 ... ok 42/232 ... ok 34/235 ... ok 4/240 ... ok 34/241 ... ok 34/242 ... ok 8/245 ... ok 46/247 ... ok 46/248 ... ok 46/249 ... ok 34/250 ... ok 46/251 ... ok 46/252 ... ok 45/255 ... ok 42/256 ... ok 41/257 ... ok 35/258 ... ok 39/262 ... ok 41/264 ... ok 42/266 ... ok 12/267 ... ok 36/268 ... ok 39/269 ... ok 4/270 ... ok 19/272 ... ok 20/275 ... ok 42/281 ... repository is empty 11/285 ... ok 23/287 ... ok 10/288 ... ok 39/289 ... ok 8/291 ... ok 39/294 ... ok 8/295 ... ok 8/296 ... ok 20/297 ... repository is empty 34/298 ... ok 6/299 ... ok 26/300 ... ok 17/303 ... ok 6/304 ... ok 42/308 ... ok 42/309 ... ok 20/310 ... ok 41/311 ... ok 41/312 ... ok 41/313 ... ok 6/314 ... ok 41/315 ... repository is empty 41/316 ... ok 41/317 ... repository is empty 51/318 ... ok 51/319 ... ok 44/320 ... ok 44/321 ... ok 20/322 ... ok 52/323 ... ok 8/329 ... ok 46/330 ... ok 51/331 ... ok 19/332 ... repository is empty 26/333 ... ok 25/334 ... ok 25/335 ... ok 8/336 ... ok 20/337 ... ok 20/338 ... ok 42/339 ... repository is empty 20/340 ... ok 42/341 ... repository is empty 43/342 ... ok 34/343 ... ok 43/344 ... ok 54/346 ... ok 44/347 ... ok 20/348 ... ok 20/349 ... ok 42/350 ... repository is empty 20/351 ... ok 20/352 ... ok 25/353 ... ok 52/354 ... ok 14/355 ... ok 20/356 ... ok 34/357 ... repository is empty 20/359 ... ok 20/360 ... ok 41/361 ... ok 39/362 ... ok 20/364 ... ok 41/365 ... ok 44/366 ... ok 49/369 ... ok 49/370 ... ok 49/371 ... ok 49/372 ... ok 49/373 ... ok 49/374 ... ok 49/375 ... ok 49/376 ... ok 51/377 ... ok 46/381 ... ok 35/382 ... ok 37/383 ... ok 39/384 ... ok 39/386 ... ok 10/387 ... ok 39/388 ... ok 20/389 ... ok 39/390 ... ok 20/393 ... ok 20/394 ... ok 20/395 ... ok 63/396 ... ok 36/397 ... ok 20/398 ... ok 42/399 ... ok 10/401 ... ok 68/402 ... ok 10/403 ... ok 41/404 ... ok 24/407 ... ok 13/409 ... ok 15/410 ... ok 19/412 ... ok 69/413 ... ok 69/414 ... ok 69/416 ... ok 69/417 ... ok 69/418 ... ok 25/419 ... ok 37/420 ... ok 41/421 ... ok 37/422 ... ok 48/424 ... ok 40/425 ... ok 38/426 ... ok 17/427 ... ok 17/428 ... ok 69/429 ... ok 39/430 ... ok 69/431 ... ok 69/432 ... ok 42/433 ... ok 20/434 ... ok 39/435 ... ok 39/436 ... ok 17/437 ... ok 39/439 ... ok 39/440 ... ok 37/442 ... ok 40/443 ... ok 9/444 ... ok 41/445 ... ok 9/446 ... ok 34/447 ... ok 39/448 ... repository is empty 48/450 ... ok 17/451 ... ok 38/453 ... ok 34/455 ... ok 41/456 ... ok 41/457 ... ok 41/458 ... ok 41/460 ... ok 41/461 ... ok 41/462 ... ok 41/463 ... ok 40/464 ... ok 37/465 ... ok 40/466 ... ok 63/467 ... ok 35/468 ... ok 21/469 ... ok 46/470 ... ok 13/471 ... ok 44/472 ... ok 35/473 ... ok 37/474 ... ok 17/475 ... ok 41/476 ... ok 8/482 ... ok 8/483 ... ok 8/484 ... ok 8/485 ... ok 8/486 ... ok 28/487 ... ok 5/489 ... ok 8/490 ... ok 8/491 ... ok 73/492 ... repository is empty 41/493 ... ok 73/494 ... ok 41/495 ... ok 39/497 ... ok 13/498 ... ok 8/502 ... ok 42/503 ... ok 28/504 ... ok 51/505 ... ok 37/507 ... ok 46/508 ... ok 38/509 ... ok 20/511 ... ok 20/512 ... ok 30/513 ... ok 17/515 ... ok 8/516 ... ok 46/517 ... ok 20/518 ... ok 41/519 ... ok 14/521 ... ok 14/522 ... ok 35/523 ... ok 43/524 ... ok 43/525 ... ok 40/526 ... ok 37/527 ... ok 20/528 ... ok 46/529 ... ok 20/530 ... ok 40/531 ... ok 38/532 ... ok 39/533 ... ok 72/534 ... ok 46/535 ... ok 77/537 ... ok 46/539 ... ok 77/540 ... ok 78/541 ... ok 73/542 ... ok 20/543 ... ok 77/544 ... repository is empty 77/545 ... repository is empty 39/546 ... ok 79/547 ... ok 44/548 ... ok 40/550 ... ok 46/552 ... ok 19/553 ... ok 39/556 ... ok 68/557 ... repository is empty 8/558 ... ok 79/559 ... ok 42/560 ... ok 79/561 ... ok 46/562 ... ok 46/563 ... ok 73/564 ... ok 40/565 ... ok 9/566 ... ok 87/567 ... ok 73/568 ... ok 37/569 ... ok 45/570 ... ok 78/572 ... ok 20/573 ... ok 37/574 ... ok 40/575 ... ok 42/576 ... ok 19/577 ... ok 42/578 ... ok 8/579 ... ok 46/580 ... ok 78/582 ... ok 76/583 ... ok 42/584 ... ok 77/585 ... ok 44/586 ... ok 8/587 ... ok 74/588 ... ok 9/589 ... ok 37/591 ... ok 35/592 ... ok 37/593 ... repository is empty 38/594 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Redis available via internal API: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
IMAP server credentials are correct? ... no Try fixing it: An error occurred: Net::IMAP::NoResponseError: Authentication failed. Check that the information in config/gitlab.yml is correct For more information see: doc/administration/reply_by_email.md Please fix the error above and rerun the checks. Init.d configured correctly? ... skipped MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results)
Checking LDAP ... Finished
Checking GitLab ...
Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 42/2 ... yes 8/5 ... yes 20/6 ... yes 20/7 ... yes 20/8 ... yes 4/9 ... yes 44/10 ... yes 2/11 ... yes 41/12 ... yes 8/13 ... yes 20/15 ... yes 42/18 ... yes 4/19 ... yes 8/23 ... yes 41/25 ... yes 41/26 ... yes 2/27 ... yes 42/29 ... yes 8/30 ... yes 4/31 ... yes 44/32 ... yes 8/33 ... yes 4/34 ... yes 20/38 ... yes 15/39 ... yes 22/40 ... yes 22/41 ... yes 22/43 ... yes 4/44 ... yes 4/45 ... yes 20/46 ... yes 23/47 ... yes 24/48 ... yes 34/49 ... yes 4/52 ... yes 24/53 ... yes 24/54 ... yes 24/55 ... yes 20/56 ... yes 25/57 ... yes 6/59 ... yes 44/62 ... yes 4/63 ... yes 23/64 ... yes 42/65 ... yes 41/66 ... yes 20/68 ... yes 10/69 ... yes 6/72 ... yes 9/73 ... yes 20/74 ... yes 8/75 ... yes 16/76 ... yes 20/77 ... yes 38/78 ... yes 41/79 ... yes 51/81 ... yes 19/82 ... yes 19/83 ... yes 19/85 ... yes 4/86 ... yes 5/87 ... yes 9/88 ... yes 27/89 ... yes 14/90 ... yes 24/91 ... yes 27/93 ... yes 4/94 ... yes 4/95 ... yes 8/97 ... yes 18/98 ... yes 24/99 ... yes 34/100 ... yes 38/102 ... yes 13/103 ... yes 24/105 ... yes 34/106 ... yes 34/107 ... yes 4/108 ... yes 19/109 ... yes 43/111 ... yes 4/112 ... yes 38/114 ... yes 14/116 ... yes 42/117 ... yes 13/121 ... yes 8/122 ... yes 20/123 ... yes 20/124 ... yes 8/128 ... yes 4/129 ... yes 20/130 ... yes 27/131 ... yes 20/132 ... yes 20/133 ... yes 38/137 ... yes 8/138 ... yes 38/139 ... yes 24/140 ... yes 40/141 ... yes 44/145 ... yes 44/146 ... yes 20/147 ... yes 20/148 ... yes 6/149 ... yes 4/150 ... yes 39/151 ... yes 39/152 ... yes 39/153 ... yes 39/154 ... yes 39/155 ... yes 39/156 ... yes 39/157 ... yes 39/158 ... yes 39/160 ... yes 39/161 ... yes 39/162 ... yes 39/163 ... yes 39/164 ... yes 39/166 ... yes 39/168 ... yes 41/169 ... yes 41/171 ... yes 41/172 ... yes 41/173 ... yes 43/175 ... yes 43/177 ... yes 6/180 ... yes 38/181 ... yes 39/183 ... yes 44/186 ... yes 42/188 ... yes 63/189 ... yes 63/191 ... yes 38/192 ... yes 39/195 ... yes 39/196 ... yes 39/197 ... yes 39/198 ... yes 8/199 ... yes 39/200 ... yes 45/201 ... yes 34/202 ... yes 34/204 ... yes 42/205 ... yes 20/207 ... yes 9/210 ... yes 19/211 ... yes 4/219 ... yes 42/220 ... yes 8/221 ... yes 8/222 ... yes 9/223 ... yes 9/224 ... yes 40/225 ... yes 34/227 ... yes 39/228 ... yes 34/229 ... yes 46/230 ... yes 46/231 ... yes 42/232 ... yes 34/235 ... yes 4/240 ... yes 34/241 ... yes 34/242 ... yes 8/245 ... yes 46/247 ... yes 46/248 ... yes 46/249 ... yes 34/250 ... yes 46/251 ... yes 46/252 ... yes 45/255 ... yes 42/256 ... yes 41/257 ... yes 35/258 ... yes 39/262 ... yes 41/264 ... yes 42/266 ... yes 12/267 ... yes 36/268 ... yes 39/269 ... yes 4/270 ... yes 19/272 ... yes 20/275 ... yes 42/281 ... yes 11/285 ... yes 23/287 ... yes 10/288 ... yes 39/289 ... yes 8/291 ... yes 39/294 ... yes 8/295 ... yes 8/296 ... yes 20/297 ... yes 34/298 ... yes 6/299 ... yes 26/300 ... yes 17/303 ... yes 6/304 ... yes 42/308 ... yes 42/309 ... yes 20/310 ... yes 41/311 ... yes 41/312 ... yes 41/313 ... yes 6/314 ... yes 41/315 ... yes 41/316 ... yes 41/317 ... yes 51/318 ... yes 51/319 ... yes 44/320 ... yes 44/321 ... yes 20/322 ... yes 52/323 ... yes 8/329 ... yes 46/330 ... yes 51/331 ... yes 19/332 ... yes 26/333 ... yes 25/334 ... yes 25/335 ... yes 8/336 ... yes 20/337 ... yes 20/338 ... yes 42/339 ... yes 20/340 ... yes 42/341 ... yes 43/342 ... yes 34/343 ... yes 43/344 ... yes 54/346 ... yes 44/347 ... yes 20/348 ... yes 20/349 ... yes 42/350 ... yes 20/351 ... yes 20/352 ... yes 25/353 ... yes 52/354 ... yes 14/355 ... yes 20/356 ... yes 34/357 ... yes 20/359 ... yes 20/360 ... yes 41/361 ... yes 39/362 ... yes 20/364 ... yes 41/365 ... yes 44/366 ... yes 49/369 ... yes 49/370 ... yes 49/371 ... yes 49/372 ... yes 49/373 ... yes 49/374 ... yes 49/375 ... yes 49/376 ... yes 51/377 ... yes 46/381 ... yes 35/382 ... yes 37/383 ... yes 39/384 ... yes 39/386 ... yes 10/387 ... yes 39/388 ... yes 20/389 ... yes 39/390 ... yes 20/393 ... yes 20/394 ... yes 20/395 ... yes 63/396 ... yes 36/397 ... yes 20/398 ... yes 42/399 ... yes 10/401 ... yes 68/402 ... yes 10/403 ... yes 41/404 ... yes 24/407 ... yes 13/409 ... yes 15/410 ... yes 19/412 ... yes 69/413 ... yes 69/414 ... yes 69/416 ... yes 69/417 ... yes 69/418 ... yes 25/419 ... yes 37/420 ... yes 41/421 ... yes 37/422 ... yes 48/424 ... yes 40/425 ... yes 38/426 ... yes 17/427 ... yes 17/428 ... yes 69/429 ... yes 39/430 ... yes 69/431 ... yes 69/432 ... yes 42/433 ... yes 20/434 ... yes 39/435 ... yes 39/436 ... yes 17/437 ... yes 39/439 ... yes 39/440 ... yes 37/442 ... yes 40/443 ... yes 9/444 ... yes 41/445 ... yes 9/446 ... yes 34/447 ... yes 39/448 ... yes 48/450 ... yes 17/451 ... yes 38/453 ... yes 34/455 ... yes 41/456 ... yes 41/457 ... yes 41/458 ... yes 41/460 ... yes 41/461 ... yes 41/462 ... yes 41/463 ... yes 40/464 ... yes 37/465 ... yes 40/466 ... yes 63/467 ... yes 35/468 ... yes 21/469 ... yes 46/470 ... yes 13/471 ... yes 44/472 ... yes 35/473 ... yes 37/474 ... yes 17/475 ... yes 41/476 ... yes 8/482 ... yes 8/483 ... yes 8/484 ... yes 8/485 ... yes 8/486 ... yes 28/487 ... yes 5/489 ... yes 8/490 ... yes 8/491 ... yes 73/492 ... yes 41/493 ... yes 73/494 ... yes 41/495 ... yes 39/497 ... yes 13/498 ... yes 8/502 ... yes 42/503 ... yes 28/504 ... yes 51/505 ... yes 37/507 ... yes 46/508 ... yes 38/509 ... yes 20/511 ... yes 20/512 ... yes 30/513 ... yes 17/515 ... yes 8/516 ... yes 46/517 ... yes 20/518 ... yes 41/519 ... yes 14/521 ... yes 14/522 ... yes 35/523 ... yes 43/524 ... yes 43/525 ... yes 40/526 ... yes 37/527 ... yes 20/528 ... yes 46/529 ... yes 20/530 ... yes 40/531 ... yes 38/532 ... yes 39/533 ... yes 72/534 ... yes 46/535 ... yes 77/537 ... yes 46/539 ... yes 77/540 ... yes 78/541 ... yes 73/542 ... yes 20/543 ... yes 77/544 ... yes 77/545 ... yes 39/546 ... yes 79/547 ... yes 44/548 ... yes 40/550 ... yes 46/552 ... yes 19/553 ... yes 39/556 ... yes 68/557 ... yes 8/558 ... yes 79/559 ... yes 42/560 ... yes 79/561 ... yes 46/562 ... yes 46/563 ... yes 73/564 ... yes 40/565 ... yes 9/566 ... yes 87/567 ... yes 73/568 ... yes 37/569 ... yes 45/570 ... yes 78/572 ... yes 20/573 ... yes 37/574 ... yes 40/575 ... yes 42/576 ... yes 19/577 ... yes 42/578 ... yes 8/579 ... yes 46/580 ... yes 78/582 ... yes 76/583 ... yes 42/584 ... yes 77/585 ... yes 44/586 ... yes 8/587 ... yes 74/588 ... yes 9/589 ... yes 37/591 ... yes 35/592 ... yes 37/593 ... yes 38/594 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.3.5 ? ... yes (2.3.6) Git version >= 2.7.3 ? ... yes (2.14.3) Git user has default SSH configuration? ... yes Active users: ... 34 Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)
Checking GitLab ... Finished
Results of GitLab environment info
Expand for output related to GitLab environment info
root@gitlab:/var/log/gitlab/nginx# gitlab-rake gitlab:env:infoSystem information System: Debian 8.10 Proxy: no Current User: git Using RVM: no Ruby Version: 2.3.6p384 Gem Version: 2.6.13 Bundler Version:1.13.7 Rake Version: 12.3.0 Redis Version: 3.2.11 Git Version: 2.14.3 Sidekiq Version:5.0.5 Go Version: unknown
GitLab information Version: 10.4.0-ee Revision: 00592f0 Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql DB Version: 9.6.5 URL: https://gitlab.bytemark.co.uk HTTP Clone URL: https://gitlab.bytemark.co.uk/some-group/some-project.git SSH Clone URL: git@gitlab.bytemark.co.uk:some-group/some-project.git Elasticsearch: no Geo: no Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 5.11.0 Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git