LDAP operations error causes users to become blocked

Summary

Users on my company Gitlab instance randomly become blocked - causing them to be sent back to the login screen to log in again (which takes several tries).

The application log says the user does not exist anymore causing the user to be blocked. Upon investigation, it appears the LDAP server my company uses occasionally returns an operations error - can seen in the production log.

In code, the ldap_search function returns an empty list (adapter.rb line 60) when this happens and causes users to be blocked when a credential check happens.

Though there might be an issue with the LDAP server, I don't think the way LDAP search errors are being handled very well. Momentarily, I've put in a hack (ftechz/gitlab-ce@35efa689) in my Gitlab instance to retry when a Operations error occurs by first making a new LDAP connection (simply retrying didn't work) then trying again.

Steps to reproduce

1. Have a possibly flaky LDAP server
2. Login
3. Wait for a few hours

Example Project

N/A

What is the current bug behavior?

User becomes blocked when LDAP server doesn't respond properly

What is the expected correct behavior?

Retry or don't block if it is an intermittent issue.

Relevant logs and/or screenshots

application.log

June 08, 2017 09:29: LDAP account "[CN...]" does not exist anymore, blocking Gitlab user "[User]" ([email])
June 08, 2017 10:08: (LDAP) saving user [email] from login with extern_uid => [CN...]
June 08, 2017 10:08: LDAP account "[CN...]" is not disabled anymore, unblocking Gitlab user "[User]" ([email])

production.log

LDAP search error: Operations Error

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:         Ubuntu 12.04
Current User:   git
Using RVM:      no
Ruby Version:   2.3.3p222
Gem Version:    2.6.6
Bundler Version:1.13.7
Rake Version:   10.5.0
Redis Version:  3.2.5
Git Version:    2.11.1
Sidekiq Version:4.2.7
GitLab information
Version:        9.1.4
Revision:       fed799a
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
URL:            https://company.com
HTTP Clone URL: https://company.com/some-group/some-project.git
SSH Clone URL:  git@company.com:some-group/some-project.git
Using LDAP:     yes
Using Omniauth: no
GitLab Shell
Version:        5.0.2
Repository storage paths:

• default: /home/git/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab Shell ...
GitLab Shell version >= 5.0.2 ? ... OK (5.0.2)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
6/1 ... ok
4/2 ... ok
6/3 ... ok
6/4 ... ok
6/5 ... ok
17/10 ... ok
12/11 ... ok
4/13 ... ok
2/14 ... repository is empty
6/15 ... ok
2/17 ... ok
6/18 ... ok
13/19 ... ok
24/20 ... ok
16/21 ... ok
6/22 ... ok
6/23 ... ok
26/24 ... ok
26/26 ... ok
26/27 ... ok
5/28 ... ok
32/29 ... ok
4/30 ... ok
11/31 ... ok
6/32 ... ok
26/33 ... ok
6/34 ... ok
39/36 ... ok
38/37 ... ok
38/38 ... ok
6/39 ... ok
6/40 ... ok
39/41 ... ok
6/42 ... ok
18/43 ... repository is empty
11/45 ... ok
39/47 ... ok
47/48 ... ok
47/49 ... ok
18/50 ... ok
6/53 ... ok
14/54 ... ok
6/55 ... ok
6/56 ... ok
49/57 ... ok
38/58 ... ok
49/59 ... ok
39/61 ... ok
39/64 ... ok
7/68 ... ok
6/69 ... ok
41/72 ... ok
41/73 ... ok
41/74 ... ok
41/75 ... ok
41/74 ... ok
41/75 ... ok
41/76 ... ok
45/77 ... ok
45/79 ... ok
47/80 ... ok
6/81 ... ok
6/82 ... ok
41/84 ... ok
41/85 ... ok
6/87 ... ok
16/89 ... ok
33/90 ... ok
53/91 ... ok
7/92 ... ok
16/93 ... ok
6/95 ... ok
6/96 ... ok
25/97 ... ok
6/98 ... ok
55/99 ... ok
40/100 ... ok
2/101 ... ok
39/102 ... ok
26/103 ... ok
26/104 ... ok
42/105 ... ok
4/106 ... ok
11/107 ... ok
55/108 ... ok
6/109 ... ok
29/110 ... ok
11/111 ... ok
39/112 ... ok
39/113 ... ok
11/114 ... ok
6/115 ... ok
49/116 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /home/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
(Results removed)
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... no
Try fixing it:
sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads
sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ;
sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ;
For more information see:
doc/install/installation.md in section "GitLab"
Please fix the error above and rerun the checks.
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
6/1 ... yes
4/2 ... yes
6/3 ... yes
6/4 ... yes
6/5 ... yes
17/10 ... yes
12/11 ... yes
4/13 ... yes
2/14 ... yes
6/15 ... yes
2/17 ... yes
6/18 ... yes
13/19 ... yes
24/20 ... yes
16/21 ... yes
6/22 ... yes
6/23 ... yes
26/24 ... yes
26/26 ... yes
26/27 ... yes
5/28 ... yes
32/29 ... yes
4/30 ... yes
11/31 ... yes
6/32 ... yes
26/33 ... yes
6/34 ... yes
39/36 ... yes
38/37 ... yes
38/38 ... yes
6/39 ... yes
6/40 ... yes
39/41 ... yes
6/42 ... yes
18/43 ... yes
11/45 ... yes
39/47 ... yes
47/48 ... yes
47/49 ... yes
18/50 ... yes
6/53 ... yes
14/54 ... yes
6/55 ... yes
6/56 ... yes
49/57 ... yes
38/58 ... yes
49/59 ... yes
39/61 ... yes
39/64 ... yes
7/68 ... yes
6/69 ... yes
41/72 ... yes
41/73 ... yes
41/74 ... yes
41/75 ... yes
41/76 ... yes
45/77 ... yes
45/79 ... yes
47/80 ... yes
6/81 ... yes
6/82 ... yes
41/84 ... yes
41/85 ... yes
6/87 ... yes
16/89 ... yes
33/90 ... yes
53/91 ... yes
7/92 ... yes
16/93 ... yes
6/95 ... yes
6/96 ... yes
25/97 ... yes
6/98 ... yes
55/99 ... yes
40/100 ... yes
2/101 ... yes
39/102 ... yes
26/103 ... yes
26/104 ... yes
42/105 ... yes
4/106 ... yes
11/107 ... yes
55/108 ... yes
6/109 ... yes
29/110 ... yes
11/111 ... yes
39/112 ... yes
39/113 ... yes
11/114 ... yes
6/115 ... yes
49/116 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.11.1)
Active users: 31
Checking GitLab ... Finished

Possible fixes

Hacked up solution ftechz/gitlab-ce!1