LDAP operations error causes users to become blocked
Summary
Users on my company Gitlab instance randomly become blocked - causing them to be sent back to the login screen to log in again (which takes several tries).
The application log says the user does not exist anymore
causing the user to be blocked. Upon investigation, it appears the LDAP server my company uses occasionally returns an operations error
- can seen in the production log.
In code, the ldap_search
function returns an empty list (adapter.rb line 60) when this happens and causes users to be blocked when a credential check happens.
Though there might be an issue with the LDAP server, I don't think the way LDAP search errors are being handled very well.
Momentarily, I've put in a hack (ftechz/gitlab-ce@35efa689) in my Gitlab instance to retry when a Operations error
occurs by first making a new LDAP connection (simply retrying didn't work) then trying again.
Steps to reproduce
- Have a possibly flaky LDAP server
- Login
- Wait for a few hours
Example Project
N/A
What is the current bug behavior?
User becomes blocked when LDAP server doesn't respond properly
What is the expected correct behavior?
Retry or don't block if it is an intermittent issue.
Relevant logs and/or screenshots
application.log
June 08, 2017 09:29: LDAP account "[CN...]" does not exist anymore, blocking Gitlab user "[User]" ([email])
June 08, 2017 10:08: (LDAP) saving user [email] from login with extern_uid => [CN...]
June 08, 2017 10:08: LDAP account "[CN...]" is not disabled anymore, unblocking Gitlab user "[User]" ([email])
production.log
LDAP search error: Operations Error
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
System information System: Ubuntu 12.04 Current User: git Using RVM: no Ruby Version: 2.3.3p222 Gem Version: 2.6.6 Bundler Version:1.13.7 Rake Version: 10.5.0 Redis Version: 3.2.5 Git Version: 2.11.1 Sidekiq Version:4.2.7GitLab information Version: 9.1.4 Revision: fed799a Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: postgresql URL: https://company.com HTTP Clone URL: https://company.com/some-group/some-project.git SSH Clone URL: git@company.com:some-group/some-project.git Using LDAP: yes Using Omniauth: no
GitLab Shell Version: 5.0.2 Repository storage paths:
- default: /home/git/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git
Results of GitLab application Check
Expand for output related to the GitLab application check
Checking GitLab Shell ...
GitLab Shell version >= 5.0.2 ? ... OK (5.0.2) Repo base directory exists? default... yes Repo storage directories are symlinks? default... no Repo paths owned by git:git? default... yes Repo paths access is drwxrws---? default... yes hooks directories in repos are links: ... 6/1 ... ok 4/2 ... ok 6/3 ... ok 6/4 ... ok 6/5 ... ok 17/10 ... ok 12/11 ... ok 4/13 ... ok 2/14 ... repository is empty 6/15 ... ok 2/17 ... ok 6/18 ... ok 13/19 ... ok 24/20 ... ok 16/21 ... ok 6/22 ... ok 6/23 ... ok 26/24 ... ok 26/26 ... ok 26/27 ... ok 5/28 ... ok 32/29 ... ok 4/30 ... ok 11/31 ... ok 6/32 ... ok 26/33 ... ok 6/34 ... ok 39/36 ... ok 38/37 ... ok 38/38 ... ok 6/39 ... ok 6/40 ... ok 39/41 ... ok 6/42 ... ok 18/43 ... repository is empty 11/45 ... ok 39/47 ... ok 47/48 ... ok 47/49 ... ok 18/50 ... ok 6/53 ... ok 14/54 ... ok 6/55 ... ok 6/56 ... ok 49/57 ... ok 38/58 ... ok 49/59 ... ok 39/61 ... ok 39/64 ... ok 7/68 ... ok 6/69 ... ok 41/72 ... ok 41/73 ... ok 41/74 ... ok 41/75 ... ok 41/74 ... ok 41/75 ... ok 41/76 ... ok 45/77 ... ok 45/79 ... ok 47/80 ... ok 6/81 ... ok 6/82 ... ok 41/84 ... ok 41/85 ... ok 6/87 ... ok 16/89 ... ok 33/90 ... ok 53/91 ... ok 7/92 ... ok 16/93 ... ok 6/95 ... ok 6/96 ... ok 25/97 ... ok 6/98 ... ok 55/99 ... ok 40/100 ... ok 2/101 ... ok 39/102 ... ok 26/103 ... ok 26/104 ... ok 42/105 ... ok 4/106 ... ok 11/107 ... ok 55/108 ... ok 6/109 ... ok 29/110 ... ok 11/111 ... ok 39/112 ... ok 39/113 ... ok 11/114 ... ok 6/115 ... ok 49/116 ... ok Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Check GitLab API access: OK Access to /home/gitlab/.ssh/authorized_keys: OK Send ping to redis server: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
Server: ldapmain LDAP authentication... Success LDAP users with access to your GitLab server (only showing the first 100 results) (Results removed) Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config outdated? ... no Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory setup correctly? ... no Try fixing it: sudo chown -R git /var/opt/gitlab/gitlab-rails/uploads sudo find /var/opt/gitlab/gitlab-rails/uploads -type f -exec chmod 0644 {} ; sudo find /var/opt/gitlab/gitlab-rails/uploads -type d -not -path /var/opt/gitlab/gitlab-rails/uploads -exec chmod 0700 {} ; For more information see: doc/install/installation.md in section "GitLab" Please fix the error above and rerun the checks. Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) projects have namespace: ... 6/1 ... yes 4/2 ... yes 6/3 ... yes 6/4 ... yes 6/5 ... yes 17/10 ... yes 12/11 ... yes 4/13 ... yes 2/14 ... yes 6/15 ... yes 2/17 ... yes 6/18 ... yes 13/19 ... yes 24/20 ... yes 16/21 ... yes 6/22 ... yes 6/23 ... yes 26/24 ... yes 26/26 ... yes 26/27 ... yes 5/28 ... yes 32/29 ... yes 4/30 ... yes 11/31 ... yes 6/32 ... yes 26/33 ... yes 6/34 ... yes 39/36 ... yes 38/37 ... yes 38/38 ... yes 6/39 ... yes 6/40 ... yes 39/41 ... yes 6/42 ... yes 18/43 ... yes 11/45 ... yes 39/47 ... yes 47/48 ... yes 47/49 ... yes 18/50 ... yes 6/53 ... yes 14/54 ... yes 6/55 ... yes 6/56 ... yes 49/57 ... yes 38/58 ... yes 49/59 ... yes 39/61 ... yes 39/64 ... yes 7/68 ... yes 6/69 ... yes 41/72 ... yes 41/73 ... yes 41/74 ... yes 41/75 ... yes 41/76 ... yes 45/77 ... yes 45/79 ... yes 47/80 ... yes 6/81 ... yes 6/82 ... yes 41/84 ... yes 41/85 ... yes 6/87 ... yes 16/89 ... yes 33/90 ... yes 53/91 ... yes 7/92 ... yes 16/93 ... yes 6/95 ... yes 6/96 ... yes 25/97 ... yes 6/98 ... yes 55/99 ... yes 40/100 ... yes 2/101 ... yes 39/102 ... yes 26/103 ... yes 26/104 ... yes 42/105 ... yes 4/106 ... yes 11/107 ... yes 55/108 ... yes 6/109 ... yes 29/110 ... yes 11/111 ... yes 39/112 ... yes 39/113 ... yes 11/114 ... yes 6/115 ... yes 49/116 ... yes Redis version >= 2.8.0? ... yes Ruby version >= 2.1.0 ? ... yes (2.3.3) Your git bin path is "/opt/gitlab/embedded/bin/git" Git version >= 2.7.3 ? ... yes (2.11.1) Active users: 31
Checking GitLab ... Finished
Possible fixes
Hacked up solution ftechz/gitlab-ce!1