User disappear randomly with LDAP group synchronization

Summary

Not sure if I am misconfiguring LDAP, but randomly I lose user from a Group I setup to synchronize with LDAP. When I click on "Sync Now" it is a hit or miss on when it update memberships correctly.

Originally I thought maybe I am just clicking on the "Sync Now" button too much so I just let it be, but I am guessing somehow during scheduled group sync the users disappear.

I can't find in the log on how to debug this issue. Looked in both application.log and production.log. Also checked with gitlab-ctl tail

Steps to reproduce

  • Click on "Sync Now" until you see user disappear in group

  • Wait a couple hours until you notice User disappear from the group

What is the current bug behavior?

User just disappears from the Group caused by LDAP Group Sync

What is the expected correct behavior?

User should not disappear from Group using the LDAP Group Synchronization

Relevant logs and/or screenshots

Please do note that our AD do use spaces in CN and OU

Pictures

25 min ago a sync happened...LDAP user be gone 😟 image

Manual sync to make it work image

Me going crazy in shell and looks good to me

irb(main):073:0> EE::Gitlab::LDAP::Sync::Group::execute_all_providers(Group.find_by(name: 'My_Group'))
Started syncing all providers for 'My_Group' group
Instantiating EE::Gitlab::LDAP::Group with LDIF:
dn: CN=My_Group,OU=Some Groups,DC=MyCompany,DC=local
cn: My_Group
member: CN=User 1,OU=MyCompany Users,DC=MyCompany,DC=local
member: CN=User 2,OU=MyCompany Users,DC=MyCompany,DC=local
member: CN=User 3,OU=MyCompany Users,DC=MyCompany,DC=local
member: CN=User 4,OU=MyCompany Users,DC=MyCompany,DC=local

Members in 'My_Group' LDAP group: ["cn=User 1,ou=MyCompany users,dc=MyCompany,dc=local", "cn=User 2,ou=MyCompany users,dc=MyCompany,dc=local", "cn=User 3,ou=MyCompany users,dc=MyCompany,dc=local", "cn=User 4,ou=MyCompany users,dc=MyCompany,dc=local"]
Resolved 'My_Group' group member access: {"cn=User 1,ou=MyCompany users,dc=MyCompany,dc=local"=>40, "cn=User 2,ou=MyCompany users,dc=MyCompany,dc=local"=>40, "cn=User 3,ou=MyCompany users,dc=MyCompany,dc=local"=>40, "cn=User 4,ou=MyCompany users,dc=MyCompany,dc=local"=>40}
Updating existing membership for 'My_Group' group
Adding new members to 'My_Group' group
EE::Gitlab::LDAP::Sync::Group: User with DN `cn=User 3,ou=MyCompany users,dc=MyCompany,dc=local` should have access to 'My_Group' group but there is no user in GitLab with that identity. Membership will be updated once the user signs in for the first time.
Finished syncing all pro

My ldap config in gitlab.rb


gitlab_rails['ldap_enabled'] = true
gitlab_rails['ldap_servers'] = YAML.load <<-EOS # remember to close this block with 'EOS' below
main: # 'main' is the GitLab 'provider ID' of this LDAP server
  ## label
  #
  # A human-friendly name for your LDAP server. It is OK to change the label later,
  # for instance if you find out it is too large to fit on the web page.
  #
  # Example: 'Paris' or 'Acme, Ltd.'
  label: 'MyCompany'

  # Example: 'ldap.mydomain.com'
  host: 'REDACTED-AD-SERVER'
  # This port is an example, it is sometimes different but it is always an integer and not a string
  port: 3268 # usually 636 for SSL
  uid: 'sAMAccountName' # This should be the attribute, not the value that maps to uid.

  # Examples: 'america\\momo' or 'CN=Gitlab Git,CN=Users,DC=mydomain,DC=com'
  bind_dn: 'CN=BIND-USER,OU=SOME THING,DC=MyCompany,DC=local'
  password: 'REDACTED'

  # Encryption method. The "method" key is deprecated in favor of
  # "encryption".
  #
  #   Examples: "start_tls" or "simple_tls" or "plain"
  #
  #   Deprecated values: "tls" was replaced with "start_tls" and "ssl" was
  #   replaced with "simple_tls".
  #
  encryption: 'plain'

  # Enables SSL certificate verification if encryption method is
  # "start_tls" or "simple_tls". Defaults to true since GitLab 10.0 for
  # security. This may break installations upon upgrade to 10.0, that did
  # not know their LDAP SSL certificates were not setup properly. For
  # example, when using self-signed certificates, the ca_file path may
  # need to be specified.
  verify_certificates: true

  # Specifies the path to a file containing a PEM-format CA certificate,
  # e.g. if you need to use an internal CA.
  #
  #   Example: '/etc/ca.pem'
  #
  ca_file: ''

  # Specifies the SSL version for OpenSSL to use, if the OpenSSL default
  # is not appropriate.
  #
  #   Example: 'TLSv1_1'
  #
  ssl_version: ''

  # Set a timeout, in seconds, for LDAP queries. This helps avoid blocking
  # a request if the LDAP server becomes unresponsive.
  # A value of 0 means there is no timeout.
 timeout: 10

  # This setting specifies if LDAP server is Active Directory LDAP server.
  # For non AD servers it skips the AD specific queries.
  # If your LDAP server is not AD, set this to false.
  active_directory: true

  # If allow_username_or_email_login is enabled, GitLab will ignore everything
  # after the first '@' in the LDAP username submitted by the user on login.
  #
  # Example:
  # - the user enters 'jane.doe@example.com' and 'p@ssw0rd' as LDAP credentials;
  # - GitLab queries the LDAP server with 'jane.doe' and 'p@ssw0rd'.
  #
  # If you are using "uid: 'userPrincipalName'" on ActiveDirectory you need to
  # disable this setting, because the userPrincipalName contains an '@'.
  allow_username_or_email_login: false

  # To maintain tight control over the number of active users on your GitLab installation,
  # enable this setting to keep new users blocked until they have been cleared by the admin
  # (default: false).
  block_auto_created_users: false

  # Base where we can search for users
  #
  base: 'DC=MyCompany,DC=local'

  # Filter LDAP users
  #
  #   Format: RFC 4515 https://tools.ietf.org/search/rfc4515
  #   Ex. (employeeType=developer)
  #
  #   Note: GitLab does not support omniauth-ldap's custom filter syntax.
  #
  #   Example for getting only specific users:
  #   '(&(objectclass=user)(|(samaccountname=momo)(samaccountname=toto)))'
  #
  user_filter: '(&(objectclass=user)
                  (!(objectClass=computer))
                  (!(userAccountControl:1.2.840.113556.1.4.803:=2))
                  (|(memberOf:1.2.840.113556.1.4.1941:=CN=GROUP1,OU=Someother Groups,DC=MyCompany,DC=local)
                    (memberOf:1.2.840.113556.1.4.1941:=CN=My_Group,OU=Some Groups,DC=MyCompany,DC=local)
                    (memberOf:1.2.840.113556.1.4.1941:=CN=GROUP3,OU=Some Groups,DC=MyCompany,DC=local)))'
  # LDAP attributes that GitLab will use to create an account for the LDAP user.
  # The specified attribute can either be the attribute name as a string (e.g. 'mail'),
  # or an array of attribute names to try in order (e.g. ['mail', 'email']).
  # Note that the user's LDAP login will always be the attribute specified as `uid` above.
  attributes:
    # The username will be used in paths for the user's own projects
    # (like `gitlab.example.com/username/project`) and when mentioning
    # them in issues, merge request and comments (like `@username`).
    # If the attribute specified for `username` contains an email address,
    # the GitLab username will be the part of the email address before the '@'.
    username: ['uid', 'userid', 'sAMAccountName']
    email:    ['mail', 'email', 'userPrincipalName']

    # If no full name could be found at the attribute specified for `name`,
    # the full name is determined using the attributes specified for
    # `first_name` and `last_name`.
    name:       'cn'
    first_name: 'givenName'
    last_name:  'sn'

  ## EE only

  # Base where we can search for groups
  #
  #   Ex. ou=groups,dc=gitlab,dc=example
  #
  group_base: 'OU=Some Groups,DC=MyCompany,DC=local'

  admin_group: 'My_Group'
  # The CN of a group containing GitLab administrators
  #
  #   Ex. administrators
  #
  #   Note: Not `cn=administrators` or the full DN
  #
  # The LDAP attribute containing a user's public SSH key
  #
  #   Ex. ssh_public_key
  #
  sync_ssh_keys: false

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info 
System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.3.5p376
Gem Version:    2.6.13
Bundler Version:1.13.7
Rake Version:   12.1.0
Redis Version:  3.2.5
Git Version:    2.13.6
Sidekiq Version:5.0.4
Go Version:     unknown

GitLab information
Version:        10.1.4-ee
Revision:       a512054
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     postgresql
DB Version:     9.6.5
URL:            http://redacted-server
HTTP Clone URL: http://redacted-server/some-group/some-project.git
SSH Clone URL:  git@redacted-server:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: no


GitLab Shell
Version:        5.9.3
Repository storage paths:
  • default: /var/opt/gitlab/git-data/repositories Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks Git: /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check 
Checking GitLab Shell ...


GitLab Shell version >= 5.9.3 ? ... OK (5.9.3)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:root, or git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
8/1 ... repository is empty
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Redis available via internal API: OK


Access to /var/opt/gitlab/.ssh/authorized_keys: OK
gitlab-shell self-check successful


Checking GitLab Shell ... Finished


Checking Sidekiq ...


Running? ... yes
Number of Sidekiq processes ... 1


Checking Sidekiq ... Finished


Reply by email is disabled in config/gitlab.yml
Checking LDAP ...


Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)


REDACTED LDAP ACCOUNTS!!!!!


Checking LDAP ... Finished


Checking GitLab ...


Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... skipped (no tmp uploads folder yet)
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
8/1 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.3.3 ? ... yes (2.3.5)
Git version >= 2.7.3 ? ... yes (2.13.6)
Git user has default SSH configuration? ... yes
Active users: ... 6
Elasticsearch version 5.1 - 5.5? ... skipped (elasticsearch is disabled)


Checking GitLab ... Finished
Edited by Hoa Nguyen