Unable to login with 2FA enabled after upgrading to 13.3.4-ee (ad7bd7db)

Summary

Unable to login to any account with 2fa enabled after upgrading to 13.3.4

Internal discussion

Steps to reproduce

Login via LDAP login, when prompted for 2fa code, enter code, or press button on security key.

What is the current bug behavior?

Got message: An error occurred. Please sign in again.

Disabling 2FA via either another logged in session or via the rails console allows login again. Enabling 2FA again prevents future logins. This happens with both TOTP and WebAuth hardware tokens. Recovery codes do not work either.

This is happening on three systems, 2 of which are Debian, one is CentOS. Omnibus installation on all three. All three are configured for LDAP authentication.

What is the expected correct behavior?

Normal login

Relevant logs and/or screenshots

production.log

Started POST "/users/auth/ldapmain/callback" for ***.**.***.*** at 2020-09-02 17:04:47 -0700
Processing by Ldap::OmniauthCallbacksController#ldapmain as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "username"=>"******", "password"=>"[FILTERED]", "remember_me"=>"1"}
Completed 200 OK in 212ms (Views: 49.7ms | ActiveRecord: 50.3ms | Elasticsearch: 0.0ms | Allocations: 89791)
Started POST "/users/sign_in" for ***.**.***.*** at 2020-09-02 17:04:50 -0700
Processing by SessionsController#create as HTML
  Parameters: {"utf8"=>"✓", "authenticity_token"=>"[FILTERED]", "user"=>{"remember_me"=>"1", "otp_attempt"=>"[FILTERED]"}}
Redirected to https://gitlab.********.com/users/sign_in
Filter chain halted as :authenticate_with_two_factor rendered or redirected
Completed 302 Found in 7ms (ActiveRecord: 0.8ms | Elasticsearch: 0.0ms | Allocations: 1768)
Started GET "/users/sign_in" for ***.**.***.*** at 2020-09-02 17:04:51 -0700
Processing by SessionsController#new as HTML
Completed 200 OK in 29ms (Views: 16.3ms | ActiveRecord: 1.6ms | Elasticsearch: 0.0ms | Allocations: 13137)

Output of checks

Results of GitLab environment info

Expand for output related to GitLab environment info

System information
System:		Debian 10
Proxy:		no
Current User:	git
Using RVM:	no
Ruby Version:	2.6.6p146
Gem Version:	2.7.10
Bundler Version:1.17.3
Rake Version:	12.3.3
Redis Version:	5.0.9
Git Version:	2.28.0
Sidekiq Version:5.2.9
Go Version:	go1.11.6 linux/amd64

GitLab information
Version:	13.3.4-ee
Revision:	ad7bd7db412
Directory:	/opt/gitlab/embedded/service/gitlab-rails
DB Adapter:	PostgreSQL
DB Version:	12.3
URL:		https://gitlab.linss.com
HTTP Clone URL:	https://gitlab.linss.com/some-group/some-project.git
SSH Clone URL:	git@gitlab.linss.com:some-group/some-project.git
Elasticsearch:	yes
Geo:		no
Using LDAP:	yes
Using Omniauth:	no

GitLab Shell
Version:	13.6.0
Repository storage paths:
- default: 	/var/opt/gitlab/git-data/repositories
GitLab Shell path:		/opt/gitlab/embedded/service/gitlab-shell
Git:		/opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.6.0 ? ... OK (13.6.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Checking Reply by email ...
IMAP server credentials are correct? ... Checking incoming@gitlab.linss.com
yes
Init.d configured correctly? ... skipped
MailRoom running? ... skipped
Checking Reply by email ... Finished
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapmain
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 2 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
5/3 ... yes
5/4 ... yes
5/5 ... yes
5/6 ... yes
4/7 ... yes
4/10 ... yes
4/11 ... yes
4/12 ... yes
7/14 ... yes
7/15 ... yes
11/18 ... yes
4/19 ... yes
5/20 ... yes
4/21 ... yes
4/22 ... yes
4/24 ... yes
4/25 ... yes
10/26 ... yes
4/27 ... yes
4/29 ... yes
5/30 ... yes
5/31 ... yes
4/32 ... yes
5/34 ... yes
4/35 ... yes
5/36 ... yes
4/37 ... yes
4/38 ... yes
4/39 ... yes
5/40 ... yes
5/41 ... yes
5/42 ... yes
4/43 ... yes
4/44 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 1
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... no
Try fixing it:
Please migrate all projects to hashed storage
as legacy storage is deprecated in 13.0 and support will be removed in 14.0.
For more information see:
doc/administration/repository_storage_types.md
Elasticsearch version 6.x - 7.x? ... yes (7.9.0)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished