Some (but not all) Users Cannot Set Up 2FA: Invalid Pin

Summary

After turning on 2FA enforcement, a small number of users cannot successfully set up 2FA, recieving an Invalid PIN error. It appears this can be worked round with this, but it is not clear what causes the issue in the first place.

• The issue affects only a few users on the server.
• These users have tried multiple 2FA apps with the same result.
• The time on the user's phones and the server is within a second or so.
• The problem appears to be device specific.
• Using time correction (e.g. in Google Authenticator) does not resolve the problem.
  • Devices whose clock is several seconds (more that the affected users' devices) out from the server generate accepted codes.
• The problem spontanously resolved for some, but not all users.
• We did recently migrate to a new server and external URL, and enable LDAP auth, however, most users have no issues.
  • The users who have the issue include both those with accounts prior to the move, and those with accounts created from LDAP.
• An NTP service is active and succesfully syncing the server time.

Steps to reproduce

The issue is specific to certain user devices, (and accounts?) making it very hard to reporduce.

What is the current bug behavior?

1. A user signs in for the first time since 2FA enforcement activated, and is redirected to the 2FA set up page.
2. User scans the QR code, or manually enters the details.
3. User enters a code from the new 2FA entry in their app and recieves an invalid pin error.

A possibly relevant note: The icon at the top left corner of the interface, which links to the dashboard, appears as a broken image link on the 2FA set up page for affected users. For other users it seems to appear normally.

What is the expected correct behavior?

After entering a valid code, the user should be forwarded to the backup codes page.

Relevant logs and/or screenshots

I would apprecieate some guidance on which logs might contain relevant information, it's not clear which component on gitlab handles 2FA.

Output of checks

Results of GitLab environment info

Running in docker using the official image and docker compose, behind a reverse proxy.

Expand for output related to GitLab environment info

System information
System:
Proxy:          no
Current User:   git
Using RVM:      no
Ruby Version:   2.6.6p146
Gem Version:    2.7.10
Bundler Version:1.17.3
Rake Version:   12.3.3
Redis Version:  5.0.9
Git Version:    2.28.0
Sidekiq Version:5.2.9
Go Version:     unknown

GitLab information
Version:        13.4.2-ee
Revision:       34869f45ee8
Directory:      /opt/gitlab/embedded/service/gitlab-rails
DB Adapter:     PostgreSQL
DB Version:     11.9
URL:            https://
HTTP Clone URL: https:///some-group/some-project.git
SSH Clone URL:  git@:some-group/some-project.git
Elasticsearch:  no
Geo:            no
Using LDAP:     yes
Using Omniauth: yes
Omniauth Providers: gitlab, github, bitbucket

GitLab Shell
Version:        13.7.0
Repository storage paths:
- default:      /var/opt/gitlab/git-data/repositories
GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
Git:            /opt/gitlab/embedded/bin/git

Results of GitLab application Check

Expand for output related to the GitLab application check

Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 13.7.0 ? ... OK (13.7.0)
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Internal API available: OK
Redis available via internal API: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... Server: ldapldapmain
not verifying SSL hostname of LDAPS server ''
LDAP authentication... Success
LDAP users with access to your GitLab server (only showing the first 100 results)
User output sanitized. Found 69 users of 100 limit.
Checking LDAP ... Finished
Checking GitLab App ...
Git configured correctly? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config up to date? ... yes
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory exists? ... yes
Uploads directory has correct permissions? ... yes
Uploads directory tmp has correct permissions? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
Projects have namespace: ...
4/1 ... yes
2/2 ... yes
2/3 ... yes
5/8 ... yes
15/66 ... yes
15/67 ... yes
15/68 ... yes
15/69 ... yes
15/70 ... yes
15/71 ... yes
15/72 ... yes
15/73 ... yes
15/74 ... yes
15/75 ... yes
15/76 ... yes
15/77 ... yes
15/78 ... yes
15/79 ... yes
15/80 ... yes
15/81 ... yes
15/82 ... yes
15/83 ... yes
15/84 ... yes
15/85 ... yes
15/87 ... yes
15/88 ... yes
15/89 ... yes
15/90 ... yes
15/91 ... yes
16/114 ... yes
16/121 ... yes
16/122 ... yes
16/123 ... yes
16/124 ... yes
16/125 ... yes
16/126 ... yes
16/127 ... yes
16/128 ... yes
16/129 ... yes
16/130 ... yes
16/131 ... yes
16/132 ... yes
16/133 ... yes
16/134 ... yes
16/135 ... yes
16/136 ... yes
16/137 ... yes
16/138 ... yes
16/139 ... yes
16/140 ... yes
16/141 ... yes
16/142 ... yes
16/143 ... yes
16/144 ... yes
16/145 ... yes
16/146 ... yes
16/147 ... yes
16/148 ... yes
5/149 ... yes
35/168 ... yes
16/169 ... yes
2/170 ... yes
15/171 ... yes
16/172 ... yes
16/174 ... yes
16/175 ... yes
16/176 ... yes
16/177 ... yes
16/178 ... yes
16/179 ... yes
16/180 ... yes
15/181 ... yes
2/182 ... yes
2/183 ... yes
1/184 ... yes
16/185 ... yes
16/186 ... yes
16/187 ... yes
16/188 ... yes
7/189 ... yes
19/190 ... yes
2/191 ... yes
19/192 ... yes
19/193 ... yes
19/194 ... yes
8/196 ... yes
8/197 ... yes
25/198 ... yes
19/199 ... yes
19/203 ... yes
8/204 ... yes
26/205 ... yes
15/209 ... yes
15/210 ... yes
16/211 ... yes
16/212 ... yes
19/213 ... yes
15/214 ... yes
20/215 ... yes
16/216 ... yes
2/217 ... yes
16/218 ... yes
16/219 ... yes
32/220 ... yes
4/222 ... yes
35/223 ... yes
32/225 ... yes
2/226 ... yes
25/227 ... yes
32/228 ... yes
39/229 ... yes
15/231 ... yes
16/232 ... yes
15/233 ... yes
16/234 ... yes
25/235 ... yes
2/236 ... yes
15/237 ... yes
15/238 ... yes
15/239 ... yes
15/240 ... yes
16/241 ... yes
16/242 ... yes
16/243 ... yes
16/244 ... yes
62/245 ... yes
16/247 ... yes
15/248 ... yes
39/249 ... yes
69/250 ... yes
Redis version >= 4.0.0? ... yes
Ruby version >= 2.5.3 ? ... yes (2.6.6)
Git version >= 2.24.0 ? ... yes (2.28.0)
Git user has default SSH configuration? ... yes
Active users: ... 41
Is authorized keys file accessible? ... yes
GitLab configured to store new projects in hashed storage? ... yes
All projects are in hashed storage? ... yes
Elasticsearch version 6.x - 7.x? ... skipped (elasticsearch is disabled)
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished

Possible Workaround

The workaround here: https://forum.gitlab.com/t/cant-add-2fa/29087/12 seems to work, so far. Disabling 2FA from the admin interface, restarting the application and reenabling did not resolve the issue. Update: This apparently only resolved the issue for one user, so might not have been the thing that resolved it at all.