ssh keys disappear from unblocked user
Summary
We have a user who reports their ssh keys disappear from their account. This has been repeatable, even after re-adding the keys. While they show in the user profile page they are functional: the user can use them to access repositories. A minute or two later the keys disappear from the profile page and no longer work for repository access.
Steps to reproduce
Unfortunately I don't have steps to reproduce on a clean instance. The behaviour has only been reported with a specific user account and doesn't occur with new test accounts we've made.
Possibly relevant: We are using the omnibus debian community edition packages for our instance (https://gitlab.xiph.org/). The user was blocked (incorrectly) by an admin while we were dealing with the spam influx over the past couple of weeks. After unblocking them, their keys and project affiliations were gone. Admin re-granted project roles, and the user re-uploaded keys. Roles stayed but keys seem to be removed again after a few minutes. They may have been blocked during a database migration. We recently upgraded:
- 15.11.5 -> 16.0.4 2023 June 11
- 16.0.4 -> 16.0.5 2023 June 16
- 16.0.5 -> 16.1.0 2023 June 24
- 16.1.0 -> 16.1.1 2023 July 1
So one theory is the database migration script missed something, and some cleanup job is deleting their keys?
Other symptoms are the "Abuse Reports" admin page showing a stack of (removed user)
entries with no enabled actions, and complaints in the backup script log similar to #371828
Happy to have pointers for additional avenues of investigation on the running instance.
What is the current bug behavior?
- User adds ssh key to acount
- New key is visible on user account page
- A few minutes later, the account admin page says "There are no SSH keys associated with this account."
What is the expected correct behavior?
SSH keys should stay active and accessible until an action is taken to remove them.
Relevant logs and/or screenshots
Output of checks
Results of GitLab environment info
Expand for output related to GitLab environment info
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:env:info`) System information System: Debian 10 Current User: git Using RVM: no Ruby Version: 3.0.6p216 Gem Version: 3.4.13 Bundler Version:2.4.14 Rake Version: 13.0.6 Redis Version: 6.2.11 Sidekiq Version:6.5.7 Go Version: unknown GitLab information Version: 16.1.1 Revision: 9ce736bb2cd Directory: /opt/gitlab/embedded/service/gitlab-rails DB Adapter: PostgreSQL DB Version: 13.11 URL: https://gitlab.xiph.org HTTP Clone URL: https://gitlab.xiph.org/some-group/some-project.git SSH Clone URL: git@gitlab.xiph.org:some-group/some-project.git Using LDAP: no Using Omniauth: yes Omniauth Providers: github GitLab Shell Version: 14.23.0 Repository storages: - default: unix:/var/opt/gitlab/gitaly/gitaly.socket GitLab Shell path: /opt/gitlab/embedded/service/gitlab-shell
Results of GitLab application Check
Expand for output related to the GitLab application check
(For installations with omnibus-gitlab package run and paste the output of: `sudo gitlab-rake gitlab:check SANITIZE=true`)Checking GitLab subtasks ...
Checking GitLab Shell ...
GitLab Shell: ... GitLab Shell version >= 14.23.0 ? ... OK (14.23.0) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Gitaly ...
Gitaly: ... default ... OK
Checking Gitaly ... Finished
Checking Sidekiq ...
Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1
Checking Sidekiq ... Finished
Checking Incoming Email ...
Incoming Email: ... Reply by email is disabled in config/gitlab.yml
Checking Incoming Email ... Finished
Checking LDAP ...
LDAP: ... LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab App ...
Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Cable config exists? ... yes Resque config exists? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Systemd unit files or init script exist? ... skipped (omnibus-gitlab has neither init script nor systemd units) Systemd unit files or init script up-to-date? ... skipped (omnibus-gitlab has neither init script nor systemd units) Projects have namespace: ... 2/2 ... yes 2/3 ... yes 2/5 ... yes 3/6 ... yes 2/7 ... yes 2/9 ... yes 2/10 ... yes 2/11 ... yes 2/12 ... yes 2/13 ... yes 2/14 ... yes 2/15 ... yes 2/16 ... yes 2/18 ... yes 2/19 ... yes 2/20 ... yes 2/21 ... yes 2/22 ... yes 2/23 ... yes 2/24 ... yes 2/25 ... yes 2/26 ... yes 2/27 ... yes 2/28 ... yes 2/29 ... yes 2/30 ... yes 2/31 ... yes 2/32 ... yes 2/33 ... yes 2/34 ... yes 2/35 ... yes 2/36 ... yes 2/37 ... yes 2/38 ... yes 2/39 ... yes 2/40 ... yes 2/41 ... yes 2/42 ... yes 2/43 ... yes 2/44 ... yes 2/45 ... yes 2/46 ... yes 1/47 ... yes 2/48 ... yes 2/49 ... yes 2/50 ... yes 2/51 ... yes 357/52 ... yes 357/53 ... yes 1015/54 ... yes 212/55 ... yes 212/56 ... yes 212/57 ... yes 1015/58 ... yes 1/59 ... yes 1/60 ... yes 1/61 ... yes 1/62 ... yes 1/63 ... yes 1015/64 ... yes 1015/66 ... yes 2/67 ... yes 1015/68 ... yes 2/69 ... yes 1320/70 ... yes 1334/71 ... yes 2/72 ... yes 1511/74 ... yes 2/76 ... yes 1927/77 ... yes 1927/78 ... yes 1927/79 ... yes 1927/80 ... yes 1927/81 ... yes 908/82 ... yes 2062/83 ... yes 2062/84 ... yes 2062/85 ... yes 2073/86 ... yes 2073/87 ... yes 2073/88 ... yes 2062/89 ... yes 1017/91 ... yes 2/92 ... yes 1/93 ... yes 2237/96 ... yes 2246/97 ... yes 2256/98 ... yes 1012/99 ... yes 2359/100 ... yes 1012/101 ... yes 2/102 ... yes 2/104 ... yes 246/105 ... yes 246/107 ... yes 246/108 ... yes 246/109 ... yes 212/111 ... yes 246/112 ... yes 2474/113 ... yes 1012/114 ... yes 2498/116 ... yes 2502/117 ... yes 2/118 ... yes 2/119 ... yes 2/120 ... yes 2172/121 ... yes 2545/123 ... yes 212/124 ... yes 2557/125 ... yes 2557/126 ... yes 357/127 ... yes 2592/128 ... yes 2001/130 ... yes 2654/131 ... yes 2189/132 ... yes 2189/133 ... yes 2/134 ... yes 634/135 ... yes 634/136 ... yes 634/137 ... yes 2703/138 ... yes 2702/139 ... yes 357/140 ... yes 357/141 ... yes 357/142 ... yes 2/143 ... yes 2716/144 ... yes 714/145 ... yes 2716/146 ... yes 2/148 ... yes 1/149 ... yes 212/151 ... yes 2932/152 ... yes 2/153 ... yes 2831/154 ... yes 2951/155 ... yes 2951/156 ... yes 2951/157 ... yes 2831/158 ... yes 2831/159 ... yes 2831/160 ... yes 2964/161 ... yes 2966/162 ... yes 2964/163 ... yes 357/164 ... yes 212/165 ... yes 2189/166 ... yes 3006/168 ... yes 357/169 ... yes 357/170 ... yes 357/171 ... yes 357/172 ... yes 2/175 ... yes 3105/176 ... yes 3105/177 ... yes 3105/178 ... yes 1015/180 ... yes 1/182 ... yes 634/183 ... yes 3006/184 ... yes 3372/185 ... yes 2/186 ... yes 3620/188 ... yes 634/189 ... yes 2244/190 ... yes 3194/191 ... yes 3681/192 ... yes 3696/193 ... yes 3696/194 ... yes 3696/195 ... yes 3777/198 ... yes 3777/199 ... yes 3777/200 ... yes 3777/201 ... yes 3777/202 ... yes 3777/203 ... yes 212/204 ... yes 2271/205 ... yes 1015/206 ... yes 714/207 ... yes 1012/210 ... yes 714/211 ... yes 1012/212 ... yes 3777/213 ... yes 4169/214 ... yes 4169/215 ... yes 203/216 ... yes 4436/217 ... yes Redis version >= 6.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (3.0.6) Git user has default SSH configuration? ... yes Active users: ... 1841 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes
Checking GitLab App ... Finished
Checking GitLab subtasks ... Finished
(we will only investigate if the tests are passing)