After restoring backup, new SSH public keys randomly supersede other users' existing keys
Versions
RHEL 6.5
GitLab 8.6.4
GitLab Shell 2.6.12
GitLab API v3
Git 2.7.4
Ruby 2.1.8p440
Rails 4.2.5.2
PostgreSQL 9.3.6
I have never upgraded GitLab; this is the first version that was installed.
Events leading to the problem
I installed GitLab and my team evaluated it. I happened to enter my SSH public key first, so in ~git/.ssh/authorized_keys
, my public key got the name key-1
.
As part of our evaluation I made a backup and restored it.
After I restored the backup, new user Shung Wang was the first to enter his SSH public key. In ~git/.ssh/authorized_keys
, his key got the duplicate name key-1
. This is order dependent, so you never know whose keys will get the same name.
The interesting part of ~git/.ssh/authorized_keys
is:
# Managed by gitlab-shell
command="/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-1",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAA...2SkSQ== jm...@wavesemi.com
########################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################################
command="/opt/gitlab/embedded/service/gitlab-shell/bin/gitlab-shell key-1",no-port-forwarding,no-X11-forwarding,no-agent-forwarding,no-pty ssh-rsa AAA...nKQ== wang....@wavesemi.com
...
Now, whenever I try to access the git repositories via SSH, the server thinks I am Shung Wang. For example, when I tested my SSH connection from my Ubuntu 14.04 laptop, I got this:
ssh -T git@gitserver
Welcome to GitLab, Shung Wang!
As a second test, I tried to clone a private repository to which Shung did not have access:
git clone git@gitserver:sw/devops.git
Cloning into 'devops'...
GitLab: The project you were looking for could not be found.
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Then I made Shung a member of the devops project, and git clone
succeeded.
Obviously this is a most unsatisfactory security situation. I think it is a pretty big deal.
Bug fix
It appears that GitLab maintains a counter for the SSH key names, and this counter can get out of sync with file ~git/.ssh/authorized_keys
, particularly after a backup is restored. I do not have the expertise to offer a merge request, but I see two approaches:
- When adding keys, search
~git/.ssh/authorized_keys
for the highest existing key number, add one and use that as the new key name. But perhaps it is time consuming on large installations to find this number each time a user adds a public key. - Create the key name using the email address in the SSH public key
It would be very nice to add to gitlab-rake gitlab:check
a test for unique key names in ~git/.ssh/authorized_keys
.
Workaround
Until the bug is fixed, users can try this workaround:
- After you restore a backup, log in as some existing user and register a new SSH public key
- Search file
~git/.ssh/authorized_keys
forkey-1
. If you find two, you have the problem described above. - Those lines of
######...
are not decoration. They are keys that were deleted by users. When a key is deleted, GitLab replaces every character in it with a#
, presumably so the remaining keys do not move around in the file. Replace all characters in all SSH keys created before the backup restore with the#
character in a fashion similar to what you see in the~git/.ssh/authorized_keys
shown above. - Tell all your users they must re-enter their SSH public keys. If they complain, remind them to be thankful that the rest of the backup worked.
Rather than all that tedious editing in step 3, I suspect that you can simply move file ~git/.ssh/authorized_keys
aside and replace it with an empty file, and then tell everyone to re-enter their SSH public keys. However I did not try this myself.