SSH Clone Permission denied (publickey) - All Users & All Repositories
Summary
Cloning with SSH no longer works on my GitLab instance. It was working fine until a few days ago. The issue doesn't seem to be with my SSH key as I have tried creating a new one as well as trying it with a different user. I always get the same Permission denied (publickey)
response on every repository. It could be completely unrelated, but it stopped working around the same time that I upgraded to GitLab 8.14.4. HTTPS cloning still works fine.
Steps to reproduce
git clone git@gitlab.filiosoft.com:filiosoft-osp/polr-mobile.git
Expected behavior
Cloning with SSH should clone the repository.
Actual behavior
When I attempt to clone with SSH, it fails.
git clone git@gitlab.filiosoft.com:filiosoft-osp/polr-mobile.git
(in PowerShell) results in this:
PS D:\Dev> git clone git@gitlab.filiosoft.com:filiosoft-osp/polr-mobile.git
Cloning into 'polr-mobile'...
Permission denied (publickey).
fatal: Could not read from remote repository.
Please make sure you have the correct access rights
and the repository exists.
Relevant logs and/or screenshots
ssh -v git@gitlab.filiosoft.com
(in Git Bash) results in this:
$ ssh -v git@gitlab.filiosoft.com
OpenSSH_7.3p1, OpenSSL 1.0.2j 26 Sep 2016
debug1: Reading configuration data /c/Users/noah/.ssh/config
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: Connecting to gitlab.filiosoft.com [40.76.44.230] port 22.
debug1: Connection established.
debug1: identity file /c/Users/noah/.ssh/id_rsa type 1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/noah/.ssh/id_rsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/noah/.ssh/id_dsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/noah/.ssh/id_dsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/noah/.ssh/id_ecdsa type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/noah/.ssh/id_ecdsa-cert type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/noah/.ssh/id_ed25519 type -1
debug1: key_load_public: No such file or directory
debug1: identity file /c/Users/noah/.ssh/id_ed25519-cert type -1
debug1: Enabling compatibility mode for protocol 2.0
debug1: Local version string SSH-2.0-OpenSSH_7.3
debug1: Remote protocol version 2.0, remote software version OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8
debug1: match: OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.8 pat OpenSSH_6.6.1* compat 0x04000000
debug1: Authenticating to gitlab.filiosoft.com:22 as 'git'
debug1: SSH2_MSG_KEXINIT sent
debug1: SSH2_MSG_KEXINIT received
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:Wfc9+xGczpt1nuyH9ow8U71dlMAU90AUnodJwpxSyOw
debug1: Host 'gitlab.filiosoft.com' is known and matches the ECDSA host key.
debug1: Found key in /c/Users/noah/.ssh/known_hosts:3
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug1: rekey after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS received
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug1: Authentications that can continue: publickey
debug1: Next authentication method: publickey
debug1: Offering RSA public key: /c/Users/noah/.ssh/id_rsa
debug1: Authentications that can continue: publickey
debug1: Trying private key: /c/Users/noah/.ssh/id_dsa
debug1: Trying private key: /c/Users/noah/.ssh/id_ecdsa
debug1: Trying private key: /c/Users/noah/.ssh/id_ed25519
debug1: No more authentication methods to try.
Permission denied (publickey).
Output of checks
Results of GitLab application Check
noah@GitLab:~$ sudo gitlab-rake gitlab:check SANITIZE=true
I, [2016-12-23T02:00:28.264069 #7613] INFO -- sentry: ** [Raven] Raven 2.0.2 ready to catch errors
Checking GitLab Shell ...
GitLab Shell version >= 4.1.1 ? ... OK (4.1.1)
Repo base directory exists?
default... yes
Repo storage directories are symlinks?
default... no
Repo paths owned by git:git?
default... yes
Repo paths access is drwxrws---?
default... yes
hooks directories in repos are links: ...
6/4 ... ok
6/5 ... ok
6/8 ... ok
6/9 ... ok
6/10 ... ok
6/11 ... ok
6/26 ... ok
6/27 ... ok
13/29 ... ok
6/32 ... ok
6/34 ... ok
2/36 ... ok
6/37 ... ok
6/38 ... ok
6/39 ... ok
6/41 ... ok
13/49 ... ok
13/50 ... ok
6/51 ... ok
14/52 ... ok
14/53 ... ok
6/54 ... ok
14/62 ... ok
6/63 ... ok
6/64 ... ok
2/65 ... ok
14/66 ... ok
14/67 ... ok
6/68 ... ok
Running /opt/gitlab/embedded/service/gitlab-shell/bin/check
Check GitLab API access: OK
Access to /var/opt/gitlab/.ssh/authorized_keys: OK
Send ping to redis server: OK
gitlab-shell self-check successful
Checking GitLab Shell ... Finished
Checking Sidekiq ...
Running? ... yes
Number of Sidekiq processes ... 1
Checking Sidekiq ... Finished
Checking Reply by email ...
Reply by email is disabled in config/gitlab.yml
Checking Reply by email ... Finished
Checking LDAP ...
LDAP is disabled in config/gitlab.yml
Checking LDAP ... Finished
Checking GitLab ...
Git configured with autocrlf=input? ... yes
Database config exists? ... yes
All migrations up? ... yes
Database contains orphaned GroupMembers? ... no
GitLab config exists? ... yes
GitLab config outdated? ... no
Log directory writable? ... yes
Tmp directory writable? ... yes
Uploads directory setup correctly? ... yes
Init script exists? ... skipped (omnibus-gitlab has no init script)
Init script up-to-date? ... skipped (omnibus-gitlab has no init script)
projects have namespace: ...
6/4 ... yes
6/5 ... yes
6/8 ... yes
6/9 ... yes
6/10 ... yes
6/11 ... yes
6/26 ... yes
6/27 ... yes
13/29 ... yes
6/32 ... yes
6/34 ... yes
2/36 ... yes
6/37 ... yes
6/38 ... yes
6/39 ... yes
6/41 ... yes
13/49 ... yes
13/50 ... yes
6/51 ... yes
14/52 ... yes
14/53 ... yes
6/54 ... yes
14/62 ... yes
6/63 ... yes
6/64 ... yes
2/65 ... yes
14/66 ... yes
14/67 ... yes
6/68 ... yes
Redis version >= 2.8.0? ... yes
Ruby version >= 2.1.0 ? ... yes (2.3.3)
Your git bin path is "/opt/gitlab/embedded/bin/git"
Git version >= 2.7.3 ? ... yes (2.8.4)
Active users: 3
Checking GitLab ... Finished
Results of GitLab environment info
noah@GitLab:~$ sudo gitlab-rake gitlab:env:info
I, [2016-12-23T02:03:24.398551 #8233] INFO -- sentry: ** [Raven] Raven 2.0.2 ready to catch errors
System information
System: Ubuntu 14.04
Current User: git
Using RVM: no
Ruby Version: 2.3.3p222
Gem Version: 2.6.6
Bundler Version:1.13.6
Rake Version: 10.5.0
Sidekiq Version:4.2.7
GitLab information
Version: 8.15.0
Revision: 54850bf
Directory: /opt/gitlab/embedded/service/gitlab-rails
DB Adapter: postgresql
URL: https://gitlab.filiosoft.com
HTTP Clone URL: https://gitlab.filiosoft.com/some-group/some-project.git
SSH Clone URL: git@gitlab.filiosoft.com:some-group/some-project.git
Using LDAP: no
Using Omniauth: yes
Omniauth Providers: crowd
GitLab Shell
Version: 4.1.1
Repository storage paths:
- default: /var/opt/gitlab/git-data/repositories
Hooks: /opt/gitlab/embedded/service/gitlab-shell/hooks/
Git: /opt/gitlab/embedded/bin/git