"ecdsa-sk" and "ed25519-sk" SSH keys do not work on GitLab SaaS
In https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78934 we added the ability to users to add two new types("ecdsa-sk" and "ed25519-sk") of SSH keys to their accounts. [OpenSSH 8.2](https://www.openssh.com/releasenotes.html#8.2) added support for those keys.
## Staging
At Feb 4, 2022, 12:07 PM: This MR was [deployed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78934#note_b87f5fbff9aee1aa3fe835bba22e2a91858d15e8) to https://staging.gitlab.com
It was confirmed that "ecdsa-sk" and "ed25519-sk" SSH keys works on Staging by using `ssh -Tvvv git@staging.gitlab.com`:
<details>
<summary markdown="span">Output</summary>
```shell
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "staging.gitlab.com" port 22
debug2: ssh_connect_direct
debug1: Connecting to staging.gitlab.com [172.65.216.50] port 22.
debug1: Connection established.
debug1: identity file /home/bogdanvlviv/.ssh/id_rsa type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_rsa-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_dsa type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_dsa-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ecdsa type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ed25519 type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ed25519-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ed25519_sk type 12
debug1: identity file /home/bogdanvlviv/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_xmss type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to staging.gitlab.com:22 as 'git'
debug3: hostkeys_foreach: reading file "/home/bogdanvlviv/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/bogdanvlviv/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from staging.gitlab.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group14-sha1
debug2: host key algorithms: ecdsa-sha2-nistp256,ssh-ed25519,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: ciphers stoc: aes128-gcm@openssh.com,chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr
debug2: MACs ctos: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: MACs stoc: hmac-sha2-256-etm@openssh.com,hmac-sha2-256,hmac-sha1,hmac-sha1-96
debug2: compression ctos: none
debug2: compression stoc: none
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256@libssh.org
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw
debug3: hostkeys_foreach: reading file "/home/bogdanvlviv/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/bogdanvlviv/.ssh/known_hosts:7
debug3: load_hostkeys: loaded 1 keys from staging.gitlab.com
debug3: hostkeys_foreach: reading file "/home/bogdanvlviv/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/bogdanvlviv/.ssh/known_hosts:8
debug3: load_hostkeys: loaded 1 keys from 172.65.216.50
debug1: Host 'staging.gitlab.com' is known and matches the ECDSA host key.
debug1: Found key in /home/bogdanvlviv/.ssh/known_hosts:7
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_rsa
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_dsa
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_ecdsa
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_ed25519
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_ed25519_sk ED25519-SK SHA256:fs4/pqM70MIUue2+m5utQnVNB/h/ikV11TKC1voQpUw authenticator
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-rsa,ecdsa-sha2-nistp256,sk-ecdsa-sha2-nistp256@openssh.com,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-256,rsa-sha2-512>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey
debug3: start over, passed a different list publickey
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_rsa
debug3: no such identity: /home/bogdanvlviv/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_dsa
debug3: no such identity: /home/bogdanvlviv/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_ecdsa
debug3: no such identity: /home/bogdanvlviv/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_ecdsa_sk
debug3: no such identity: /home/bogdanvlviv/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_ed25519
debug3: no such identity: /home/bogdanvlviv/.ssh/id_ed25519: No such file or directory
debug1: Offering public key: /home/bogdanvlviv/.ssh/id_ed25519_sk ED25519-SK SHA256:fs4/pqM70MIUue2+m5utQnVNB/h/ikV11TKC1voQpUw authenticator
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 60
debug1: Server accepts key: /home/bogdanvlviv/.ssh/id_ed25519_sk ED25519-SK SHA256:fs4/pqM70MIUue2+m5utQnVNB/h/ikV11TKC1voQpUw authenticator
debug3: sign_and_send_pubkey: ED25519-SK SHA256:fs4/pqM70MIUue2+m5utQnVNB/h/ikV11TKC1voQpUw
debug3: sign_and_send_pubkey: signing using sk-ssh-ed25519@openssh.com SHA256:fs4/pqM70MIUue2+m5utQnVNB/h/ikV11TKC1voQpUw
notify_start: exec(/usr/bin/ssh-askpass): No such file or directory
debug3: start_helper: started pid=85950
debug3: ssh_msg_send: type 5
debug3: ssh_msg_recv entering
debug1: start_helper: starting /usr/lib/openssh/ssh-sk-helper
debug1: ssh-sk-helper: ready to sign with key ED25519-SK, provider internal: msg len 184, compat 0x0
debug1: sshsk_sign: provider "internal", key ED25519-SK, flags 0x21
debug1: find_device: found 1 device(s)
debug1: find_device: trying device 0: /dev/hidraw3
debug1: try_device: fido_dev_get_assert: FIDO_ERR_SUCCESS
debug1: find_device: found key
debug1: ssh-sk-helper: reply len 111
debug3: ssh_msg_send: type 5
debug3: reap_helper: pid=85950
debug3: send packet: type 50
debug3: receive packet: type 52
debug1: Authentication succeeded (publickey).
Authenticated to staging.gitlab.com ([172.65.216.50]:22).
debug2: fd 5 setting O_NONBLOCK
debug3: fd 6 is O_NONBLOCK
debug1: channel 0: new [client-session]
debug3: ssh_session2_open: channel_new: 0
debug2: channel 0: send open
debug3: send packet: type 90
debug1: Entering interactive session.
debug1: pledge: network
debug3: receive packet: type 91
debug2: channel_input_open_confirmation: channel 0: callback start
debug2: fd 3 setting TCP_NODELAY
debug3: ssh_packet_set_tos: set IP_TOS 0x08
debug2: client_session2_setup: id 0
debug1: Sending environment.
debug3: Ignored env SHELL
debug3: Ignored env SESSION_MANAGER
debug3: Ignored env WINDOWID
debug3: Ignored env QT_ACCESSIBILITY
debug3: Ignored env XDG_CONFIG_DIRS
debug3: Ignored env XDG_MENU_PREFIX
debug3: Ignored env TERM_PROGRAM_VERSION
debug3: Ignored env GNOME_DESKTOP_SESSION_ID
debug3: Ignored env TMUX
debug3: Ignored env XTERM_VERSION
debug1: Sending env LC_ADDRESS = en_CA.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env GNOME_SHELL_SESSION_MODE
debug1: Sending env LC_NAME = en_CA.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env SSH_AUTH_SOCK
debug3: Ignored env XMODIFIERS
debug3: Ignored env DESKTOP_SESSION
debug1: Sending env LC_MONETARY = en_CA.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env SSH_AGENT_PID
debug3: Ignored env XTERM_SHELL
debug3: Ignored env EDITOR
debug3: Ignored env GTK_MODULES
debug3: Ignored env PWD
debug3: Ignored env XDG_SESSION_DESKTOP
debug3: Ignored env LOGNAME
debug3: Ignored env XDG_SESSION_TYPE
debug3: Ignored env GPG_AGENT_INFO
debug3: Ignored env XAUTHORITY
debug3: Ignored env FZF_DEFAULT_COMMAND
debug3: Ignored env WINDOWPATH
debug3: Ignored env HOME
debug3: Ignored env USERNAME
debug3: Ignored env IM_CONFIG_PHASE
debug1: Sending env LC_PAPER = en_CA.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug1: Sending env LANG = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env LS_COLORS
debug3: Ignored env XDG_CURRENT_DESKTOP
debug3: Ignored env INVOCATION_ID
debug3: Ignored env XTERM_LOCALE
debug3: Ignored env MANAGERPID
debug3: Ignored env LESSCLOSE
debug3: Ignored env XDG_SESSION_CLASS
debug3: Ignored env TERM
debug1: Sending env LC_IDENTIFICATION = en_CA.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env ASDF_DIR
debug3: Ignored env LESSOPEN
debug3: Ignored env USER
debug3: Ignored env TMUX_PANE
debug3: Ignored env DISPLAY
debug3: Ignored env SHLVL
debug1: Sending env LC_TELEPHONE = en_CA.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env QT_IM_MODULE
debug1: Sending env LC_MEASUREMENT = en_CA.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env PAPERSIZE
debug3: Ignored env XDG_RUNTIME_DIR
debug1: Sending env LC_TIME = en_CA.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug1: Sending env LC_ALL = en_US.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env JOURNAL_STREAM
debug3: Ignored env XDG_DATA_DIRS
debug3: Ignored env PATH
debug3: Ignored env GDMSESSION
debug3: Ignored env DBUS_SESSION_BUS_ADDRESS
debug3: Ignored env RUBYLIB
debug3: Ignored env GIO_LAUNCHED_DESKTOP_FILE_PID
debug1: Sending env LC_NUMERIC = en_CA.UTF-8
debug2: channel 0: request env confirm 0
debug3: send packet: type 98
debug3: Ignored env OLDPWD
debug3: Ignored env TERM_PROGRAM
debug3: Ignored env _
debug2: channel 0: request shell confirm 1
debug3: send packet: type 98
debug2: channel_input_open_confirmation: channel 0: callback done
debug2: channel 0: open confirm rwindow 2097152 rmax 32768
debug3: receive packet: type 99
debug2: channel_input_status_confirm: type 99 id 0
debug2: shell request accepted on channel 0
debug3: receive packet: type 96
debug2: channel 0: rcvd eof
debug2: channel 0: output open -> drain
debug3: receive packet: type 98
debug1: client_input_channel_req: channel 0 rtype exit-status reply 0
debug3: receive packet: type 97
debug2: channel 0: rcvd close
debug2: channel 0: chan_shutdown_read (i0 o1 sock -1 wfd 4 efd 6 [write])
debug2: channel 0: input open -> closed
debug3: channel 0: will not send data after close
Welcome to GitLab, @bdenkovych!
debug3: channel 0: will not send data after close
debug2: channel 0: obuf empty
debug2: channel 0: chan_shutdown_write (i3 o1 sock -1 wfd 5 efd 6 [write])
debug2: channel 0: output drain -> closed
debug2: channel 0: almost dead
debug2: channel 0: gc: notify user
debug2: channel 0: gc: user detached
debug2: channel 0: send close
debug3: send packet: type 97
debug2: channel 0: is dead
debug2: channel 0: garbage collecting
debug1: channel 0: free: client-session, nchannels 1
debug3: channel 0: status: The following connections are open:
#0 client-session (t4 r0 i3/0 o3/0 e[write]/0 fd -1/-1/6 sock -1 cc -1)
debug3: send packet: type 1
debug3: fd 1 is not O_NONBLOCK
debug1: fd 2 clearing O_NONBLOCK
Transferred: sent 2824, received 1420 bytes, in 0.4 seconds
Bytes per second: sent 7177.0, received 3608.8
debug1: Exit status 0
```
</details>
## Production
At Feb 4, 2022 4:33PM: This MR was [deployed](https://gitlab.com/gitlab-org/gitlab/-/merge_requests/78934#note_9b3a5ec78ffedb5205def36c08b2454d2df601ee) to https://gitlab.com
It was confirmed that "ecdsa-sk" and "ed25519-sk" SSH keys do not work on Production by using `ssh -Tvvv git@gitlab.com`:
<details>
<summary markdown="span">Output</summary>
```shell
OpenSSH_8.2p1 Ubuntu-4ubuntu0.4, OpenSSL 1.1.1f 31 Mar 2020
debug1: Reading configuration data /etc/ssh/ssh_config
debug1: /etc/ssh/ssh_config line 19: include /etc/ssh/ssh_config.d/*.conf matched no files
debug1: /etc/ssh/ssh_config line 21: Applying options for *
debug2: resolving "gitlab.com" port 22
debug2: ssh_connect_direct
debug1: Connecting to gitlab.com [172.65.251.78] port 22.
debug1: Connection established.
debug1: identity file /home/bogdanvlviv/.ssh/id_rsa type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_rsa-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_dsa type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_dsa-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ecdsa type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ecdsa-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ecdsa_sk type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ecdsa_sk-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ed25519 type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ed25519-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_ed25519_sk type 12
debug1: identity file /home/bogdanvlviv/.ssh/id_ed25519_sk-cert type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_xmss type -1
debug1: identity file /home/bogdanvlviv/.ssh/id_xmss-cert type -1
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
debug2: fd 3 setting O_NONBLOCK
debug1: Authenticating to gitlab.com:22 as 'git'
debug3: hostkeys_foreach: reading file "/home/bogdanvlviv/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/bogdanvlviv/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.com
debug3: order_hostkeyalgs: prefer hostkeyalgs: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521
debug3: send packet: type 20
debug1: SSH2_MSG_KEXINIT sent
debug3: receive packet: type 20
debug1: SSH2_MSG_KEXINIT received
debug2: local client KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,ext-info-c
debug2: host key algorithms: ecdsa-sha2-nistp256-cert-v01@openssh.com,ecdsa-sha2-nistp384-cert-v01@openssh.com,ecdsa-sha2-nistp521-cert-v01@openssh.com,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521,sk-ecdsa-sha2-nistp256-cert-v01@openssh.com,ssh-ed25519-cert-v01@openssh.com,sk-ssh-ed25519-cert-v01@openssh.com,rsa-sha2-512-cert-v01@openssh.com,rsa-sha2-256-cert-v01@openssh.com,ssh-rsa-cert-v01@openssh.com,sk-ecdsa-sha2-nistp256@openssh.com,ssh-ed25519,sk-ssh-ed25519@openssh.com,rsa-sha2-512,rsa-sha2-256,ssh-rsa
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com,zlib
debug2: compression stoc: none,zlib@openssh.com,zlib
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug2: peer server KEXINIT proposal
debug2: KEX algorithms: curve25519-sha256,curve25519-sha256@libssh.org,ecdh-sha2-nistp256,ecdh-sha2-nistp384,ecdh-sha2-nistp521,diffie-hellman-group-exchange-sha256,diffie-hellman-group16-sha512,diffie-hellman-group18-sha512,diffie-hellman-group14-sha256,diffie-hellman-group14-sha1
debug2: host key algorithms: rsa-sha2-512,rsa-sha2-256,ssh-rsa,ecdsa-sha2-nistp256,ssh-ed25519
debug2: ciphers ctos: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: ciphers stoc: chacha20-poly1305@openssh.com,aes128-ctr,aes192-ctr,aes256-ctr,aes128-gcm@openssh.com,aes256-gcm@openssh.com
debug2: MACs ctos: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: MACs stoc: umac-64-etm@openssh.com,umac-128-etm@openssh.com,hmac-sha2-256-etm@openssh.com,hmac-sha2-512-etm@openssh.com,hmac-sha1-etm@openssh.com,umac-64@openssh.com,umac-128@openssh.com,hmac-sha2-256,hmac-sha2-512,hmac-sha1
debug2: compression ctos: none,zlib@openssh.com
debug2: compression stoc: none,zlib@openssh.com
debug2: languages ctos:
debug2: languages stoc:
debug2: first_kex_follows 0
debug2: reserved 0
debug1: kex: algorithm: curve25519-sha256
debug1: kex: host key algorithm: ecdsa-sha2-nistp256
debug1: kex: server->client cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug1: kex: client->server cipher: chacha20-poly1305@openssh.com MAC: <implicit> compression: none
debug3: send packet: type 30
debug1: expecting SSH2_MSG_KEX_ECDH_REPLY
debug3: receive packet: type 31
debug1: Server host key: ecdsa-sha2-nistp256 SHA256:HbW3g8zUjNSksFbqTiUWPWg2Bq1x8xdGUrliXFzSnUw
debug3: hostkeys_foreach: reading file "/home/bogdanvlviv/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/bogdanvlviv/.ssh/known_hosts:1
debug3: load_hostkeys: loaded 1 keys from gitlab.com
debug3: hostkeys_foreach: reading file "/home/bogdanvlviv/.ssh/known_hosts"
debug3: record_hostkey: found key type ECDSA in file /home/bogdanvlviv/.ssh/known_hosts:2
debug3: load_hostkeys: loaded 1 keys from 172.65.251.78
debug1: Host 'gitlab.com' is known and matches the ECDSA host key.
debug1: Found key in /home/bogdanvlviv/.ssh/known_hosts:1
debug3: send packet: type 21
debug2: set_newkeys: mode 1
debug1: rekey out after 134217728 blocks
debug1: SSH2_MSG_NEWKEYS sent
debug1: expecting SSH2_MSG_NEWKEYS
debug3: receive packet: type 21
debug1: SSH2_MSG_NEWKEYS received
debug2: set_newkeys: mode 0
debug1: rekey in after 134217728 blocks
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_rsa
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_dsa
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_ecdsa
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_ecdsa_sk
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_ed25519
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_ed25519_sk ED25519-SK SHA256:fs4/pqM70MIUue2+m5utQnVNB/h/ikV11TKC1voQpUw authenticator
debug1: Will attempt key: /home/bogdanvlviv/.ssh/id_xmss
debug2: pubkey_prepare: done
debug3: send packet: type 5
debug3: receive packet: type 7
debug1: SSH2_MSG_EXT_INFO received
debug1: kex_input_ext_info: server-sig-algs=<ssh-ed25519,ssh-rsa,rsa-sha2-256,rsa-sha2-512,ssh-dss,ecdsa-sha2-nistp256,ecdsa-sha2-nistp384,ecdsa-sha2-nistp521>
debug3: receive packet: type 6
debug2: service_accept: ssh-userauth
debug1: SSH2_MSG_SERVICE_ACCEPT received
debug3: send packet: type 50
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,keyboard-interactive
debug3: start over, passed a different list publickey,keyboard-interactive
debug3: preferred gssapi-with-mic,publickey,keyboard-interactive,password
debug3: authmethod_lookup publickey
debug3: remaining preferred: keyboard-interactive,password
debug3: authmethod_is_enabled publickey
debug1: Next authentication method: publickey
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_rsa
debug3: no such identity: /home/bogdanvlviv/.ssh/id_rsa: No such file or directory
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_dsa
debug3: no such identity: /home/bogdanvlviv/.ssh/id_dsa: No such file or directory
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_ecdsa
debug3: no such identity: /home/bogdanvlviv/.ssh/id_ecdsa: No such file or directory
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_ecdsa_sk
debug3: no such identity: /home/bogdanvlviv/.ssh/id_ecdsa_sk: No such file or directory
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_ed25519
debug3: no such identity: /home/bogdanvlviv/.ssh/id_ed25519: No such file or directory
debug1: Offering public key: /home/bogdanvlviv/.ssh/id_ed25519_sk ED25519-SK SHA256:fs4/pqM70MIUue2+m5utQnVNB/h/ikV11TKC1voQpUw authenticator
debug3: send packet: type 50
debug2: we sent a publickey packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,keyboard-interactive
debug1: Trying private key: /home/bogdanvlviv/.ssh/id_xmss
debug3: no such identity: /home/bogdanvlviv/.ssh/id_xmss: No such file or directory
debug2: we did not send a packet, disable method
debug3: authmethod_lookup keyboard-interactive
debug3: remaining preferred: password
debug3: authmethod_is_enabled keyboard-interactive
debug1: Next authentication method: keyboard-interactive
debug2: userauth_kbdint
debug3: send packet: type 50
debug2: we sent a keyboard-interactive packet, wait for reply
debug3: receive packet: type 51
debug1: Authentications that can continue: publickey,keyboard-interactive
debug3: userauth_kbdint: disable: no info_req_seen
debug2: we did not send a packet, disable method
debug1: No more authentication methods to try.
git@gitlab.com: Permission denied (publickey,keyboard-interactive).
```
</details>
## What is the root cause
By comparing both outputs I noticed that on Production we have OpenSSH 7.9 installed:
```shell
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version OpenSSH_7.9p1 Debian-10+deb10u2
debug1: match: OpenSSH_7.9p1 Debian-10+deb10u2 pat OpenSSH* compat 0x04000000
```
We need to update OpenSSH to 8.2+ to make "ecdsa-sk" and "ed25519-sk" SSH keys work.
On Staging, we [enabled `gitlab-sshd` instead of `openssh`](ttps://gitlab.com/gitlab-com/gl-infra/production/-/issues/6082) recently:
```shell
debug1: Local version string SSH-2.0-OpenSSH_8.2p1 Ubuntu-4ubuntu0.4
debug1: Remote protocol version 2.0, remote software version Go
debug1: no match: Go
```
As I understand [`gitlab-sshd` uses OpenSSH 8.8 under the hood](https://gitlab.com/gitlab-com/gl-infra/production/-/issues/6243). That is why new types of SSH keys work on https://staging.gitlab.com.
## How to resolve
Updating OpenSSH to 8.2+ or enabling `gitlab-sshd` on Production should resolve this issue. Related to https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/603.
It is likely to be done in %14.8 :tada: : https://gitlab.com/groups/gitlab-com/gl-infra/-/epics/603#note_834496304
I opened this issue
- for visibility
- to understand the issue better
- confirm the way we can resolve it or find another one
issue