Skip to content

Can't authenticate, or identity Anonymous for new SSH keys

Summary

Can't authenticate, or identity Anonymous for new SSH keys after upgrading to 14.4.1

Steps to reproduce

  1. Upgraded GitLab CE Omnibus installation to 14.4.1
  2. Add new pubkey to user account
  3. Example Project

    What is the current bug behavior?

    git pull
    resulted in
    remote: 
    remote: ========================================================================
    remote: 
    remote: The project you were looking for could not be found or you don't have permission to view it.
    remote: 
    remote: ========================================================================
    remote: 
    fatal: Could not read from remote repository.
    
    Please make sure you have the correct access rights
    and the repository exists.
    

    What is the expected correct behavior?

    Server should be able to recognize my user with the corresponding SSH key.

    Relevant logs and/or screenshots

    First I verified that a server that had RSA SSH keys added before the upgrade to 14.4.1 still works.

    I then tested

    ssh -Tv git@my.server.org

    Working server shows:

    debug1: Offering public key: xxxxx RSA SHA256:xxxxx
    debug1: Server accepts key: xxxxx RSA SHA256:xxxxx
    debug1: Authentication succeeded (publickey).
    Authenticated to my.server.org ([xx.xx.xx.xx]:22).
    debug1: channel 0: new [client-session]
    debug1: Requesting no-more-sessions@openssh.com
    debug1: Entering interactive session.
    debug1: pledge: network
    debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
    debug1: Remote: /var/opt/gitlab/.ssh/authorized_keys:19: key options: command user-rc
    debug1: Remote: /var/opt/gitlab/.ssh/authorized_keys:19: key options: command user-rc
    debug1: Sending environment.
    debug1: Sending env LANG = en_US.UTF-8
    Welcome to GitLab, @user!
    

    On the new server the RSA keys are accepted, but it shows "Welcome to GitLab, Anonymous!"

    debug1: Offering public key: xxxxx RSA SHA256:xxxxx
    debug1: Server accepts key: xxxxx RSA SHA256:xxxxx
    debug1: Authentication succeeded (publickey).
    Authenticated to my.server.org ([xx.xx.xx.xx]:22).
    debug1: channel 0: new [client-session]
    debug1: Requesting no-more-sessions@openssh.com
    debug1: Entering interactive session.
    debug1: pledge: network
    debug1: client_input_global_request: rtype hostkeys-00@openssh.com want_reply 0
    debug1: Remote: /var/opt/gitlab/.ssh/authorized_keys:16: key options: command user-rc
    debug1: Remote: /var/opt/gitlab/.ssh/authorized_keys:16: key options: command user-rc
    debug1: Sending environment.
    debug1: Sending env LANG = en_US.UTF-8
    Welcome to GitLab, Anonymous!
    

    I then generated new ed25519 keys and added them to my account. I moved the RSA keys so they would not be tried. The ed25519 keys failed outright and goes to password authentication.

    debug1: Host 'my.server.org' is known and matches the ECDSA host key.
    debug1: Found key in /xxxx/.ssh/known_hosts:1
    debug1: rekey out after 134217728 blocks
    debug1: SSH2_MSG_NEWKEYS sent
    debug1: expecting SSH2_MSG_NEWKEYS
    debug1: SSH2_MSG_NEWKEYS received
    debug1: rekey in after 134217728 blocks
    debug1: Will attempt key: /xxxx/.ssh/id_rsa 
    debug1: Will attempt key: /xxxx/.ssh/id_dsa 
    debug1: Will attempt key: /xxxx/.ssh/id_ecdsa 
    debug1: Will attempt key: /xxxx/.ssh/id_ecdsa_sk 
    debug1: Will attempt key: /xxxx/.ssh/id_ed25519 ED25519 SHA256:xxxx
    debug1: Will attempt key: /xxxx/.ssh/id_ed25519_sk 
    debug1: Will attempt key: /xxxx/.ssh/id_xmss 
    debug1: SSH2_MSG_EXT_INFO received
    debug1: kex_input_ext_info: server-sig-algs=
    debug1: SSH2_MSG_SERVICE_ACCEPT received
    debug1: Authentications that can continue: publickey,password
    debug1: Next authentication method: publickey
    debug1: Trying private key: /xxxx/.ssh/id_rsa
    debug1: Trying private key: /xxxx/.ssh/id_dsa
    debug1: Trying private key: /xxxx/.ssh/id_ecdsa
    debug1: Trying private key: /xxxx/.ssh/id_ecdsa_sk
    debug1: Offering public key: /xxxx/id_ed25519 ED25519 SHA256:xxxx
    debug1: Authentications that can continue: publickey,password
    debug1: Trying private key: /xxxx/.ssh/id_ed25519_sk
    debug1: Trying private key: /xxxx/.ssh/id_xmss
    debug1: Next authentication method: password
    git@my.server.org's password: 
    

    I disabled "Use authorized_keys file to authenticate SSH keys" and it still failed.

    Output of checks

    Results of GitLab environment info

    Expand for output related to GitLab environment info
    $ sudo gitlab-rake gitlab:env:info
    [sudo] password for systems: 
    
    System information
    System:         Ubuntu 20.04
    Current User:   git
    Using RVM:      no
    Ruby Version:   2.7.4p191
    Gem Version:    3.1.4
    Bundler Version:2.1.4
    Rake Version:   13.0.6
    Redis Version:  6.0.16
    Git Version:    2.33.0.
    Sidekiq Version:6.2.2
    Go Version:     unknown
    
    GitLab information
    Version:        14.4.1
    Revision:       1a23d731c9f
    Directory:      /opt/gitlab/embedded/service/gitlab-rails
    DB Adapter:     PostgreSQL
    DB Version:     12.7
    URL:            https://gitlab.griffithmedia.org
    HTTP Clone URL: https://gitlab.griffithmedia.org/some-group/some-project.git
    SSH Clone URL:  git@gitlab.griffithmedia.org:some-group/some-project.git
    Using LDAP:     no
    Using Omniauth: yes
    Omniauth Providers: 
    
    GitLab Shell
    Version:        13.21.1
    Repository storage paths:
    - default:      /var/opt/gitlab/git-data/repositories
    GitLab Shell path:              /opt/gitlab/embedded/service/gitlab-shell
    Git:            /opt/gitlab/embedded/bin/git
    

    Results of GitLab application Check

    Expand for output related to the GitLab application check
    $ sudo gitlab-rake gitlab:check SANITIZE=true
    Checking GitLab subtasks ...
    

    Checking GitLab Shell ...

    GitLab Shell: ... GitLab Shell version >= 13.21.1 ? ... OK (13.21.1) Running /opt/gitlab/embedded/service/gitlab-shell/bin/check Internal API available: OK Redis available via internal API: OK gitlab-shell self-check successful

    Checking GitLab Shell ... Finished

    Checking Gitaly ...

    Gitaly: ... default ... OK

    Checking Gitaly ... Finished

    Checking Sidekiq ...

    Sidekiq: ... Running? ... yes Number of Sidekiq processes (cluster/worker) ... 1/1

    Checking Sidekiq ... Finished

    Checking Incoming Email ...

    Incoming Email: ... Reply by email is disabled in config/gitlab.yml

    Checking Incoming Email ... Finished

    Checking LDAP ...

    LDAP: ... LDAP is disabled in config/gitlab.yml

    Checking LDAP ... Finished

    Checking GitLab App ...

    Git configured correctly? ... yes Database config exists? ... yes All migrations up? ... yes Database contains orphaned GroupMembers? ... no GitLab config exists? ... yes GitLab config up to date? ... yes Log directory writable? ... yes Tmp directory writable? ... yes Uploads directory exists? ... yes Uploads directory has correct permissions? ... yes Uploads directory tmp has correct permissions? ... yes Init script exists? ... skipped (omnibus-gitlab has no init script) Init script up-to-date? ... skipped (omnibus-gitlab has no init script) Projects have namespace: ... 6/1 ... yes 6/2 ... yes 6/3 ... yes 6/4 ... yes 5/5 ... yes 6/6 ... yes 6/7 ... yes 4/9 ... yes 6/10 ... yes 6/11 ... yes 3/12 ... yes 4/13 ... yes 3/15 ... yes 5/16 ... yes 2/17 ... yes 6/18 ... yes 2/26 ... yes 11/27 ... yes 14/28 ... yes 11/29 ... yes 11/31 ... yes 11/32 ... yes 11/33 ... yes 11/34 ... yes 11/35 ... yes 11/36 ... yes 11/37 ... yes 11/38 ... yes 3/39 ... yes 3/40 ... yes 3/41 ... yes 3/42 ... yes 3/43 ... yes 3/44 ... yes 3/45 ... yes 3/46 ... yes 2/47 ... yes Redis version >= 5.0.0? ... yes Ruby version >= 2.7.2 ? ... yes (2.7.4) Git version >= 2.33.0 ? ... yes (2.33.0) Git user has default SSH configuration? ... yes Active users: ... 9 Is authorized keys file accessible? ... yes GitLab configured to store new projects in hashed storage? ... yes All projects are in hashed storage? ... yes

    Checking GitLab App ... Finished

    Checking GitLab subtasks ... Finished

    Possible fixes

Edited by 🤖 GitLab Bot 🤖