ssh-rsa keys (SHA-1 with old clients) no longer accepted in Gitlab 16.0 docker image

Gitlab 16.0 Community Edition docker image comes with a new baseline image of Ubuntu 22.04. This image comes with OpenSSH 8.9, a release that does not accept ssh-rsa keys by default. See release notes for OpenSSH 8.8.

This caught us by surprise since there is not mention about that in changelogs, and also according to documentation RSA keys are still being supported by Gitlab itself.

But when trying to ssh -Tvvv git@gitlab.our.host the following message appears in the container's logs:

==> /var/log/gitlab/sshd/current <==
2023-06-30_07:20:14.92265 userauth_pubkey: key type ssh-rsa not in PubkeyAcceptedAlgorithms [preauth]

I've managed to resolve this issue by adding the following crutch:

echo PubkeyAcceptedKeyTypes=+ssh-rsa >> /assets/sshd_config
service ssh reload

However I'd like to see this issue mentioned in the documentation and/or fixed on the level of gitlab-ce docker image.

EDIT: It's actually not as bad as it was described originally, see explanation in this comment.

This issue was automatically tagged with the label grouppipeline execution by TanukiStan, a machine learning classification model, with a probability of 0.67.

If this label is incorrect, please tag this issue with the correct group label as well as automation:ml wrong to help TanukiStan learn from its mistakes.

If you are unsure about the correct group, please do not leave the issue without a group label. Please refer to GitLab's shared responsibility functionality guidelines for more information on how to triage this kind of issues.

Authors who do not have permission to update labels can leave the issue to be triaged by group leaders initially assigned by TanukiStan

This message was generated automatically. You're welcome to improve it.

added automation:ml grouppipeline execution labels

added devopsverify sectionci labels

Hey @V0V4N, thank you for creating this issue!

To get the right eyes on your issue more quickly, we encourage you to follow the issue triage best practices.

PLEASE NOTE: You will need to use the @gitlab-bot label command to apply labels to this issue.

To set expectations, GitLab product managers or team members can't make any promise if they will proceed with this. However, we believe everyone can contribute, and welcome you to work on this proposed change, feature or bug fix. There is a bias for action, so you don't need to wait. Try and spin up that merge request yourself.

If you need help doing so, we're always open to mentor you to drive this change.

This message was generated automatically. You're welcome to improve it.

added automation:self-triage-encouraged label

mentioned in issue gitlab-org/quality/triage-reports#13168 (closed)

mentioned in issue gitlab-org/quality/triage-reports#13262 (closed)

Hi, we also experience the problem and were surprised by lack of this in changelogs. Also the host key using rsa algo is now rejected. I've found the sshd_config is staticaly put into docker image from omnibus-gitlab so I replaced it using volume in our docker-compose like:

    volumes:
      - './sshd_config:/assets/sshd_config'

where ./sshd_config is modified to contain

HostKeyAlgorithms +ssh-rsa
PubkeyAcceptedKeyTypes +ssh-rsa

If you plan to continue support rsa keys, I suggest to put the config options directly into omnibus assets.

mentioned in commit V0V4N/omnibus-gitlab@2a801c19

mentioned in merge request omnibus-gitlab!7035 (merged)

added Category:Omnibus Package Docker groupdistribution group::distributiondeploy maintenanceusability typemaintenance labels and removed grouppipeline execution label

added devopssystems sectioncore platform labels and removed devopsverify sectionci labels

@studenyp and @V0V4N, thank you for reporting this, but also for the multiple avenues as a workaround.

added For Scheduling priority3 severity3 labels

added 1 deleted label

mentioned in merge request omnibus-gitlab!7053 (closed)

mentioned in commit omnibus-gitlab@2fd73039

mentioned in commit omnibus-gitlab@173e20a0

Hey folks, I think there's a small but important confusion here.

When testing this locally on my machine, I created an ssh-rsa key and to my surprise, it worked! After digging and reading the changelog a bit more carefully, I learned that RSA keys were not deprecated. What was deprecated was the support for the SHA1 algorithm to handle signatures. SSH keys do not have these signatures signature built into them. These are prepared by the ssh client and host during during the communication. See this passage:

This release disables RSA signatures using the SHA-1 hash algorithm by default. This change has been made as the SHA-1 hash algorithm is cryptographically broken, and it is possible to create chosen-prefix hash collisions for <USD$50K [1]

For most users, this change should be invisible and there is no need to replace ssh-rsa keys. OpenSSH has supported RFC8332 RSA/SHA-256/512 signatures since release 7.2 and existing ssh-rsa keys will automatically use the stronger algorithm where possible.

My expectation is that people having this problem probably have been using a old SSH client. If someone having this problem could confirm to me that they can fix this simply by updating their ssh client, this would be great. See also these references/discussions:

• https://ikarus.sg/rsa-is-not-dead/
• https://superuser.com/questions/1556852/how-to-check-if-your-ssh-keys-are-in-the-ssh-rsa2-format/1556861#1556861

That said, in parallel, I think we can and should still implement a way to optionally enable/disable SHA1 checking on the server side, as proposed by @V0V4N. But I wanted to clarify this as it might be a better solution for users to update their ssh clients.

Hello there! I did a more deeper dive too and it seems to me that you're right.

Initially we've discovered this issue on the host that's running Ubuntu Trusty (14.04 LTS):

ubuntu@vm-host:~$ cat /etc/os-release 
NAME="Ubuntu"
VERSION="14.04.6 LTS, Trusty Tahr"
ID=ubuntu
ID_LIKE=debian
PRETTY_NAME="Ubuntu 14.04.6 LTS"
VERSION_ID="14.04"
HOME_URL="http://www.ubuntu.com/"
SUPPORT_URL="http://help.ubuntu.com/"
BUG_REPORT_URL="http://bugs.launchpad.net/ubuntu/"

ubuntu@vm-host:~$ ssh -V
OpenSSH_6.6.1p1 Ubuntu-2ubuntu2.13, OpenSSL 1.0.1f 6 Jan 2014

ubuntu@vm-host:~$ ssh-keygen -l -f .ssh/id_rsa
2048 <SHA-1 hash>  ubuntu@vm-host (RSA)

Anyways, I'll get back to this issue inside the merge request.

changed the description

mentioned in issue Alexand/growth-and-development#2 (closed)

mentioned in commit omnibus-gitlab@2059ec7e

assigned to @V0V4N

mentioned in commit V0V4N/omnibus-gitlab@d10fd70a

We can close this after this gets merged: Adds documentation for 16.3 feature update (omnibus-gitlab!7081 - closed)

The doc MR had to be moved to gitlab-org/gitlab, as we've recently moved our docs to this other project: Inform about new Omnibus GITLAB_ALLOW_SHA1_RSA var (!128815 - merged)

mentioned in merge request omnibus-gitlab!7081 (closed)

changed title from ssh-rsa keys no longer accepted in Gitlab 16.0 docker image to ssh-rsa keys (SHA-1 with old clients) no longer accepted in Gitlab 16.0 docker image

mentioned in merge request !128815 (merged)

closed

changed milestone to %16.3

removed 1 deleted label