gnutls client should not negotiate TLS 1.3 if the private key from PKCS#11 does not support RSA-PSS nor raw-RSA
Description of problem:
When applications using GnuTLS with client certificate key in smart card connect using TLS, the library should check the PKCS#11 module capabilities before negotiating the TLS 1.3 and failing later without any mechanisms to try
originally reported in: https://bugzilla.redhat.com/show_bug.cgi?id=1681006
Version of gnutls used:
3.6.5
Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)
RHEL
How reproducible:
always
Steps to Reproduce:
- Prerequisites:
- PKCS#11 module without RSA-PSS nor raw-RSA mechanisms support
- The server requests TLS client authentication
- Start a server requesting TLS client authentication:
$ gnutls-serv --http --require-client-cert --x509cafile ca.pem -d 9 --x509certfile cert.pem --x509keyfile key.pem
- Connect to the server using wget (or any other application) compiled against GnuTLS with private key in the PKCS#11 module:
$ GNUTLS_DEBUG_LEVEL=9 wget --no-check-certificate --certificate="pkcs11:token=SomeDevice;object=cert;type=cert" --private-key="pkcs11:token=SomeDevice;object=key;type=private?pin-value=111111" --debug --tries 1 https://localhost:5556/
Actual results:
The TLS 1.3 connection fails:
[...]
gnutls[4]: checking cert compat with RSA-PSS-SHA512
gnutls[4]: checking cert compat with RSA-PSS-RSAE-SHA512
gnutls[4]: checking cert compat with ECDSA-SECP521R1-SHA512
gnutls[4]: cannot use privkey of RSA with ECDSA-SECP521R1-SHA512
gnutls[4]: checking cert compat with RSA-SHA1
gnutls[3]: ASSERT: signature.c[_gnutls_session_sign_algo_enabled]:365
gnutls[4]: Signature algorithm RSA-SHA1 is not enabled
gnutls[4]: checking cert compat with ECDSA-SHA1
gnutls[4]: cannot use privkey of RSA with ECDSA-SHA1
gnutls[3]: ASSERT: tls13/certificate_verify.c[_gnutls13_send_certificate_verify]:192
gnutls[3]: ASSERT: handshake-tls13.c[_gnutls13_handshake_client]:178
GnuTLS: The signature is incompatible with the public key.
gnutls[5]: REC[0x55bab44e9510]: Start of epoch cleanup
gnutls[5]: REC[0x55bab44e9510]: Epoch #0 freed
gnutls[5]: REC[0x55bab44e9510]: End of epoch cleanup
gnutls[5]: REC[0x55bab44e9510]: Epoch #1 freed
Closed fd 4
Unable to establish SSL connection.
Expected results:
The TLS connection should be downgraded to TLS 1.2
Edited by Anderson Sasaki