Incorrect error returned in TLS 1.3 when an unsupported signature algorithm is used by a client for Certificate Verify message signatures
Description of problem:
As I was writing tlsfuzzer tests to probe the correctness of client certificate handling by server implementations, it stood out that GNUTLS is returning a handshake_failure error when a client sends an RSA pkcs1 signature that the server should not accept. The error returned should be illegal_parameter in this case (openssl and tlslite conform).
Here is the description of the 2 errors from the RFC:
handshake_failure: Receipt of a "handshake_failure" alert message indicates that the sender was unable to negotiate an acceptable set of security parameters given the options available. illegal_parameter: A field in the handshake was incorrect or inconsistent with other fields. This alert is used for errors which conform to the formal protocol syntax but are otherwise incorrect.
The second correctly describes the situation, the client misbehaved sending a field (signature algorithm selected) that is inconsistent with other fields (the server sent proper support signature algorithms lists in the CertificateRequest message).
A handshake_failure is improper because it is applicable only when the server, after parsing a list of permissible options, discovers it can use none. It is not the case here as the server is the receiver, and the client sent an invalid parameter, not a field to negotiate upon.
Version of gnutls used:
Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)
Run the tests introduced here:
using the following command line to run a GnuTLS server:
$ gnutls-serv --http --priority NORMAL:-VERS-ALL:+VERS-TLS1.3 -p 4433 --x509keyfile=tests/serverX509Key.pem --x509certfile=tests/serverX509Cert.pem
Steps to Reproduce:
- run the server
- run the test
- observe the errors reported by the test
Invalid pkcs1 signatures produce a handshake_failure error
Invalid pkcs1 signatures produce an illegal_paramter error