Trusted CA certificates with keys that should have been rejected by the verification profile are accepted for TLS

Description of problem:

I have created a self-signed certificate with 512bit RSA private key, with otherwise "strong" checksums

I use that certificate & key with gnutls-serv / gnutls-cli, and cross-reference it with openssl s_server / s_client

openssl s_server fails to start with such key, claiming key size is too small

gnutls-serv starts with such key/certificate just fine.

openssl s_client fails to connect to gnutls-serv using such certificate, claiming key size is too small

gnutls-cli connects to gnutls-serv just fine

Version of gnutls used:

• 3.6.7-4 Debian
• 3.6.11.1-2 Ubuntu
• 3.6.11-1.fc31.x86_64 Fedora

Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)

various

How reproducible:

Steps to Reproduce:

• Terminal 1:

gnutls-serv --x509keyfile=rsa512.cert --x509certfile=rsa512.cert &

• Terminal 2:

gnutls-cli --x509cafile=rsa512.cert localhost:5556 <<EOF
EXIT
EOF

Actual results:

Connection established, ie. TLSv1.3 strong DH params, yet RSA 512 bit certificate key

Expected results:

Connection failure, as RSA 512 bit certificate reduces security below the default configured one. Gnutls is built with NORMAL priority. Thus expecting 1014 bits minimum size for RSA keys, to ensure minimum 80bits of security.

Sample RSA 512bits certificate with private key attached

rsa512.cert