gnutls_pubkey_verify_data2 calls fail erroneously with GNUTLS_E_INVALID_REQUEST when GNUTLS_VERIFY_DISABLE_CA_SIGN flag is set

Description of problem:

When calling gnutls_pubkey_verify_data2 with the GNUTLS_VERIFY_DISABLE_CA_SIGN flag set, GNUTLS_E_INVALID_REQUEST is asserted due to an erroneous comparison of the OLD_PUBKEY_VERIFY_FLAG_TLS1_RSA define (previously defined as GNUTLS_PUBKEY_VERIFY_FLAG_TLS1_RSA in gnutls_pubkey_flags_t) with the gnutls_certificate_verify_flags passed to the function.

This occurs when the bitwise AND comparison of the gnutls_certificate_verify_flags against OLD_PUBKEY_VERIFY_FLAG_TLS1_RSA returns 1, as the GNUTLS_VERIFY_DISABLE_CA_SIGN gnutls_certificate_verify_flag and the deprecated GNUTLS_PUBKEY_VERIFY_FLAG_TLS1_RSA gnutls_pubkey_flag share the least significant bit of flags.

The bug was introduced in version 3.4.2 with this specific commit: 0422a67e

I assume the OLD_PUBKEY_VERIFY_FLAG_TLS1_RSA define was added to maintain backwards compatibility with the old calling of gnutls_pubkey_verify_data2 that utilized gnutls_pubkey_flags_t. Looking at internal gnutls calls of gnutls_pubkey_verify_data2, I don't see any case where this would be necessary any more. Perhaps this comparison is deprecated as well and could be removed?

gnutls_pubkey_verify_hash2 has the same comparison and the same issue might be present there as well, though I have not tested this.

Version of gnutls used:

Tested in 3.6.7, issue is not present in versions prior to 3.4.2.

Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)

N/A, built directly from gnutls source.

How reproducible:

Steps to Reproduce:

• Call gnutls_pubkey_verify_data2 with a valid signature and the GNUTLS_VERIFY_DISABLE_CA_SIGN flag set

Actual results:

gnutls_pubkey_verify_data2 asserts GNUTLS_E_INVALID_REQUEST

Expected results:

gnutls_pubkey_verify_data2 returns 0