gnutls_pubkey_verify_data2 calls fail erroneously with GNUTLS_E_INVALID_REQUEST when GNUTLS_VERIFY_DISABLE_CA_SIGN flag is set
Description of problem:
When calling gnutls_pubkey_verify_data2
with the GNUTLS_VERIFY_DISABLE_CA_SIGN
flag set, GNUTLS_E_INVALID_REQUEST
is asserted due to an erroneous comparison of the OLD_PUBKEY_VERIFY_FLAG_TLS1_RSA
define (previously defined as GNUTLS_PUBKEY_VERIFY_FLAG_TLS1_RSA
in gnutls_pubkey_flags_t
) with the gnutls_certificate_verify_flags
passed to the function.
This occurs when the bitwise AND comparison of the gnutls_certificate_verify_flags
against OLD_PUBKEY_VERIFY_FLAG_TLS1_RSA
returns 1
, as the GNUTLS_VERIFY_DISABLE_CA_SIGN
gnutls_certificate_verify_flag
and the deprecated GNUTLS_PUBKEY_VERIFY_FLAG_TLS1_RSA
gnutls_pubkey_flag
share the least significant bit of flags.
The bug was introduced in version 3.4.2 with this specific commit: 0422a67e
I assume the OLD_PUBKEY_VERIFY_FLAG_TLS1_RSA
define was added to maintain backwards compatibility with the old calling of gnutls_pubkey_verify_data2
that utilized gnutls_pubkey_flags_t
.
Looking at internal gnutls calls of gnutls_pubkey_verify_data2
, I don't see any case where this would be necessary any more. Perhaps this comparison is deprecated as well and could be removed?
gnutls_pubkey_verify_hash2
has the same comparison and the same issue might be present there as well, though I have not tested this.
Version of gnutls used:
Tested in 3.6.7, issue is not present in versions prior to 3.4.2.
Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)
N/A, built directly from gnutls source.
How reproducible:
Steps to Reproduce:
- Call
gnutls_pubkey_verify_data2
with a valid signature and theGNUTLS_VERIFY_DISABLE_CA_SIGN
flag set
Actual results:
gnutls_pubkey_verify_data2
asserts GNUTLS_E_INVALID_REQUEST
Expected results:
gnutls_pubkey_verify_data2
returns 0