gnutls_privkey_decrypt_data frees output argument in some conditions
We were previously using gnutls_privkey_decrypt_data
this way:
gnutls_datum_t out;
int err = gnutls_privkey_decrypt_data(key, 0, &dat, &out);
as the documentation for gnutls_privkey_decrypt_data
makes no requirement for out
.
However since the last release 3.8.4, this would cause a crash when the decryption fails,
because gnutls_privkey_decrypt_data
would attempt to free the buffer pointed by out
, which would cause occasional crashes since out
is uninitialized and might point to random values.
This can be easily fixed by initializing out to a null pointer,
however I believe this is a GnuTLS bug because the documentation specifies plaintext
(out
) as an output-only argument.
If out
was a re-used stack variable pointing to another valid buffer, this would lead to unexpected/undefined behaviours as the buffer would be freed depending on if decryption succeeded.
Best regards