apparent bug in _gnutls_x509_der_encode with fix/workaround that shouldn't work
Description of problem:
Attached is a fix/workaround to _gnutls_x509_der_encode (in lib/x509/common.c) that should not work but does. The change provides a buffer to asn1_der_coding rather than zero when only the size is needed. The suspected reason that this works is without the buffer asn1_der_coding is doing some inplace modification affecting later calls. That reason is not proven. A test case (c++ program) to reproduce the problem and shell output giving limited debug output is (or will be shortly) provided. A prior version of gnutls worked.
Version of gnutls used:
3.6.14 (on FreeBSD 12.1-STABLE #0 r363326M)
Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)
FreeBSD ports collection (revision 542586) compiled with FreeBSD clang version 10.0.0
How reproducible:
100% of the time with provided c++ program. Fix works 100% of the time in cases where it should work.
Steps to Reproduce:
- one - read instructions in first few comment lines of provided test program
- two - follow directions to compile and run testcase 0 to 5 (only 1 and 5 should not produce core dump)
- three - apply patch to gnutls and relink - testcase 1 and 5 now produce success
Actual results:
Testcase 0-5 produce core dump without patch to gnutls. Testcase 1 and 5 succeed with patch.
Expected results:
Testcase 1 and 5 should succeed. Other testcases exist to provide information and are OK to core dump. But it would be nice (tm) if a few didn't fail. patch-common.c