Cannot use gnutls_pkcs7_sign() with GNUTLS_DIG_SHAKE_256

Hi all,

I'm building PQC into libjcat, using GnuTLS. I'm loading a ML-DSA87 PKCS#7 certificate (and private key) and then signing some data using gnutls_pkcs7_sign(). This works wonderfully, until I add GNUTLS_PKCS7_INCLUDE_TIME which causes the sign process to abort with GNUTLS_E_INVALID_REQUEST.

The backtrace explains a little what's happening:

#1  0x00007ffff784c0ad in gnutls_hash_fast (algorithm=GNUTLS_DIG_UNKNOWN, ptext=0x41fe68 <payload_str>, ptext_len=14, digest=digest@entry=0x7fffffffcdc0) at ../../lib/crypto-api.c:968
#2  0x00007ffff78eb8b9 in write_attributes (root=0x7ffff79ba1c3 "signerInfos.?LAST.signedAttrs", c2=0x461740, data=0x7fffffffd010, me=0x7ffff7a74eb0 <hash_algorithms+1456>, other_attrs=<optimized out>, flags=2) at ../../../lib/x509/pkcs7.c:2273
#3  gnutls_pkcs7_sign (pkcs7=0x452d50, signer=0x45dfe0, signer_key=0x458ed0, data=0x7fffffffd010, signed_attrs=<optimized out>, unsigned_attrs=<optimized out>, dig=GNUTLS_DIG_SHAKE_256, flags=2) at ../../../lib/x509/pkcs7.c:2452

So, we're calling gnutls_pkcs7_sign(dig: gnutls_digest_algorithm_t=GNUTLS_DIG_SHAKE_256), write_attributes(me: mac_entry_st) which when calls gnutls_hash_fast(MAC_TO_DIG(me->id)) -- MAC_TO_DIG() being an alias to _gnutls_mac_to_dig().

Looking at the source code for that, I see:

inline static gnutls_digest_algorithm_t
_gnutls_mac_to_dig(gnutls_mac_algorithm_t mac)
{
	if (unlikely(mac >= GNUTLS_MAC_AEAD))
		return GNUTLS_DIG_UNKNOWN;
	return (gnutls_digest_algorithm_t)mac;
}

...and sure enough, GNUTLS_MAC_SHAKE_256 is indeed larger than GNUTLS_MAC_AEAD as defined in gnutls_mac_algorithm_t from lib/includes/gnutls/gnutls.h.in

I'm happy to write a merge request to fix this, but I'm not sure what the correct fix would be. Moving GNUTLS_MAC_SHAKE_256 to be a lower number than GNUTLS_MAC_AEAD would work -- i.e. just after GNUTLS_MAC_STREEBOG_512 -- but that's probably an API break.

I personally think adding the two constants to _gnutls_mac_to_dig() would be best. Thanks!