TLS handshake fails between OpenSSL 3.6.0 and GnuTLS

Description of problem:

TLS handshake fails when connecting to specific HTTPS servers after upgrading OpenSSL from 3.5.4 to 3.6.0.

It is definitely a problem caused by updating OpenSSL to 3.6.0 because if you roll back to 3.5.4, everything works again.

This was initially detected in WebKit applications, so there is a related bug in Bugzilla.

Version of gnutls used:

3.8.10

Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)

Arch Linux

How reproducible:

1. Use Arch Linux (up to date as of October 2025).
2. Ensure OpenSSL 3.6.0 is installed.
3. Install and configure nginx with a simple self-signed TLS certificate.
4. Launch any GnuTLS client such Epiphany, wget or aria2.
5. Open https://localhost/.

Actual results:

Epiphany shows an SSL/TLS error 'Peer failed to perform TLS handshake: Error decoding the received TLS packet.'

In wget or aria2 similar error 'GnuTLS: Error decoding the received TLS packet'.

Expected results:

Everything should work fine, regardless of the version of OpenSSL installed.