max-fragment TLS extension is negotiated under DTLS but ignored for fragmenting handshake packets
(This is a copy of http://lists.gnutls.org/pipermail/gnutls-devel/2015-December/007850.html )
I am testing a development build of the BouncyCastle DTLS client against the MacPorts build of GnuTLS (gnutls-serv 3.3.19).
At first it was not connecting, the client dropping packets after the ServerHello for some reason. I eventually noticed that the client was requesting max_fragment_length extension, and the server was accepting it, agreeing on MaxFragmentLength.2^9(1). The client then appears to ignore the Certificate message as it is too large. (Without that extension, connections work fine).
See attached capture of the handshake start, noting the ServerHello with max_fragment_length=1, and the Certificate message with Fragment Length 932.