gnutls_dtls_get_data_mtu() does not return MAXIMUM transmission unit.
Test case: OpenConnect client, built against OpenSSL 1.0.2, talking to ocserv.
It'll use PSK-NEGOTIATE
but OpenSSL 1.0.2 doesn't support any AEAD ciphersuites so it ends up using PSK-AES128-CBC-SHA
.
OpenConnect correctly calculates the data MTU on the client side. For a link MTU of 1500, DTLS record MTU (without IP and UDP headers) is 1472.
The data MTU is then ((1472 - 13 - 16) & 0xfff0) - 20 - 1
== 1419 bytes.
So OpenConnect sends a MTU probe at 1418 bytes (+ AnyConnect DTLS record type byte==1419), which is a 1497-byte IP packet and fits perfectly into the MTU.
However, ocserv fails to respond — it seems to send a bunch of fragments instead. I believe this is because gnutls_dtls_get_data_mtu()
is returning the wrong value. For this connection (if I test it on the client side in OpenConnect+GnuTLS) it seems to return 1407 bytes not 1419. That would be correct for encrypt-then-mac (it's ((1472 - 13 - 16 - 20) & 0xfff0) - 1
) but is not correct for mac-then-encrypt. And in this case we are definitely doing mac-then-encrypt. The GnuTLS client doesn't put the encrypt-then-mac extension into its ClientHello, and as noted in #139 (closed) the ocserv server never chooses to respond with it even when a client does propose it.
In fact I don't think it's etm-related; I think gnutls_dtls_get_data_mtu()
is just returning a pessimal value based on some "maximum overhead" which etm would happen to hit by coincidence in this case since it would end up with 15 bytes of padding.
But if you want to do that, you can't call it gnutls_dtls_get_data_mtu()
. You have to take the 'm' out and fix the documentation. Because if the function is just defined to return some value of packet size that will be accepted, it's almost useless. That definition would permit it always to return '1'. Applications want to know the maximum transmission unit. There's a clue in the name.
FWIW my test cases for DTLS_get_data_mtu()
will iterate over a few record MTU sizes, getting the data MTU for each one.
Then they send DTLS payloads of each size between the lowest and highest value returned from DTLS_get_data_mtu()
, noting the actual DTLS record size each time.
Now we have two sets of data. One set of { record MTU, payload MTU }
tuples, and one set of { payload size, record size }
observations. For each combination of those sets, it is a test failure if either:
• payload size <= payload MTU && record size > record MTU
OR
• payload_size > payload MTU && record size <= record MTU
The second of those checks will catch the DTLS_get_data_mtu()
function being overly pessimal, and not returning the MAXIMUM transmission unit.