gnutls_get_data_mtu() can return smaller values than passed to gnutls_set_data_mtu()
Description of problem:
I am using openconnect 7.08 to contact a real cisco vpn concentrator. The concentrator uses an MTU of 1300, but openconnect (with gnutls dtls) is unable to send more than 1290 bytes.
I determined that this is because gnutls_dtls_get_data_mtu() returns the wrong value and gnutls_record_send does not allow more than 1290 bytes to be sent at a time.
I believe this is because gnutls_dtls_get_data_mtu() does not correctly reverse the internal mtu computation that gnutls_dtls_set_data_mtu performs, when CBC ciphers are used.
gnutls_dtls_set_data_mtu calls record_overhead_rt with an est_data parameter of 1301 (which came from openconnect) gnutls_dtls_get_data_mtu calls record_overhead_rt with an est_data parameter of session->internals.dtls.mtu (which I think was 1352, I didn't save the output from when I fetched that)
What I do know is that when I used the patch from http://lists.infradead.org/pipermail/openconnect-devel/2018-January/004652.html with openconnect, then a requested data mtu of 1301 with ciphersuite AES128-CBC-SHA1 resulted in an actual data mtu of 1290, but requesting 1302 bytes resulted in an actual data mtu of 1307
Version of gnutls used:
3.5.8
Distributor of gnutls (e.g., Ubuntu, Fedora, RHEL)
Debian
How reproducible:
Ciphersuite/mtu dependent.