Avoid excessive CPU usage with large inputs to idn2_lookup_u8()
The punycode encoding was done on any input sizes, the output length check happended afterwards. Due to the O(N^2) nature of the encoding, this lead to excessive CPU usage on large inputs. This was unneeded because the result was IDN2_TOO_BIG_DOMAIN anyways. It allowed a Denial-Of-Service (DOS) if the calling functions didn't have their own length check. In fact we saw this as timeout issues when fuzzing GnuTLS via OSS-Fuzz. The affected functions are idn2_lookup_u8(), idn2_lookup_ul(), idn2_to_ascii_4i, idn2_to_ascii_4i2(), idn2_to_ascii_4z(), idn2_to_ascii_8z(), idn2_to_ascii_lz(). Also the tool 'idn2' is affected in lookup/toASCII mode.
Showing with 8 additions and 0 deletions