Subgroup order validation for DH key exchanges
Lines 206-214 of https://gitlab.com/gnutls/gnutls/blob/3e958602b907584f4c34af68820f1c2e37194dd8/lib/nettle/pk.c perform validation on a received Diffie-Hellman key exchange value. If nettle supported only Diffie-Hellman groups with safe primes, the current check that y is not <= 1 or >= p-1 would be sufficient. However, it appears that nettle uses DSA primes for DH key exchanges (from wrap_nettle_pk_generate_params function), so that (p-1)/2 is not prime. In this case, it is necessary to perform an additional check that y^q == 1 mod p in order to prevent small subgroup attacks. OpenSSL recently changed their code to perform proper validation after CVE-2016-0701 (https://www.openssl.org/news/secadv/20160128.txt).