psk_ke_modes_recv_params() wrongly sets HSK_PSK_KE_MODE_INVALID

Description of problem:

If the preferred side (as per session->internals.priorities->server_precedence) only supports one algorithm and it is not the first in the other side's list of algorithms, psk_ke_modes_recv_params wrongly sets session->internals.hsk_flags to HSK_PSK_KE_MODE_INVALID.

Observed in GnuTLS 3.7.2 compiled from the official source tarball without any special configure arguments.

The issue was discovered while analyzing https://forum.filezilla-project.org/viewtopic.php?t=54333

How to reproduce:

You can easily reproduce this issue:

• Applying the attached reproducer.diff, which just swaps the algorithms in the psk_key_exchange_modes extension Client Hello.
• Run gnutls-serv -d9999
• Connect to it with gnutls-cli 127.0.0.1 -p 5556
• In the output of gnutls-serv look for |<3>| ASSERT: psk_ke_modes.c[psk_ke_modes_recv_params]:192 which is printed when HSK_PSK_KE_MODE_INVALID is set.

In this scenario the following happens in psk_ke_modes.c:

• Line 156 is reached.
• By line 174, the following values are held:
  • session->internals.priorities->server_precedence is false
  • dhpsk_pos is 0
  • psk_pos is MAX_POS
  • cli_dhpsk_pos is 1
  • cli_psk_pos is 0
• As result, neither mode is set in session->internals.hsk_flags and line 191 is reached.

Proposed patch:

I have attached a simple fix for the issue in psk_ke_modes_send_params.diff