Remote OOB read in PKCS#11 RSA decrypt path via short ClientKeyExchange

Hi all,

As far as I can tell, there is a remotely reachable out-of-bounds read in the PKCS#11-backed RSA decryption path used by TLS RSA and RSA-PSK server handshakes.

The issue is in lib/pkcs11_privkey.c, in _gnutls_pkcs11_privkey_decrypt_data2(). The function allocates a temporary buffer using ciphertext->size, but later performs a constant-time copy loop that always reads plaintext_size bytes from that buffer, even when decryption failed and the buffer is shorter than plaintext_size.

Relevant code:

unsigned long siglen = ciphertext->size;
buffer = gnutls_malloc(siglen);
...
if (rv != CKR_OK) {
        ret = pkcs11_rv_to_err(rv);
} else if (siglen != plaintext_size) {
        ret = GNUTLS_E_INVALID_REQUEST;
}

mask = ((uint32_t)ret >> 31) - 1U;
for (size_t i = 0; i < plaintext_size; i++) {
        value = (buffer[i] & mask) + (plaintext[i] & ~mask);
        plaintext[i] = value;
}

In addition to this being reachable from a public API with no documentation about length requirements, this is reachable from the server-side TLS handshake code. In both lib/auth/rsa.c and lib/auth/rsa_psk.c, the peer controls the 16-bit RSA ciphertext length in ClientKeyExchange. The code checks only that the declared length matches the remaining packet bytes, then passes the ciphertext and fixed GNUTLS_MASTER_SIZE to gnutls_privkey_decrypt_data2():

dsize = _gnutls_read_uint16(data);
if (dsize != data_size)
        return GNUTLS_E_UNEXPECTED_PACKET_LENGTH;
ciphertext.size = dsize;

gnutls_privkey_decrypt_data2(session->internals.selected_key, 0,
                             &ciphertext, ..., GNUTLS_MASTER_SIZE);

There does not appear to be a check that the ciphertext length matches the RSA modulus length, or even that it is at least GNUTLS_MASTER_SIZE. As a result, a short ciphertext such as 0 to 47 bytes can reach the PKCS#11 path. If PKCS#11 decryption fails, the copy loop still reads past the end of buffer, causing undefined behavior and likely a crash.

Reproduction steps:

1. Configure a GnuTLS server with a PKCS#11-backed RSA private key.
2. Ensure TLS RSA or RSA-PSK key exchange is enabled.
3. Connect and send a ClientKeyExchange with a short RSA ciphertext length, for example 0 to 47 bytes.
4. Ensure the packet is otherwise well formed so the handshake parser accepts the length field.
5. Observe an out-of-bounds read in _gnutls_pkcs11_privkey_decrypt_data2() under ASan or Valgrind, or a crash depending on allocator and memory layout.

A minimal fix would be to size the temporary buffer to at least plaintext_size, while keeping the constant-time copy structure intact:

diff --git a/lib/pkcs11_privkey.c b/lib/pkcs11_privkey.c
--- a/lib/pkcs11_privkey.c
+++ b/lib/pkcs11_privkey.c
@@
-       buffer = gnutls_malloc(siglen);
+       size_t alloc_len = MAX((size_t)siglen, plaintext_size);
+       buffer = gnutls_calloc(1, alloc_len);
        if (!buffer) {
                gnutls_assert();
                return GNUTLS_E_MEMORY_ERROR;
        }

This prevents the out-of-bounds read on failure without changing the constant-time copy loop. An additional exact-length validation in the TLS RSA and RSA-PSK handshake paths may also be appropriate.

Any questions, please reach out.

Cheers, Josh