ml_kem: return an error on catastrophic failure in decap

ml_kem: return an error on catastrophic failure in decap

ML-KEM decapsulation applies implicit rejection by copying the failure
key into the shared secret when the FO re-encryption check fails. This
is correct for a syntactically valid but incorrect ciphertext, and must
stay constant-time and ciphertext-dependent.

However, the same path was also taken when hash_kr() or encrypt_cpa()
failed outright, for example on a memory allocation failure inside
EVP_DigestInit_ex(). In that case decap() copied the failure key and
still returned success, so the caller derived a wrong shared secret with
no error reported. For QUIC this produces a handshake that cannot
converge: the derived keys diverge from the peer, packets fail to
decrypt, and the connection stalls until it times out, with no
diagnostic pointing at the real cause.

These primitive failures are not dependent on the ciphertext, so
reporting them as a hard error does not create a chosen-ciphertext
oracle and does not weaken the constant-time implicit rejection that
happens later via CRYPTO_memcmp() and constant_time_select_8(). Treat
them the same way the existing kdf() failure is already treated, by
raising an error and returning 0.

Also fix the comment, which referred to hash_g() where the code actually
calls hash_kr().

Reviewed-by: Bob Beck <beck@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.foundation>
MergeDate: Mon Jun 15 13:58:32 2026
(Merged from https://github.com/openssl/openssl/pull/31432)