rsa_kem: validate RSA_public_encrypt() result in RSASVE

RSA_public_encrypt() returns the number of bytes written on success and
-1 on failure. With the existing `if (ret)` check, a provider-side RSA KEM
encapsulation can incorrectly succeed when the underlying RSA public
encrypt operation fails. In that case the code reports success, returns
lengths as if encapsulation completed normally, and leaves the freshly
generated secret available instead of discarding it.

Tighten the success condition so RSASVE only succeeds when
RSA_public_encrypt() returns a positive value equal to the modulus-sized
output expected for RSA_NO_PADDING. Any other return value is treated as
failure, and the generated secret is cleansed before returning.

Fixes CVE-2026-31790
Signed-off-by: Nikola Pajkovsky <nikolap@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.foundation>
Reviewed-by: Tomas Mraz <tomas@openssl.foundation>
MergeDate: Mon Apr  6 19:45:38 2026
(cherry picked from commit 89dde74b69debbf0c4d0a0ee925de87638bbfe16)

diff --git a/providers/implementations/kem/rsa_kem.c b/providers/implementations/kem/rsa_kem.c
index 78124ba90b8d4a358c1324bbee78decc6f1ae8bb..ab28a3a1a2808dff4160d2280cfe954d46eed305 100644 (file)
--- a/providers/implementations/kem/rsa_kem.c
+++ b/providers/implementations/kem/rsa_kem.c
@@ -315,16 +315,17 @@ static int rsasve_generate(PROV_RSA_CTX *prsactx,
     /* Step(3): out = RSAEP((n,e), z) */
     ret = RSA_public_encrypt((int)nlen, secret, out, prsactx->rsa,
         RSA_NO_PADDING);
-    if (ret) {
-        ret = 1;
-        if (outlen != NULL)
-            *outlen = nlen;
-        if (secretlen != NULL)
-            *secretlen = nlen;
-    } else {
+    if (ret <= 0 || ret != (int)nlen) {
         OPENSSL_cleanse(secret, nlen);
+        return 0;
     }
-    return ret;
+
+    if (outlen != NULL)
+        *outlen = nlen;
+    if (secretlen != NULL)
+        *secretlen = nlen;
+
+    return 1;
 }
 
 /**