From: Viktor Dukhovni <openssl-users@dukhovni.org>
Date: Sat, 6 Dec 2025 04:04:36 +0000 (+1100)
Subject: The return value of RSA_*_{en,de}crypt() is signed
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=666737469234597a97fd34d58267add5418919ef;p=thirdparty%2Fopenssl.git

The return value of RSA_*_{en,de}crypt() is signed

The functions RSA_(public|private)_(en|de)crypt() return a signed
result, in particular `-1` may be returned on error, so the caller
MUST treat the value as signed.

Reviewed-by: Saša Nedvědický <sashan@openssl.org>
Reviewed-by: Matt Caswell <matt@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/29323)

(cherry picked from commit f247d36074353f44596cb941c0a5f929c2e46e67)
---

diff --git a/crypto/rsa/rsa_gen.c b/crypto/rsa/rsa_gen.c
index 073524175a2..2f21d2b6687 100644
--- a/crypto/rsa/rsa_gen.c
+++ b/crypto/rsa/rsa_gen.c
@@ -684,12 +684,12 @@ static int rsa_keygen(OSSL_LIB_CTX *libctx, RSA *rsa, int bits, int primes,
 static int rsa_keygen_pairwise_test(RSA *rsa, OSSL_CALLBACK *cb, void *cbarg)
 {
     int ret = 0;
-    unsigned int plaintxt_len;
     unsigned char *plaintxt = NULL;
-    unsigned int ciphertxt_len;
     unsigned char *ciphertxt = NULL;
     unsigned char *decoded = NULL;
-    unsigned int decoded_len;
+    int plaintxt_len;
+    int ciphertxt_len;
+    int decoded_len;
     int padding = RSA_NO_PADDING;
     OSSL_SELF_TEST *st = NULL;