From: Jiasheng Jiang <jiasheng@purdue.edu>
Date: Fri, 22 Mar 2024 22:22:23 +0000 (+0000)
Subject: signature/rsa_sig.c: Add checks for the EVP_MD_get_size()
X-Git-Tag: openssl-3.4.0-alpha1~782
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=6c0f154750a3380cced8ddab44d7ad100b6ab984;p=thirdparty%2Fopenssl.git

signature/rsa_sig.c: Add checks for the EVP_MD_get_size()

Add checks for the EVP_MD_get_size() to avoid integer overflow and then explicitly cast from int to size_t.

Fixes: 6f4b766315 ("PROV: add RSA signature implementation")
Signed-off-by: Jiasheng Jiang <jiasheng@purdue.edu>

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Todd Short <todd.short@me.com>
Reviewed-by: Neil Horman <nhorman@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23949)
---

diff --git a/providers/implementations/signature/rsa_sig.c b/providers/implementations/signature/rsa_sig.c
index 76db37dd02c..cc7353bbcaf 100644
--- a/providers/implementations/signature/rsa_sig.c
+++ b/providers/implementations/signature/rsa_sig.c
@@ -114,8 +114,14 @@ typedef struct {
 
 static size_t rsa_get_md_size(const PROV_RSA_CTX *prsactx)
 {
-    if (prsactx->md != NULL)
-        return EVP_MD_get_size(prsactx->md);
+    int md_size;
+
+    if (prsactx->md != NULL) {
+        md_size = EVP_MD_get_size(prsactx->md);
+        if (md_size <= 0)
+            return 0;
+        return md_size;
+    }
     return 0;
 }