Fix coverity-1604666

Coverity recently flaged an error in which the return value for
EVP_MD_get_size wasn't checked for negative values prior to use, which
can cause underflow later in the function.

Just add the check and error out if get_size returns an error.

Reviewed-by: Paul Dale <ppzgs1@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24896)

(cherry picked from commit 22e08c7cdc596d4f16749811d1022fb8b07a8e41)

diff --git a/crypto/rsa/rsa_oaep.c b/crypto/rsa/rsa_oaep.c
index d9be1a4f98c7ba0f70bcbb3946368b64b067b0f0..730765fe03d20dcbe332623335104578e05f5f12 100644 (file)
--- a/crypto/rsa/rsa_oaep.c
+++ b/crypto/rsa/rsa_oaep.c
@@ -186,7 +186,7 @@ int RSA_padding_check_PKCS1_OAEP_mgf1(unsigned char *to, int tlen,
 
     mdlen = EVP_MD_get_size(md);
 
-    if (tlen <= 0 || flen <= 0)
+    if (tlen <= 0 || flen <= 0 || mdlen <= 0)
         return -1;
     /*
      * |num| is the length of the modulus; |flen| is the length of the