Fix coverity-1604666

Coverity recently flaged an error in which the return value for
EVP_MD_get_size wasn't checked for negative values prior to use, which
can cause underflow later in the function.

Just add the check and error out if get_size returns an error.

Reviewed-by: Paul Dale <ppzgs1@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24896)

diff --git a/crypto/rsa/rsa_oaep.c b/crypto/rsa/rsa_oaep.c
index 0ec18b969119964543e48a17dc55c3d7abd7ad47..86a81499c2042398a7395990486e80ebe5996c2a 100644 (file)
--- a/crypto/rsa/rsa_oaep.c
+++ b/crypto/rsa/rsa_oaep.c
@@ -208,7 +208,7 @@ int RSA_padding_check_PKCS1_OAEP_mgf1(unsigned char *to, int tlen,
 
     mdlen = EVP_MD_get_size(md);
 
-    if (tlen <= 0 || flen <= 0)
+    if (tlen <= 0 || flen <= 0 || mdlen <= 0)
         return -1;
     /*
      * |num| is the length of the modulus; |flen| is the length of the