From: Bernd Edlinger <bernd.edlinger@hotmail.de>
Date: Sun, 23 Jul 2023 12:27:54 +0000 (+0200)
Subject: Make DH_check set some error bits in recently added error
X-Git-Tag: OpenSSL_1_1_1v~5
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=eec805ee71356c06f9a86192fa06507c3bb92b09;p=thirdparty%2Fopenssl.git

Make DH_check set some error bits in recently added error

The pre-existing error cases where DH_check returned zero
are not related to the dh params in any way, but are only
triggered by out-of-memory errors, therefore having *ret
set to zero feels right, but since the new error case is
triggered by too large p values that is something different.
On the other hand some callers of this function might not
be prepared to handle the return value correctly but only
rely on *ret. Therefore we set some error bits in *ret as
additional safety measure.

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/21533)
---

diff --git a/crypto/dh/dh_check.c b/crypto/dh/dh_check.c
index e5f9dd5030e..2001d2e7cb1 100644
--- a/crypto/dh/dh_check.c
+++ b/crypto/dh/dh_check.c
@@ -104,6 +104,7 @@ int DH_check(const DH *dh, int *ret)
     /* Don't do any checks at all with an excessively large modulus */
     if (BN_num_bits(dh->p) > OPENSSL_DH_CHECK_MAX_MODULUS_BITS) {
         DHerr(DH_F_DH_CHECK, DH_R_MODULUS_TOO_LARGE);
+        *ret = DH_CHECK_P_NOT_PRIME;
         return 0;
     }