DH private key size was one bit too large

In the case when no q parameter was given,
the function generate_key in dh_key.c did create
one bit too much, so the priv_key value was exceeding
the DH group size q = (p-1)/2.
When the length is used in this case the limit is also
one bit too high, but for backward compatibility this
limit was left as is, instead we have to silently reduce
the value by one.

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/27870)

(cherry picked from commit d6510d99ae4a8a23f54fdfb1473af6a920da8345)

diff --git a/crypto/dh/dh_key.c b/crypto/dh/dh_key.c
index 7132b9b68e5372dd2ad6fe7e5336f2dda8594040..7d5fa9031177dce946cbfc972c8a6d44f7500dd5 100644 (file)
--- a/crypto/dh/dh_key.c
+++ b/crypto/dh/dh_key.c
@@ -267,7 +267,7 @@ static int generate_key(DH *dh)
     int ok = 0;
     int generate_new_key = 0;
 #ifndef FIPS_MODULE
-    unsigned l;
+    int l;
 #endif
     BN_CTX *ctx = NULL;
     BIGNUM *pub_key = NULL, *priv_key = NULL;
@@ -327,11 +327,13 @@ static int generate_key(DH *dh)
                 goto err;
 #else
             if (dh->params.q == NULL) {
-                /* secret exponent length, must satisfy 2^(l-1) <= p */
-                if (dh->length != 0
-                    && dh->length >= BN_num_bits(dh->params.p))
+                /* secret exponent length, must satisfy 2^l < (p-1)/2 */
+                l = BN_num_bits(dh->params.p);
+                if (dh->length >= l)
                     goto err;
-                l = dh->length ? dh->length : BN_num_bits(dh->params.p) - 1;
+                l -= 2;
+                if (dh->length != 0 && dh->length < l)
+                    l = dh->length;
                 if (!BN_priv_rand_ex(priv_key, l, BN_RAND_TOP_ONE,
                                      BN_RAND_BOTTOM_ANY, 0, ctx))
                     goto err;