DH private key size was one bit too large

In the case when no q parameter was given,
the function generate_key in dh_key.c did create
one bit too much, so the priv_key value was exceeding
the DH group size q = (p-1)/2.
When the length is used in this case the limit is also
one bit too high, but for backward compatibility this
limit was left as is, instead we have to silently reduce
the value by one.

Reviewed-by: Viktor Dukhovni <viktor@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/27870)

diff --git a/crypto/dh/dh_key.c b/crypto/dh/dh_key.c
index fab29ff265ed18a7e783b436f9054d78e9bc845b..00b25f882cf07f69aa48038af91578028ab4b914 100644 (file)
--- a/crypto/dh/dh_key.c
+++ b/crypto/dh/dh_key.c
@@ -267,7 +267,7 @@ static int generate_key(DH *dh)
     int ok = 0;
     int generate_new_key = 0;
 #ifndef FIPS_MODULE
-    unsigned l;
+    int l;
 #endif
     BN_CTX *ctx = NULL;
     BIGNUM *pub_key = NULL, *priv_key = NULL;
@@ -327,11 +327,13 @@ static int generate_key(DH *dh)
                 goto err;
 #else
             if (dh->params.q == NULL) {
-                /* secret exponent length, must satisfy 2^(l-1) <= p */
-                if (dh->length != 0
-                    && dh->length >= BN_num_bits(dh->params.p))
+                /* secret exponent length, must satisfy 2^l < (p-1)/2 */
+                l = BN_num_bits(dh->params.p);
+                if (dh->length >= l)
                     goto err;
-                l = dh->length ? dh->length : BN_num_bits(dh->params.p) - 1;
+                l -= 2;
+                if (dh->length != 0 && dh->length < l)
+                    l = dh->length;
                 if (!BN_priv_rand_ex(priv_key, l, BN_RAND_TOP_ONE,
                                      BN_RAND_BOTTOM_ANY, 0, ctx))
                     goto err;