Update size check condition for probable primes to align with FIPS 186-5.

Fixes #28540

In FIPS 186-4 (Table B.1), the upper bounds for probable primes p and q were
defined using strict inequalities. In FIPS 186-5, Table A.1 replaces the
previous Table B.1 (dropping the nlen=1024 case) and revises the upper bound
condition to use weak inequalities.

CLA: trivial

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/28540)

diff --git a/crypto/bn/bn_rsa_fips186_4.c b/crypto/bn/bn_rsa_fips186_4.c
index c967ca962978a9968b61cc0684b504d6dd99b7ad..e42de63a352b51864a6304a81c54b51bbc524041 100644 (file)
--- a/crypto/bn/bn_rsa_fips186_4.c
+++ b/crypto/bn/bn_rsa_fips186_4.c
@@ -228,8 +228,8 @@ int ossl_bn_rsa_fips186_4_gen_prob_primes(BIGNUM *p, BIGNUM *Xpout,
     if (!bn_rsa_fips186_4_find_aux_prob_prime(Xp1i, p1i, ctx, rounds, cb)
             || !bn_rsa_fips186_4_find_aux_prob_prime(Xp2i, p2i, ctx, rounds, cb))
         goto err;
-    /* (Table B.1) auxiliary prime Max length check */
-    if ((BN_num_bits(p1i) + BN_num_bits(p2i)) >=
+    /* (FIPS 186-5 Table A.1) auxiliary prime Max length check */
+    if ((BN_num_bits(p1i) + BN_num_bits(p2i)) >
             bn_rsa_fips186_5_aux_prime_max_sum_size_for_prob_primes(nlen))
         goto err;
     /* (Steps 4.3/5.3) - generate prime */