Fix bn_gcd code to check return value when calling BN_one()

BN_one() uses the expand function which calls malloc which may fail.
All other places that reference BN_one() check the return value.

The issue is triggered by a memory allocation failure.
Detected by PR #18355

Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Paul Dale <pauli@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/18697)

diff --git a/crypto/bn/bn_gcd.c b/crypto/bn/bn_gcd.c
index 6d709811ac05f6ac00c1afa68cd084dd39e148e3..2b42c7df97f77661ba06082cf9174aa75fa61cc7 100644 (file)
--- a/crypto/bn/bn_gcd.c
+++ b/crypto/bn/bn_gcd.c
@@ -47,7 +47,8 @@ BIGNUM *bn_mod_inverse_no_branch(BIGNUM *in,
     if (R == NULL)
         goto err;
 
-    BN_one(X);
+    if (!BN_one(X))
+        goto err;
     BN_zero(Y);
     if (BN_copy(B, a) == NULL)
         goto err;
@@ -235,7 +236,8 @@ BIGNUM *int_bn_mod_inverse(BIGNUM *in,
     if (R == NULL)
         goto err;
 
-    BN_one(X);
+    if (!BN_one(X))
+        goto err;
     BN_zero(Y);
     if (BN_copy(B, a) == NULL)
         goto err;