nettle/pk: check RSA key coprimality in verify_params

Previously, gnutls_privkey_verify_params has overlooked
the scenario of p and q not being co-prime,
and proceeded with undefined behaviour that was extremely likely
to error out in practice anyway.
Now it returns GNUTLS_E_PK_INVALID_PRIVKEY in this case.

Reported-by: Kamil Frankowicz <kamil.frankowicz@cert.pl>
Related: #1818
Signed-off-by: Alexander Sosedkin <asosedkin@redhat.com>

diff --git a/lib/nettle/pk.c b/lib/nettle/pk.c
index 4047df3776958edcab0b82b9f92691cc7c571b17..e8b36f5fa935310e94ec60854bc12c69d9a46e91 100644 (file)
--- a/lib/nettle/pk.c
+++ b/lib/nettle/pk.c
@@ -4434,8 +4434,11 @@ static int wrap_nettle_pk_verify_priv_params(gnutls_pk_algorithm_t algo,
                        goto rsa_cleanup;
                }
 
-               mpz_invert(TOMPZ(t1), TOMPZ(params->params[RSA_PRIME2]),
-                          TOMPZ(params->params[RSA_PRIME1]));
+               if (!mpz_invert(TOMPZ(t1), TOMPZ(params->params[RSA_PRIME2]),
+                               TOMPZ(params->params[RSA_PRIME1]))) {
+                       ret = gnutls_assert_val(GNUTLS_E_PK_INVALID_PRIVKEY);
+                       goto rsa_cleanup;
+               }
                if (_gnutls_mpi_cmp(t1, params->params[RSA_COEF]) != 0) {
                        ret = gnutls_assert_val(GNUTLS_E_ILLEGAL_PARAMETER);
                        goto rsa_cleanup;