Fix return value checking of BN_check_prime invocations

Fix return value checking of BN_check_prime invocations

Negative return value indicates an error so we bail out.

Reviewed-by: Paul Dale <pauli@openssl.org>
Reviewed-by: Kurt Roeckx <kurt@roeckx.be>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16975)

(cherry picked from commit 680827a15f12c3b37a6335fcb992555cf300730e)