Make error checks on RSA_public_decrypt() consistent

Make error checks on RSA_public_decrypt() consistent

Some are only checking for a value < 0, some for <= 0, some for == 0, etc.
The documentation tells us that -1 is returned on error, so at least the
== 0 ones are wrong. In general, the return values are checked
inconsistently. This patch makes the return value checks consistent to
the form that seems to occur most.

Reviewed-by: Paul Dale <ppzgs1@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/28306)