signature/rsa_sig.c: Add checks for the EVP_MD_get_size()

signature/rsa_sig.c: Add checks for the EVP_MD_get_size()

Add checks for the EVP_MD_get_size() to avoid integer overflow and then explicitly cast from int to size_t.

Fixes: 6f4b766315 ("PROV: add RSA signature implementation")
Signed-off-by: Jiasheng Jiang <jiasheng@purdue.edu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Todd Short <todd.short@me.com>
Reviewed-by: Neil Horman <nhorman@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23949)