ssl/ssl_lib.c: Add the check before cast from int to unsigned

ssl/ssl_lib.c: Add the check before cast from int to unsigned

Add the check before cast from int to unsigned to avoid integer overflow since EVP_MD_get_size() may return negative numbers.

Fixes: 919ba00942 ("DANE support structures, constructructors and accessors")
Signed-off-by: Jiasheng Jiang <jiasheng@purdue.edu>
Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
(Merged from https://github.com/openssl/openssl/pull/23940)