ssl/tls13_enc.c: Replace size_t with int and add the checks

ssl/tls13_enc.c: Replace size_t with int and add the checks

Replace the type of variables with int to avoid implicit cast when they are assigned by EVP_MD_get_size().
Moreover, add the checks to avoid integer overflow.

Fixes: 6612d87b89 ("Use the correct size for TLSv1.3 finished keys")
Fixes: 34574f193b ("Add support for TLS1.3 secret generation")
Signed-off-by: Jiasheng Jiang <jiasheng@purdue.edu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
Reviewed-by: Tom Cosgrove <tom.cosgrove@arm.com>
Reviewed-by: Neil Horman <nhorman@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/23942)