crypto/bio/bio_print.c: avoid signed int overow in padlen calculation in fmtstr

In a highly unlikely situation of str being longer than INT_MAX,
a signed integer overflow in padlen calculation can be triggered.
Avoid it by reworking the check for the need of padlen calculation.

Fixes: 230c691a5218 "Fix fmtstr for BIO_printf() et al"
Signed-off-by: Eugene Syromiatnikov <esyr@openssl.org>
Reviewed-by: Neil Horman <nhorman@openssl.org>
Reviewed-by: Saša Nedvědický <sashan@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/28502)

diff --git a/crypto/bio/bio_print.c b/crypto/bio/bio_print.c
index c2db1fa6c11cb85683e0c5f4fcc0648d1fb8ec7c..ddc5bc6deee354b93c4891d719dcc8bcae1abf0b 100644 (file)
--- a/crypto/bio/bio_print.c
+++ b/crypto/bio/bio_print.c
@@ -476,9 +476,11 @@ fmtstr(struct pr_desc *desc, const char *value, int flags, int min, int max)
 
     strln = OPENSSL_strnlen(value, max < 0 ? SIZE_MAX : (size_t)max);
 
-    padlen = (int)(min - strln);
-    if (min < 0 || padlen < 0)
-        padlen = 0;
+    if (min >= 0 && strln < INT_MAX) {
+        padlen = min - (int)strln;
+        if (padlen < 0)
+            padlen = 0;
+    }
     if (max >= 0) {
         /*
          * Calculate the maximum output including padding.