crypto/bio/bio_print.c: improve the precision handling in fmtint

Per [1]:

    * A negative precision is taken as if the precision were omitted.
    * The default precision is 1.
    * For d, i, o, u, x, and X conversion specifiers, if a precision
      is specified, the '0' flag shall be ignored.

[1] https://pubs.opengroup.org/onlinepubs/9699919799/functions/printf.html

Signed-off-by: Eugene Syromiatnikov <esyr@openssl.org>
Reviewed-by: Saša Nedvědický <sashan@openssl.org>
Reviewed-by: Neil Horman <nhorman@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/28177)

diff --git a/crypto/bio/bio_print.c b/crypto/bio/bio_print.c
index 5f2afdde4cd2a916a3229161983d0e4180b1c52f..e1f7ddc641db51d33fe0325ed5bc9e0bdeb8f2e6 100644 (file)
--- a/crypto/bio/bio_print.c
+++ b/crypto/bio/bio_print.c
@@ -528,8 +528,16 @@ fmtint(struct pr_desc *desc,
     int zpadlen = 0;
     int caps = 0;
 
-    if (max < 0)
-        max = 0;
+    if (max < 0) {
+        /* A negative precision is taken as if the precision were omitted. */
+        max = 1;
+    } else {
+        /*
+         * If a precision is given with an integer conversion,
+         * the 0 flag is ignored.
+         */
+        flags &= ~DP_F_ZERO;
+    }
     uvalue = value;
     if (!(flags & DP_F_UNSIGNED)) {
         if (value < 0) {