Fix overflow in EVP_EncodeFinal

https://scan5.scan.coverity.com/#/project-view/60762/10222?selectedIssue=1677829

With recent changes, evp_encodeblock_int may return a negative value,
which EVP_EncodeFinal does not anticipate.  As the latter sets out[ret]
to "\0" where ret is the return value of evp_encodeblock_int, we may
underflow the array index and access invalid memory locations.

Only update the output buffer if the return value is greater or equal to
zero.

Reviewed-by: Saša Nedvědický <sashan@openssl.org>
Reviewed-by: Nikola Pajkovsky <nikolap@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/29525)

diff --git a/crypto/evp/encode.c b/crypto/evp/encode.c
index bbd36c0820b671d8f7dd14336842a2b2f86e9261..dd5992d09e6c2f4fa3bae0d72cbae03b34641194 100644 (file)
--- a/crypto/evp/encode.c
+++ b/crypto/evp/encode.c
@@ -457,10 +457,12 @@ void EVP_EncodeFinal(EVP_ENCODE_CTX *ctx, unsigned char *out, int *outl)
     if (ctx->num != 0) {
         ret = evp_encodeblock_int(ctx, out, ctx->enc_data, ctx->num,
             &wrap_cnt);
-        if ((ctx->flags & EVP_ENCODE_CTX_NO_NEWLINES) == 0)
-            out[ret++] = '\n';
-        out[ret] = '\0';
-        ctx->num = 0;
+        if (ossl_assert(ret >= 0)) {
+            if ((ctx->flags & EVP_ENCODE_CTX_NO_NEWLINES) == 0)
+                out[ret++] = '\n';
+            out[ret] = '\0';
+            ctx->num = 0;
+        }
     }
     *outl = ret;
 }