Check ASN1_item_ndef_i2d() return value.

Return an error instead of trying to malloc a negative number.
The other usage in this file already had a similar check, and the caller
should have put an entry on the error stack already.

Note that we only check the initial calls to obtain the encoded length,
and assume that the follow-up call to actually encode to the allocated
storage will succeed if the first one did.

Fixes: #14177
Reviewed-by: Shane Lontis <shane.lontis@oracle.com>
(Merged from https://github.com/openssl/openssl/pull/14308)

(cherry picked from commit 90b4247cc5dca58cee9da5f6975bb38fd200100a)

diff --git a/crypto/asn1/bio_ndef.c b/crypto/asn1/bio_ndef.c
index 6222c99074de8d5394ed2c1bdd6732548ff98b12..5642262719478ef8c16899551c49a1847352e921 100644 (file)
--- a/crypto/asn1/bio_ndef.c
+++ b/crypto/asn1/bio_ndef.c
@@ -113,6 +113,8 @@ static int ndef_prefix(BIO *b, unsigned char **pbuf, int *plen, void *parg)
     ndef_aux = *(NDEF_SUPPORT **)parg;
 
     derlen = ASN1_item_ndef_i2d(ndef_aux->val, NULL, ndef_aux->it);
+    if (derlen < 0)
+        return 0;
     if ((p = OPENSSL_malloc(derlen)) == NULL) {
         ASN1err(ASN1_F_NDEF_PREFIX, ERR_R_MALLOC_FAILURE);
         return 0;