crypto/pkcs7/pk7_lib.c: fix return value in PKCS7_set_digest()

Return 0 when the passed object to PKCS7_set_digest() is not of type PKCS7
digest.

Fixes: c5a55463892d "Add support for digested data PKCS#7 type."
Resolves: https://github.com/openssl/openssl/issues/31551

Reviewed-by: Matt Caswell <matt@openssl.foundation>
Reviewed-by: Norbert Pocs <norbertp@openssl.org>
Reviewed-by: Eugene Syromiatnikov <esyr@openssl.org>
Reviewed-by: Tim Hudson <tjh@openssl.org>
MergeDate: Mon Jun 22 08:17:32 2026
(Merged from https://github.com/openssl/openssl/pull/31559)

diff --git a/crypto/pkcs7/pk7_lib.c b/crypto/pkcs7/pk7_lib.c
index 0a1d0f61a54874cff878cf22de00e988c0fa0c86..a9640769cf008ffee99b57c88af96c0a62604e1e 100644 (file)
--- a/crypto/pkcs7/pk7_lib.c
+++ b/crypto/pkcs7/pk7_lib.c
@@ -537,7 +537,7 @@ int PKCS7_set_digest(PKCS7 *p7, const EVP_MD *md)
     }
 
     ERR_raise(ERR_LIB_PKCS7, PKCS7_R_WRONG_CONTENT_TYPE);
-    return 1;
+    return 0;
 }
 
 STACK_OF(PKCS7_SIGNER_INFO) *PKCS7_get_signer_info(PKCS7 *p7)