rsa-pss: add tests checking for SHAKE usage in RSA-PSS

FIPS 186-5, RFC 8692, RFC 8702 all agree and specify that Shake shall
be used directly as MGF (not as a hash in MGF1). Add tests that try to
specify shake hash as MGF1 to ensure that fails.

Separately the above standards specify how to use SHAKE as a message
digest with either fixed or minimum output lengths. However, currently
shake is not part of allowed hashes.

Note that rsa_setup_md()/rsa_setup_mgf1_md() call
ossl_digest_rsa_sign_get_md_nid() ->
ossl_digest_get_approved_nid_with_sha1() ->
ossl_digest_get_approved_nid() which only contain sha1/sha2/sha3
digests without XOF.

The digest test case will need to be replace if/when shake with
minimum output lengths is added to ossl_digest_get_approved_nid().

Reviewed-by: Dmitry Belyavskiy <beldmit@gmail.com>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/24387)

diff --git a/providers/common/digest_to_nid.c b/providers/common/digest_to_nid.c
index 49af04ad2a210c4e6216563ae71bbcc5926309a0..ff347bacd5fe94a53408e0356f1a9b960a16777b 100644 (file)
--- a/providers/common/digest_to_nid.c
+++ b/providers/common/digest_to_nid.c
@@ -39,6 +39,7 @@ int ossl_digest_md_to_nid(const EVP_MD *md, const OSSL_ITEM *it, size_t it_len)
  */
 int ossl_digest_get_approved_nid(const EVP_MD *md)
 {
+    /* TODO: FIPS 180-5 RFC 8692 RFC 8702 allow SHAKE */
     static const OSSL_ITEM name_to_nid[] = {
         { NID_sha1,      OSSL_DIGEST_NAME_SHA1      },
         { NID_sha224,    OSSL_DIGEST_NAME_SHA2_224  },

diff --git a/test/recipes/30-test_evp_data/evppkey_rsa_common.txt b/test/recipes/30-test_evp_data/evppkey_rsa_common.txt
index f5398f7342272f692be1e30aba5d6303bf10f0e6..0036acdb0f9f2079212b13a74df72092d89a389c 100644 (file)
--- a/test/recipes/30-test_evp_data/evppkey_rsa_common.txt
+++ b/test/recipes/30-test_evp_data/evppkey_rsa_common.txt
@@ -989,6 +989,43 @@ Verify = RSA-PSS-BAD2
 Result = KEYOP_INIT_ERROR
 Reason = invalid salt length
 
+# Test sign with MGF1 using shake fails
+Sign = RSA-PSS
+Ctrl = digest:sha256
+Ctrl = rsa_padding_mode:pss
+Ctrl = rsa_mgf1_md:shake256
+Input = ""
+Output = ""
+Result = PKEY_CTRL_ERROR
+
+# Test verify with MGF1 using shake fails
+Verify = RSA-PSS
+Ctrl = digest:sha256
+Ctrl = rsa_padding_mode:pss
+Ctrl = rsa_mgf1_md:shake256
+Input = ""
+Output = ""
+Result = PKEY_CTRL_ERROR
+
+# Test sign with digest using shake fails. Remove once FIPS 186-5 /
+# RFC-8702 / RFC-8692 SHAKE digest implemented
+Sign = RSA-PSS
+Ctrl = digest:shake256
+Ctrl = rsa_padding_mode:pss
+Ctrl = rsa_mgf1_md:sha256
+Input = ""
+Output = ""
+Result = PKEY_CTRL_ERROR
+
+# Test sign with digest using shake fails. Remove once FIPS 186-5 /
+# RFC-8702 / RFC-8692 SHAKE digest implemented
+Verify = RSA-PSS
+Ctrl = digest:shake256
+Ctrl = rsa_padding_mode:pss
+Ctrl = rsa_mgf1_md:sha256
+Input = ""
+Output = ""
+Result = PKEY_CTRL_ERROR
 
 # Additional RSA-PSS and RSA-OAEP tests converted from
 # ftp://ftp.rsasecurity.com/pub/pkcs/pkcs-1/pkcs-1v2-1-vec.zip