Fix regression when X509_V_FLAG_CRL_CHECK_ALL is set, while X509_V_FLAG_CRL_CHECK...

Fixes #28758

When X509_V_FLAG_CRL_CHECK is not set, the man pages document that X509_V_FLAG_CRL_CHECK_ALL is ignored.
Prior to 3.6.0, this was indeed the case.

In 3.6.0, the behavior changed, and setting X509_V_FLAG_CRL_CHECK_ALL began to imply X509_V_FLAG_CRL_CHECK.
This unfortunately breaks the majority of ruby installations, which relied on the documented behavior.

For consistency, this commit applies the same logic to the new X509_V_FLAG_OCSP_RESP_CHECK and X509_V_FLAG_OCSP_RESP_CHECK_ALL flags,
which are still undocumented as of 3.6.0.

All existing tests continue to pass.  They also make the assumption that the xxx_CHECK_ALL flags are irrelevant unless xxx_CHECK is set.
We could add a new test for this regression.  I'll leave that to another commit.

CLA: trivial

Reviewed-by: Nikola Pajkovsky <nikolap@openssl.org>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/28797)

diff --git a/crypto/x509/x509_vfy.c b/crypto/x509/x509_vfy.c
index 9f7a12f3bd68032398e3dd606468d01d3a114650..7729fb80338204951b3ce0edbcd41360527c9d07 100644 (file)
--- a/crypto/x509/x509_vfy.c
+++ b/crypto/x509/x509_vfy.c
@@ -192,8 +192,7 @@ static int verify_cb_crl(X509_STORE_CTX *ctx, int err)
 /*
  * Inform the verify callback of an error, OCSP-specific variant.
  * It is called also on OCSP response errors, if the
- * X509_V_FLAG_OCSP_RESP_CHECK or X509_V_FLAG_OCSP_RESP_CHECK_ALL flag
- * is set.
+ * X509_V_FLAG_OCSP_RESP_CHECK flag is set.
  * Here, the error depth and certificate are already set, we just specify
  * the error number.
  *
@@ -1063,10 +1062,10 @@ trusted:
 static int check_revocation(X509_STORE_CTX *ctx)
 {
     int i = 0, last = 0, ok = 0;
-    int crl_check_enabled = (ctx->param->flags & (X509_V_FLAG_CRL_CHECK | X509_V_FLAG_CRL_CHECK_ALL)) != 0;
-    int crl_check_all_enabled = (ctx->param->flags & X509_V_FLAG_CRL_CHECK_ALL) != 0;
-    int ocsp_check_enabled = (ctx->param->flags & (X509_V_FLAG_OCSP_RESP_CHECK | X509_V_FLAG_OCSP_RESP_CHECK_ALL)) != 0;
-    int ocsp_check_all_enabled = (ctx->param->flags & X509_V_FLAG_OCSP_RESP_CHECK_ALL) != 0;
+    int crl_check_enabled = (ctx->param->flags & X509_V_FLAG_CRL_CHECK) != 0;
+    int crl_check_all_enabled = crl_check_enabled && (ctx->param->flags & X509_V_FLAG_CRL_CHECK_ALL) != 0;
+    int ocsp_check_enabled = (ctx->param->flags & X509_V_FLAG_OCSP_RESP_CHECK) != 0;
+    int ocsp_check_all_enabled = ocsp_check_enabled && (ctx->param->flags & X509_V_FLAG_OCSP_RESP_CHECK_ALL) != 0;
 
     if (!crl_check_enabled && !ocsp_check_enabled)
         return 1;