Use ASN1_BIT_STRING_get_bit to check for netscape certificate usage

The ASN_BIT_STRING object has become opaque in OpenSSL 4.0. So instead
of accessing the internal, we have to use a method now to check these
attributes.

The bit counting in ASN.1 and of this method is a bit strange and
it will count bits from the left instead of the right, so the previous
mask of 0x80 for clients is now 0 and 0x40 for server is now 1.

Change-Id: I77500d435f212a4bf42ee8cfca07d0285fe694f2
Signed-off-by: Arne Schwabe <arne@rfc2549.org>
Acked-by: Frank Lichtenheld <frank@lichtenheld.com>
Gerrit URL: https://gerrit.openvpn.net/c/openvpn/+/1587
Message-Id: <20260404072336.30014-1-gert@greenie.muc.de>
URL: https://www.mail-archive.com/openvpn-devel@lists.sourceforge.net/msg36446.html
Signed-off-by: Gert Doering <gert@greenie.muc.de>
(cherry picked from commit 8ce6f8d166acf02d6833b9f37fb5cd2ca903a4a5)

diff --git a/src/openvpn/ssl_verify_openssl.c b/src/openvpn/ssl_verify_openssl.c
index 08946cdce0d885b314d2d6149923ce39deda01fe..c70022f13817f6546ac01bbb77d27781018f4beb 100644 (file)
--- a/src/openvpn/ssl_verify_openssl.c
+++ b/src/openvpn/ssl_verify_openssl.c
@@ -609,22 +609,27 @@ x509_verify_ns_cert_type(openvpn_x509_cert_t *peer_cert, const int usage)
     if (usage == NS_CERT_CHECK_CLIENT)
     {
         /*
-         * Unfortunately, X509_check_purpose() does some weird thing that
+         * Unfortunately, X509_check_purpose() before OpenSSL 4.0 does some weird thing that
          * prevent it to take a const argument
          */
         result_t result =
             X509_check_purpose(peer_cert, X509_PURPOSE_SSL_CLIENT, 0) ? SUCCESS : FAILURE;
-
         /*
-         * old versions of OpenSSL allow us to make the less strict check we used to
-         * do. If this less strict check pass, warn user that this might not be the
-         * case when its distribution will update to OpenSSL 1.1
+         * Note that we did not check for netscape certificate type here but
+         * instead a general SSL/TLS client purpose. These nscert attributes
+         * might stop being accepted by TLS libraries in the future.
+         * Currently, OpenSSL 4.0 and aws-lc 1.9.0 still consider nscert client
+         * as acceptable.
+         *
+         * So in case that this check failed, we now check if this is caused
+         * by the check above no longer recognising nscert attributes.
          */
         if (result == FAILURE)
         {
             ASN1_BIT_STRING *ns;
             ns = X509_get_ext_d2i(peer_cert, NID_netscape_cert_type, NULL, NULL);
-            result = (ns && ASN1_STRING_length(ns) > 0 && (ASN1_STRING_get0_data(ns)[0] & NS_SSL_CLIENT)) ? SUCCESS : FAILURE;
+            // bit 0 is to check if certificate is the client certificate
+            result = ASN1_BIT_STRING_get_bit(ns, 0) ? SUCCESS : FAILURE;
             if (result == SUCCESS)
             {
                 msg(M_WARN, "X509: Certificate is a client certificate yet it's purpose "
@@ -637,22 +642,27 @@ x509_verify_ns_cert_type(openvpn_x509_cert_t *peer_cert, const int usage)
     if (usage == NS_CERT_CHECK_SERVER)
     {
         /*
-         * Unfortunately, X509_check_purpose() does some weird thing that
+         * Unfortunately, X509_check_purpose() before OpenSSL 4.0 does some weird thing that
          * prevent it to take a const argument
          */
         result_t result =
             X509_check_purpose(peer_cert, X509_PURPOSE_SSL_SERVER, 0) ? SUCCESS : FAILURE;
 
         /*
-         * old versions of OpenSSL allow us to make the less strict check we used to
-         * do. If this less strict check pass, warn user that this might not be the
-         * case when its distribution will update to OpenSSL 1.1
+         * Note that we did not check for netscape certificate type here but
+         * instead a general SSL/TLS server purpose. These nscert attributes
+         * might stop being accepted by TLS libraries in the future.
+         * Currently, OpenSSL 4.0 and aws-lc 1.9.0 still consider nscert server
+         * as acceptable.
+         *
+         * So in case that this check failed, we now check if this is caused
+         * by the check above no longer recognising nscert attributes.
          */
         if (result == FAILURE)
         {
-            ASN1_BIT_STRING *ns;
-            ns = X509_get_ext_d2i(peer_cert, NID_netscape_cert_type, NULL, NULL);
-            result = (ns && ASN1_STRING_length(ns) > 0 && (ASN1_STRING_get0_data(ns)[0] & NS_SSL_SERVER)) ? SUCCESS : FAILURE;
+            ASN1_BIT_STRING *ns = X509_get_ext_d2i(peer_cert, NID_netscape_cert_type, NULL, NULL);
+            // Server bit is 1 for ASN1_BIT_STRING_get_bit
+            result = ASN1_BIT_STRING_get_bit(ns, 1) ? SUCCESS : FAILURE;
             if (result == SUCCESS)
             {
                 msg(M_WARN, "X509: Certificate is a server certificate yet it's purpose "