Pass an EVP_PKEY for SSL_SECOP_TMP_DH in the security callback

The security operation SSL_SECOP_TMP_DH is defined to take an EVP_PKEY
in the "other" parameter:

 /* Temporary DH key */
 # define SSL_SECOP_TMP_DH                (7 | SSL_SECOP_OTHER_PKEY)

In most places this is what is passed. All these places occur server side.
However there is one client side call of this security operation and it
passes a DH object instead. This is incorrect according to the
definition of SSL_SECOP_TMP_DH, and is inconsistent with all of the other
locations.

Our own default security callback, and the debug callback in the apps,
never look at this value and therefore this issue was never noticed
previously. In theory a client side application could be relying on this
behaviour and could be broken by this change. This is probably fairly
unlikely but can't be ruled out.

Reviewed-by: Paul Dale <paul.dale@oracle.com>
Reviewed-by: Tomas Mraz <tmraz@fedoraproject.org>
Reviewed-by: Ben Kaduk <kaduk@mit.edu>
(Merged from https://github.com/openssl/openssl/pull/13136)

diff --git a/ssl/statem/statem_clnt.c b/ssl/statem/statem_clnt.c
index 64e392cfbfc77219552ada4166481595474a30cd..3bf8aacfc0b87e8b139895054b7427770db961d3 100644 (file)
--- a/ssl/statem/statem_clnt.c
+++ b/ssl/statem/statem_clnt.c
@@ -2145,18 +2145,19 @@ static int tls_process_ske_dhe(SSL *s, PACKET *pkt, EVP_PKEY **pkey)
     }
     bnpub_key = NULL;
 
-    if (!ssl_security(s, SSL_SECOP_TMP_DH, DH_security_bits(dh), 0, dh)) {
-        SSLfatal(s, SSL_AD_HANDSHAKE_FAILURE, SSL_F_TLS_PROCESS_SKE_DHE,
-                 SSL_R_DH_KEY_TOO_SMALL);
-        goto err;
-    }
-
     if (EVP_PKEY_assign_DH(peer_tmp, dh) == 0) {
         SSLfatal(s, SSL_AD_INTERNAL_ERROR, SSL_F_TLS_PROCESS_SKE_DHE,
                  ERR_R_EVP_LIB);
         goto err;
     }
 
+    if (!ssl_security(s, SSL_SECOP_TMP_DH, EVP_PKEY_security_bits(peer_tmp),
+                      0, peer_tmp)) {
+        SSLfatal(s, SSL_AD_HANDSHAKE_FAILURE, SSL_F_TLS_PROCESS_SKE_DHE,
+                 SSL_R_DH_KEY_TOO_SMALL);
+        goto err;
+    }
+
     s->s3->peer_tmp = peer_tmp;
 
     /*