Disallow SSL_key_update() if there are writes pending

If an application is halfway through writing application data it should
not be allowed to attempt an SSL_key_update() operation. Instead the
SSL_write() operation should be completed.

Fixes #12485

Reviewed-by: Ben Kaduk <kaduk@mit.edu>
Reviewed-by: Tomas Mraz <tomas@openssl.org>
(Merged from https://github.com/openssl/openssl/pull/16098)

diff --git a/ssl/ssl_lib.c b/ssl/ssl_lib.c
index e1c95ddc6064e872169427237be1eba8748a73e0..ffd0a0bc6d1763b71f1648d88cc2e32922d39c0a 100644 (file)
--- a/ssl/ssl_lib.c
+++ b/ssl/ssl_lib.c
@@ -2119,6 +2119,11 @@ int SSL_key_update(SSL *s, int updatetype)
         return 0;
     }
 
+    if (RECORD_LAYER_write_pending(&s->rlayer)) {
+        SSLerr(SSL_F_SSL_KEY_UPDATE, SSL_R_BAD_WRITE_RETRY);
+        return 0;
+    }
+
     ossl_statem_set_in_init(s, 1);
     s->key_update = updatetype;
     return 1;