Fix false success on zero BIO write

Fix false success on zero BIO write

The TLS record layer treated BIO_write returning 0 as success unless
the BIO retry flag was set. For a write with data pending in the write
buffer, this could report success to the application while leaving record
data pending internally.

Capture the pending buffer length before the transport write. If a
positive length write returns zero, return retry when the BIO retry flag
is set and fatal otherwise. Do not queue an SSL reason for the fatal
zero-without-retry case: a custom BIO can return zero without retry
without this implying an SSL library or protocol error, so SSL_get_error()
reports SSL_ERROR_SYSCALL with an empty error queue.

Preserve the KTLS empty fragment case where the pending length is already
zero and a zero byte write is expected to succeed.

For DTLS, the same zero return without retry condition is now treated as
fatal; the failed buffer is still dropped in the existing DTLS write error
path. This intentionally tightens the previous behavior, which could treat
the zero return as success.

Add regression coverage with a custom BIO that returns 0 without setting
retry for a positive length write.

Fixes #31009

Reviewed-by: Jakub Zelenka <jakub.zelenka@openssl.foundation>
Reviewed-by: Matt Caswell <matt@openssl.foundation>
Reviewed-by: Andrew Dinh <andrewd@openssl.org>
MergeDate: Mon Aug  3 12:37:56 2026
(Merged from https://github.com/openssl/openssl/pull/31021)