Daniel Stenberg [Thu, 27 Jun 2024 14:37:25 +0000 (16:37 +0200)]
KNOWN_BUGS: three new bugs
These have lingered in the issue tracker for a long time without action.
We don't expect any fixes in the near term either. Move them to the
KNOWN_BUGS document.
Viktor Szakats [Thu, 27 Jun 2024 00:38:38 +0000 (02:38 +0200)]
CI: add whitespace checker
Fix issues detected.
Also:
- One of the `.vc` files used LF EOLs, while the other didn't.
Make that one also use LF EOLs, as this is apparently supported by
`nmake`.
- Drop `.dsw` and `.btn` types from `.gitattributes`.
The repository doesn't use them.
- Sync section order with the rest of files in
`tests/certs/EdelCurlRoot-ca.prm`.
- Indent/align `.prm` and `.pem` files.
- Delete dummy `[something]` section from `.prm` and `.pem` files.
Mental note:
MSVC `.sln` files seem to accept spaces for indentation and also support
LF line-endings. I cannot test this and I don't know what's more
convenient when updating them, so left them as-is, with specific
exclusions.
Stefan Eissing [Wed, 19 Jun 2024 10:40:06 +0000 (12:40 +0200)]
lib: graceful connection shutdown
When libcurl discards a connection there are two phases this may go
through: "shutdown" and "closing". If a connection is aborted, the
shutdown phase is skipped and it is closed right away.
The connection filters attached to the connection implement the phases
in their `do_shutdown()` and `do_close()` callbacks. Filters carry now a
`shutdown` flags next to `connected` to keep track of the shutdown
operation.
Filters are shut down from top to bottom. If a filter is not connected,
its shutdown is skipped. Notable filters that *do* something during
shutdown are HTTP/2 and TLS. HTTP/2 sends the GOAWAY frame. TLS sends
its close notify and expects to receive a close notify from the server.
As sends and receives may EAGAIN on the network, a shutdown is often not
successful right away and needs to poll the connection's socket(s). To
facilitate this, such connections are placed on a new shutdown list
inside the connection cache.
Since managing this list requires the cooperation of a multi handle,
only the connection cache belonging to a multi handle is used. If a
connection was in another cache when being discarded, it is removed
there and added to the multi's cache. If no multi handle is available at
that time, the connection is shutdown and closed in a one-time,
best-effort attempt.
When a multi handle is destroyed, all connection still on the shutdown
list are discarded with a final shutdown attempt and close. In curl
debug builds, the environment variable `CURL_GRACEFUL_SHUTDOWN` can be
set to make this graceful with a timeout in milliseconds given by the
variable.
The shutdown list is limited to the max number of connections configured
for a multi cache. Set via CURLMOPT_MAX_TOTAL_CONNECTIONS. When the
limit is reached, the oldest connection on the shutdown list is
discarded.
- In multi_wait() and multi_waitfds(), collect all connection caches
involved (each transfer might carry its own) into a temporary list.
Let each connection cache on the list contribute sockets and
POLLIN/OUT events it's connections are waiting for.
- in multi_perform() collect the connection caches the same way and let
them peform their maintenance. This will make another non-blocking
attempt to shutdown all connections on its shutdown list.
- for event based multis (multi->socket_cb set), add the sockets and
their poll events via the callback. When `multi_socket()` is invoked
for a socket not known by an active transfer, forward this to the
multi's cache for processing. On closing a connection, remove its
socket(s) via the callback.
TLS connection filters MUST NOT send close nofity messages in their
`do_close()` implementation. The reason is that a TLS close notify
signals a success. When a connection is aborted and skips its shutdown
phase, the server needs to see a missing close notify to detect
something has gone wrong.
A graceful shutdown of FTP's data connection is performed implicitly
before regarding the upload/download as complete and continuing on the
control connection. For FTP without TLS, there is just the socket close
happening. But with TLS, the sent/received close notify signals that the
transfer is complete and healthy. Servers like `vsftpd` verify that and
reject uploads without a TLS close notify.
- added test_19_* for shutdown related tests
- test_19_01 and test_19_02 test for TCP RST packets
which happen without a graceful shutdown and should
no longer appear otherwise.
- add test_19_03 for handling shutdowns by the server
- add test_19_04 for handling shutdowns by curl
- add test_19_05 for event based shutdowny by server
- add test_30_06/07 and test_31_06/07 for shutdown checks
on FTP up- and downloads.
Daniel Stenberg [Tue, 25 Jun 2024 11:37:51 +0000 (13:37 +0200)]
managen: output tabs for each 8 leading spaces
This replacing of eight leading spaces into tabs was already done for
the embedded uncompressed version in tool_hugehelp.c so it does not save
anything there. But the gzip compressed version ends up almost 2K
smaller.
Stefan Eissing [Tue, 25 Jun 2024 09:35:48 +0000 (11:35 +0200)]
multi: multi_getsock(), check correct socket
- in phase CONNECTING/TUNNELING/PROTOCONNECT, retrieve
the socket from the connection filters and do not rely
on `conn->sockfd` being already set by the transfer.
- this applies to the default behaviour, a protocol handler
may override this via its callbacks.
- add a warning message in multi_getsock() when the transfer
is expected to have something in its pollset, but instead
it is empty.
Reported-by: saurabhsingh-dev on github
Fixes #13998
Closes #14011
Alex Snast [Mon, 24 Jun 2024 21:28:23 +0000 (14:28 -0700)]
wolfssl: assume key_file equal to clientcert in the absence of key_file
When user sets CURLOPT_SSLCERT but leaves CURLOPT_SSLKEY unset assume
the path passed in CURLOPT_SSLCERT holds the ssl key which is what we do
in openssl implementation.
Daniel Stenberg [Mon, 24 Jun 2024 11:57:02 +0000 (13:57 +0200)]
managen: "added in" fixes
- up the limit: remove all mentions of 7.60 or earlier from manpage
7.60 is 6 years old now.
- warn on "broken" added in lines, as they avoid detection
- fixup added in markup in a few curldown files
Daniel Stenberg [Mon, 24 Jun 2024 08:53:29 +0000 (10:53 +0200)]
managen: cleanups to generate nicer-looking output
- output "see also" last
- when there are multiple mutex items, use commas between all of them
except the last.
- call them mututally exclusive WITH not TO other options.
- remove trailing space from added in, add newline prefix
- smoother language for requires
Daniel Stenberg [Sun, 23 Jun 2024 14:05:44 +0000 (16:05 +0200)]
TODO: -h option
Support "curl -h --insecure" etc to output the manpage section for the
--insecure command line option in the terminal. Should be possible to
work with either long or short versions of command line options.
Daniel Stenberg [Wed, 19 Jun 2024 09:47:26 +0000 (11:47 +0200)]
VULN-DISCLOSURE-POLICY: NULL dereferences and crashes
If a malicious server can trigger a NULL dereference in curl or
otherwise cause curl to crash (and nothing worse), chances are big that
we do not consider that a security problem.
Yedaya Katsman [Fri, 14 Jun 2024 08:19:32 +0000 (11:19 +0300)]
docs: reference non deprecated libcurl options
There are a places where man pages reference deprecated CURLOPT options,
where it doesn't make sense, replace them with the reccomended
replacement option.
also remove reference to the removed mesalink TLS backend
Daniel Stenberg [Fri, 14 Jun 2024 06:46:50 +0000 (08:46 +0200)]
gnutls: pass in SNI name, not hostname when checking cert
The function we use is called 'gnutls_x509_crt_check_hostname()' but if
we pass in the hostname with a trailing dot, the check fails. If we pass
in the SNI name, which cannot have a trailing dot, it succeeds for
https://pyropus.ca./
I consider this as a flaw in GnuTLS and have submitted this issue
upstream:
https://gitlab.com/gnutls/gnutls/-/issues/1548
In order to work with old and existing GnuTLS versions, we still need
this change no matter how they view the issue or might change it in the
future.
Fixes #13428 Reported-by: Ryan Carsten Schmidt
Closes #13949
Jay Satiro [Fri, 26 Apr 2024 06:29:20 +0000 (02:29 -0400)]
tool_cb_hdr: allow etag and content-disposition for 3xx reply
- Parse etag and content-disposition headers for 3xx replies.
For example, a server may send a content-disposition filename header
with a redirect reply (3xx) but not with the final response (2xx).
Without this change curl would ignore the server's specified filename
and continue to use the filename extracted from the user-specified URL.
Prior to this change, 75d79a4 had limited etag and content-disposition
to 2xx replies only.
Tests-by: Daniel Stenberg Reported-by: Morgan Willcock
Fixes https://github.com/curl/curl/issues/13302
Closes #13484
Daniel Stenberg [Thu, 13 Jun 2024 15:02:55 +0000 (17:02 +0200)]
transfer: set CSELECT_IN if there is data pending
When aborting the transfer loop early, like when there is rate limiting
in effect, there might be buffered data already read off the socket so
the socket might not signal reability. Therefore we must set the
CSELECT_IN manually if data_pending_() suggests there might be more data
to get. This is particularly noticeable with SSH when the underlying
library has drained the socket and holds pending data in its buffer.
Reported-by: alervd on github
Fixes #13695
Closes #13943
Viktor Szakats [Thu, 13 Jun 2024 11:43:22 +0000 (13:43 +0200)]
cmake: allow SOVERSION override with `CURL_LIBCURL_SOVERSION`
Allow overriding SOVERSION with the new CMake option:
`CURL_LIBCURL_SOVERSION=ON/OFF`
For certain target platforms the shared libcurl library filename
contains the SOVERSION. This new option allows to enable/disable
this behavior manually. If set, it takes precedence over the default
setting.
Daniel Stenberg [Thu, 13 Jun 2024 14:08:19 +0000 (16:08 +0200)]
configure: use AC_MSG_WARN for TLS/experimental warning texts
- no longer warns for mbedtls
- warns for each item on individual lines
- no longer shows irrelevant TLS libraries when multiple are selected
- removes ech repetition
Stefan Eissing [Fri, 7 Jun 2024 12:38:51 +0000 (14:38 +0200)]
transfer: do not use EXPIRE_NOW while blocked
- When a transfer sets `data->state.select_bits`, it is
scheduled for rerun with EXPIRE_NOW. If such a transfer
is blocked (due to PAUSE, for example), this will lead to
a busy loop.
- multi.c: check for transfer block
- sendf.*: add Curl_xfer_is_blocked()
- sendf.*: add client reader `is_paused()` callback
- implement is_paused()` callback where needed
Patrick Monnerat [Wed, 12 Jun 2024 12:18:16 +0000 (14:18 +0200)]
os400: make it compilable again
A newly introduced use of getsockname() in the cli tool makes it require
the ascii wrapper module, which is not available outside of the library:
as the tool only uses the address family field (binary), disable
wrappers outside of libcurl.
Fix setsockopt() parameter type mismatch using a (void *) cast.
Viktor Szakats [Fri, 7 Jun 2024 22:41:24 +0000 (00:41 +0200)]
libcurl.pc: add `Requires.private`, `Requires` for static linking
- cmake: populate for dependencies.
- autotools: populate for dependencies.
(including mbedtls, though the script does not detect
mbedtls through pkgconfig. mbedtls 3.6.0 now supports it.)
Viktor Szakats [Wed, 5 Jun 2024 20:35:48 +0000 (22:35 +0200)]
cmake: bring `curl-config.cmake` closer to `FindCURL`
Set `CURL_LIBRARIES` and `CURL_INCLUDE_DIRS` variables
for compatibility with CMake's `FindCURL.cmake`:
https://github.com/Kitware/CMake/blob/b411d0146c2e06acfb0c823bb039e99f0191b611/Modules/FindCURL.cmake#L209
For dependent projects, CMake's suggestion is to replace
`CURL_LIBRARIES` with `CURL::libcurl`, and drop `CURL_INCLUDE_DIRS`.
Reported-by: Aurélien Pierre
Ref: https://curl.se/mail/lib-2024-06/0014.html
Ref: https://gitlab.kitware.com/cmake/cmake/-/issues/24580
Closes #13897
Daniel Stenberg [Wed, 12 Jun 2024 06:55:07 +0000 (08:55 +0200)]
file: separate fake headers and body with a stand-alone CRLF
Instead of bolting on the extra CRLF to the final header - as that makes
the behavior inconsistent and not as documented. The final CRLF is now
also made unconditional, just like it is for HTTP.
Stefan Eissing [Mon, 10 Jun 2024 11:32:13 +0000 (13:32 +0200)]
lib: xfer_setup and non-blocking shutdown
- clarify Curl_xfer_setup() with RECV/SEND flags and different calls for
which socket they operate on. Add a shutdown flag for secondary
sockets
- change Curl_xfer_setup() calls to new functions
- implement non-blocking connection shutdown at the end of receiving or
sending a transfer
projects / thirdparty / curl.git / log
