Yedaya Katsman [Fri, 14 Jun 2024 08:19:32 +0000 (11:19 +0300)]
docs: reference non deprecated libcurl options
There are a places where man pages reference deprecated CURLOPT options,
where it doesn't make sense, replace them with the reccomended
replacement option.
also remove reference to the removed mesalink TLS backend
Daniel Stenberg [Fri, 14 Jun 2024 06:46:50 +0000 (08:46 +0200)]
gnutls: pass in SNI name, not hostname when checking cert
The function we use is called 'gnutls_x509_crt_check_hostname()' but if
we pass in the hostname with a trailing dot, the check fails. If we pass
in the SNI name, which cannot have a trailing dot, it succeeds for
https://pyropus.ca./
I consider this as a flaw in GnuTLS and have submitted this issue
upstream:
https://gitlab.com/gnutls/gnutls/-/issues/1548
In order to work with old and existing GnuTLS versions, we still need
this change no matter how they view the issue or might change it in the
future.
Fixes #13428 Reported-by: Ryan Carsten Schmidt
Closes #13949
Jay Satiro [Fri, 26 Apr 2024 06:29:20 +0000 (02:29 -0400)]
tool_cb_hdr: allow etag and content-disposition for 3xx reply
- Parse etag and content-disposition headers for 3xx replies.
For example, a server may send a content-disposition filename header
with a redirect reply (3xx) but not with the final response (2xx).
Without this change curl would ignore the server's specified filename
and continue to use the filename extracted from the user-specified URL.
Prior to this change, 75d79a4 had limited etag and content-disposition
to 2xx replies only.
Tests-by: Daniel Stenberg Reported-by: Morgan Willcock
Fixes https://github.com/curl/curl/issues/13302
Closes #13484
Daniel Stenberg [Thu, 13 Jun 2024 15:02:55 +0000 (17:02 +0200)]
transfer: set CSELECT_IN if there is data pending
When aborting the transfer loop early, like when there is rate limiting
in effect, there might be buffered data already read off the socket so
the socket might not signal reability. Therefore we must set the
CSELECT_IN manually if data_pending_() suggests there might be more data
to get. This is particularly noticeable with SSH when the underlying
library has drained the socket and holds pending data in its buffer.
Reported-by: alervd on github
Fixes #13695
Closes #13943
Viktor Szakats [Thu, 13 Jun 2024 11:43:22 +0000 (13:43 +0200)]
cmake: allow SOVERSION override with `CURL_LIBCURL_SOVERSION`
Allow overriding SOVERSION with the new CMake option:
`CURL_LIBCURL_SOVERSION=ON/OFF`
For certain target platforms the shared libcurl library filename
contains the SOVERSION. This new option allows to enable/disable
this behavior manually. If set, it takes precedence over the default
setting.
Daniel Stenberg [Thu, 13 Jun 2024 14:08:19 +0000 (16:08 +0200)]
configure: use AC_MSG_WARN for TLS/experimental warning texts
- no longer warns for mbedtls
- warns for each item on individual lines
- no longer shows irrelevant TLS libraries when multiple are selected
- removes ech repetition
Stefan Eissing [Fri, 7 Jun 2024 12:38:51 +0000 (14:38 +0200)]
transfer: do not use EXPIRE_NOW while blocked
- When a transfer sets `data->state.select_bits`, it is
scheduled for rerun with EXPIRE_NOW. If such a transfer
is blocked (due to PAUSE, for example), this will lead to
a busy loop.
- multi.c: check for transfer block
- sendf.*: add Curl_xfer_is_blocked()
- sendf.*: add client reader `is_paused()` callback
- implement is_paused()` callback where needed
Patrick Monnerat [Wed, 12 Jun 2024 12:18:16 +0000 (14:18 +0200)]
os400: make it compilable again
A newly introduced use of getsockname() in the cli tool makes it require
the ascii wrapper module, which is not available outside of the library:
as the tool only uses the address family field (binary), disable
wrappers outside of libcurl.
Fix setsockopt() parameter type mismatch using a (void *) cast.
Viktor Szakats [Fri, 7 Jun 2024 22:41:24 +0000 (00:41 +0200)]
libcurl.pc: add `Requires.private`, `Requires` for static linking
- cmake: populate for dependencies.
- autotools: populate for dependencies.
(including mbedtls, though the script does not detect
mbedtls through pkgconfig. mbedtls 3.6.0 now supports it.)
Viktor Szakats [Wed, 5 Jun 2024 20:35:48 +0000 (22:35 +0200)]
cmake: bring `curl-config.cmake` closer to `FindCURL`
Set `CURL_LIBRARIES` and `CURL_INCLUDE_DIRS` variables
for compatibility with CMake's `FindCURL.cmake`:
https://github.com/Kitware/CMake/blob/b411d0146c2e06acfb0c823bb039e99f0191b611/Modules/FindCURL.cmake#L209
For dependent projects, CMake's suggestion is to replace
`CURL_LIBRARIES` with `CURL::libcurl`, and drop `CURL_INCLUDE_DIRS`.
Reported-by: Aurélien Pierre
Ref: https://curl.se/mail/lib-2024-06/0014.html
Ref: https://gitlab.kitware.com/cmake/cmake/-/issues/24580
Closes #13897
Daniel Stenberg [Wed, 12 Jun 2024 06:55:07 +0000 (08:55 +0200)]
file: separate fake headers and body with a stand-alone CRLF
Instead of bolting on the extra CRLF to the final header - as that makes
the behavior inconsistent and not as documented. The final CRLF is now
also made unconditional, just like it is for HTTP.
Stefan Eissing [Mon, 10 Jun 2024 11:32:13 +0000 (13:32 +0200)]
lib: xfer_setup and non-blocking shutdown
- clarify Curl_xfer_setup() with RECV/SEND flags and different calls for
which socket they operate on. Add a shutdown flag for secondary
sockets
- change Curl_xfer_setup() calls to new functions
- implement non-blocking connection shutdown at the end of receiving or
sending a transfer
Daniel Stenberg [Mon, 10 Jun 2024 11:33:16 +0000 (13:33 +0200)]
tool_writeout: bsearch the variable name
As the list of variable names grows, doing a simple loop to find the
name get increasingly worse. This switches to a bsearch.
Also: do a case sensitive check for the variable name. The names have
not been documented to be case insensitive and there is no point in
having them so.
Stefan Eissing [Thu, 6 Jun 2024 10:40:38 +0000 (12:40 +0200)]
multi: prepare multi_wait() for future shutdown usage
- new struct curl_pollfds and struct curl_waitfds
- add structs and methods to init/add/cleanup an array of pollfd and
struct curl_waitfd. Use in multi_wait() and multi_waitfds() to
populate the sets for polling.
- place USE_WINSOCK WSAEventSelect() setting into a separate loop over
all collected pfds
Stefan Eissing [Fri, 7 Jun 2024 08:12:39 +0000 (10:12 +0200)]
connection: shutdown TLS (for FTP) better
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension
that enables a TCP connection to use different paths.
Multipath TCP has been used for several use cases. On smartphones, MPTCP
enables seamless handovers between cellular and Wi-Fi networks while
preserving established connections. This use-case is what pushed Apple
to use MPTCP since 2013 in multiple applications [2]. On dual-stack
hosts, Multipath TCP enables the TCP connection to automatically use the
best performing path, either IPv4 or IPv6. If one path fails, MPTCP
automatically uses the other path.
To benefit from MPTCP, both the client and the server have to support
it. Multipath TCP is a backward-compatible TCP extension that is enabled
by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...).
Multipath TCP is included in the Linux kernel since version 5.6 [3]. To
use it on Linux, an application must explicitly enable it when creating
the socket. No need to change anything else in the application.
This attached patch adds an --mptcp option which allows the creation of
an MPTCP socket instead of TCP on Linux. If Multipath TCP is not
supported on the system, an error will be reported. It is important to
note that if the end server doesn't support MPTCP, the connection will
continue after a seamless fallback to TCP.
Daniel Stenberg [Wed, 5 Jun 2024 12:08:02 +0000 (14:08 +0200)]
managen: warn on excessively long help texts
Help texts at 49 characters or longer get a warning displayed because
they make --help output uglier and we should make an effort to keep the
help texts short and succinct.
The warning is only for display, it does not break the build. That is
left for the future if necessary.
I picked 49 because the longest current text is 48.
Stefan Eissing [Tue, 4 Jun 2024 12:00:12 +0000 (14:00 +0200)]
vtls: new io_need flags for poll handling
- decouple need to recv/send from negotiation state, we need
this later in shutdown handling as well
- move ssl enums from urldata.h to vtls_int.h
- implement use of `connssl->io_need` in vtls.c. and all backends
Daniel Stenberg [Tue, 4 Jun 2024 15:00:05 +0000 (17:00 +0200)]
url: allow DoH transfers to override max connection limit
When reaching the set maximum limit of allowed connections, allow a new
connection anyway if the transfer is created for the (internal) purpose
of doing a DoH name resolve. Otherwise, unrelated "normal" transfers can
starve out new DoH requests making it impossible to name resolve for new
transfers.
Viktor Szakats [Mon, 3 Jun 2024 21:06:56 +0000 (23:06 +0200)]
windows: fix UWP builds, add GHA job
Add new job to test building for UWP (aka `CURL_WINDOWS_APP`).
Fix fallouts when building for UWP:
- rand: do not use `BCryptGenRandom()`.
- cmake: disable using win32 LDAP.
- cmake: disable telnet.
- version_win32: fix code before declaration.
- schannel: disable `HAS_MANUAL_VERIFY_API`.
- schannel: disable `SSLSUPP_PINNEDPUBKEY`
and make `schannel_checksum()` a stub.
Ref: e178fbd40a896f2098278ae61e1166c88e7b31d0 #1429
- schannel: make `cert_get_name_string()` a failing stub.
- system_win32: make `Curl_win32_impersonating()` a failing stub.
- system_win32: try to fix `Curl_win32_init()` (untested).
- threads: fix to use `CreateThread()`.
- src: disable searching `PATH` for the CA bundle.
- src: disable bold text support and capability detection.
- src: disable `getfiletime()`/`setfiletime()`.
- tests: make `win32_load_system_library()` a failing stub.
- tests/server/util: make it compile.
- tests/server/sockfilt: make it compile.
- tests/lib3026: fix to use `CreateThread()`.
See individual commits for build error details.
Some of these fixes may have better solutions, and some may not work
as expected. The goal of this patch is to make curl build for UWP.
Orgad Shaneh [Fri, 17 May 2024 11:44:44 +0000 (14:44 +0300)]
socket: support binding to interface *AND* IP
Introduce new notation for CURLOPT_INTERFACE / --interface:
ifhost!<interface>!<host>
Binding to an interface doesn't set the address, and an interface can
have multiple addresses.
When binding to an address (without interface), the kernel is free to
choose the route, and it can route through any device that can access
the target address, not necessarily the one with the chosen address.
Moreover, it is possible for different interfaces to have the same IP
address, on which case we need to provide a way to be more specific.
Factor out the parsing part of interface option, and add unit tests:
1663.
Andy Pan [Sat, 1 Jun 2024 18:26:11 +0000 (02:26 +0800)]
socketpair: add `eventfd` and use `SOCK_NONBLOCK` for `socketpair()`
Currently, we use `pipe` for `wakeup_create`, which requires ***two***
file descriptors. Furthermore, given its complexity inside, `pipe` is a
bit heavyweight for just a simple event wait/notify mechanism.
`eventfd` would be a more suitable solution for this kind of scenario,
kernel also advocates for developers to use `eventfd` instead of `pipe`
in some simple use cases:
Applications can use an eventfd file descriptor instead of a pipe
(see pipe(2) in all cases where a pipe is used simply to signal
events. The kernel overhead of an eventfd file descriptor is much
lower than that of a pipe, and only one file descriptor is required
(versus the two required for a pipe).
This change adds the new backend of `eventfd` for `wakeup_create` and
uses it where available, eliminating the overhead of `pipe`. Also, it
optimizes the `wakeup_create` to eliminate the system calls that make
file descriptors non-blocking by moving the logic of setting
non-blocking flags on file descriptors to `socketpair.c` and using
`SOCK_NONBLOCK` for `socketpair(2)`, `EFD_NONBLOCK` for `eventfd(2)`.
- Remove the locking callback code that demonstrates how to meet
requirements of threading libraries (mainly OpenSSL).
Locking callback code has not been needed for many years. According to
the documentation for OpenSSL and GnuTLS they are thread-safe by design,
assuming support for the underlying OS threading API is built-in.
Viktor Szakats [Tue, 4 Jun 2024 00:41:26 +0000 (02:41 +0200)]
lib: fix thread entry point to return `DWORD` on WinCE
We already do this in `tests/server/util.c`:
https://github.com/curl/curl/blob/97e5e37cc8269660bc5d4a1936f10f2390b97c5a/tests/server/util.c#L604-L606
and in `sockfilt.c`, `lib3026.c`.
Andy Pan [Sun, 2 Jun 2024 06:40:42 +0000 (14:40 +0800)]
socket: use SOCK_NONBLOCK to eliminate extra system call
Every time function `cf_socket_open()` is called to create a socket,
`curlx_nonblock()` is called to make that socket non-blocking. And
`curlx_nonblock()` will cost us 1 or 2 system calls (2 for `fcntl()`, 1
for `ioctl()`, etc.), meanwhile, tucking `SOCK_NONBLOCK` and
`SOCK_CLOEXEC` into the `type` argument for `socket()` is widely
supported across UNIX-like OS: Linux, *BSD, Solaris, etc. With that
ability, we can save 1 or 2 system calls on each socket.
Another change in this PR is to eliminate the redundant
`curlx_nonblock()` call on the socket in `cf_udp_setup_quic()` as that
socket created by `cf_socket_open()` is already non-blocking.
Viktor Szakats [Tue, 4 Jun 2024 00:46:05 +0000 (02:46 +0200)]
GHA: show cmake error log in Windows and non-native workflows
CMake configure doesn't fail often, but when it does, it helps to see
its `CMakeFiles/CMakeConfigureLog.yaml` output. This file is present
since CMake v3.26:
https://cmake.org/cmake/help/v3.26/manual/cmake-configure-log.7.html
(Older CMake versions save similar contend to
`CMakeFiles\CMakeOutput.log` and
`CMakeFiles\CMakeError.log`. This patch doesn't deal with that because
the workflows touched are all running a newer CMake.)
After this patch, we dump the content if cmake fails. Syncing this with
autotools, where we already did that.
Stefan Eissing [Fri, 31 May 2024 11:01:17 +0000 (13:01 +0200)]
mbedtls: v3.6.0 workarounds
- add special sauce to disable unwanted peer verification by mbedtls
when negotiating TLS v1.3
- add special sauce for MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET
return code on *writing* TLS data. We assume the data had not been
written and EAGAIN.
- return correct Curl error code when peer verification failed.
- disable test_08_05 with 50 HTTP/1.1 connections, as mbedtls reports a
memory allocation failed during handshake.
- bump CI mbedtls version to 3.6.0
Stefan Eissing [Mon, 27 May 2024 14:50:15 +0000 (16:50 +0200)]
gnutls: support CA caching
- similar to openssl, use a shared 'credentials' instance
among TLS connections with a plain configuration.
- different to openssl, a connection with a client certificate
is not eligible to sharing.
- document CURLOPT_CA_CACHE_TIMEOUT in man page
Andy Pan [Mon, 3 Jun 2024 07:15:55 +0000 (15:15 +0800)]
tcpkeepalive: support setting TCP keep-alive parameters on Solaris <11.4
Solaris didn't support TCP_KEEPIDLE and TCP_KEEPINTVL until 11.4,
before that it use TCP_KEEPALIVE_THRESHOLD and TCP_KEEPALIVE_ABORT_THRESHOLD
as the substitute. Therefore, for Solaris <11.4 we need to use this substitute
for setting TCP keep-alive parameters.
Viktor Szakats [Sun, 2 Jun 2024 16:22:48 +0000 (18:22 +0200)]
GHA: fix caching old mingw-w64 toolchains in the Windows workflow
- stop altering the `PATH` via `GITHUB_ENV`. This confused the
`actions/cache` post-job, which needs to run in the exact same
environment as its pre-job, to have a consistent cache entry "version"
hash. Altering the `PATH` via `GITHUB_ENV` spills into the the
post-job and breaks this hash. GHA doesn't reset the env automatically
and I have not found a way to do it manually.
- add double-quotes where missing.
- move cache directory under `USERPROFILE` to not rely on absolute
paths.
projects / thirdparty / curl.git / log
