Viktor Szakats [Thu, 13 Jun 2024 11:43:22 +0000 (13:43 +0200)]
cmake: allow SOVERSION override with `CURL_LIBCURL_SOVERSION`
Allow overriding SOVERSION with the new CMake option:
`CURL_LIBCURL_SOVERSION=ON/OFF`
For certain target platforms the shared libcurl library filename
contains the SOVERSION. This new option allows to enable/disable
this behavior manually. If set, it takes precedence over the default
setting.
Daniel Stenberg [Thu, 13 Jun 2024 14:08:19 +0000 (16:08 +0200)]
configure: use AC_MSG_WARN for TLS/experimental warning texts
- no longer warns for mbedtls
- warns for each item on individual lines
- no longer shows irrelevant TLS libraries when multiple are selected
- removes ech repetition
Stefan Eissing [Fri, 7 Jun 2024 12:38:51 +0000 (14:38 +0200)]
transfer: do not use EXPIRE_NOW while blocked
- When a transfer sets `data->state.select_bits`, it is
scheduled for rerun with EXPIRE_NOW. If such a transfer
is blocked (due to PAUSE, for example), this will lead to
a busy loop.
- multi.c: check for transfer block
- sendf.*: add Curl_xfer_is_blocked()
- sendf.*: add client reader `is_paused()` callback
- implement is_paused()` callback where needed
Patrick Monnerat [Wed, 12 Jun 2024 12:18:16 +0000 (14:18 +0200)]
os400: make it compilable again
A newly introduced use of getsockname() in the cli tool makes it require
the ascii wrapper module, which is not available outside of the library:
as the tool only uses the address family field (binary), disable
wrappers outside of libcurl.
Fix setsockopt() parameter type mismatch using a (void *) cast.
Viktor Szakats [Fri, 7 Jun 2024 22:41:24 +0000 (00:41 +0200)]
libcurl.pc: add `Requires.private`, `Requires` for static linking
- cmake: populate for dependencies.
- autotools: populate for dependencies.
(including mbedtls, though the script does not detect
mbedtls through pkgconfig. mbedtls 3.6.0 now supports it.)
Viktor Szakats [Wed, 5 Jun 2024 20:35:48 +0000 (22:35 +0200)]
cmake: bring `curl-config.cmake` closer to `FindCURL`
Set `CURL_LIBRARIES` and `CURL_INCLUDE_DIRS` variables
for compatibility with CMake's `FindCURL.cmake`:
https://github.com/Kitware/CMake/blob/b411d0146c2e06acfb0c823bb039e99f0191b611/Modules/FindCURL.cmake#L209
For dependent projects, CMake's suggestion is to replace
`CURL_LIBRARIES` with `CURL::libcurl`, and drop `CURL_INCLUDE_DIRS`.
Reported-by: Aurélien Pierre
Ref: https://curl.se/mail/lib-2024-06/0014.html
Ref: https://gitlab.kitware.com/cmake/cmake/-/issues/24580
Closes #13897
Daniel Stenberg [Wed, 12 Jun 2024 06:55:07 +0000 (08:55 +0200)]
file: separate fake headers and body with a stand-alone CRLF
Instead of bolting on the extra CRLF to the final header - as that makes
the behavior inconsistent and not as documented. The final CRLF is now
also made unconditional, just like it is for HTTP.
Stefan Eissing [Mon, 10 Jun 2024 11:32:13 +0000 (13:32 +0200)]
lib: xfer_setup and non-blocking shutdown
- clarify Curl_xfer_setup() with RECV/SEND flags and different calls for
which socket they operate on. Add a shutdown flag for secondary
sockets
- change Curl_xfer_setup() calls to new functions
- implement non-blocking connection shutdown at the end of receiving or
sending a transfer
Daniel Stenberg [Mon, 10 Jun 2024 11:33:16 +0000 (13:33 +0200)]
tool_writeout: bsearch the variable name
As the list of variable names grows, doing a simple loop to find the
name get increasingly worse. This switches to a bsearch.
Also: do a case sensitive check for the variable name. The names have
not been documented to be case insensitive and there is no point in
having them so.
Stefan Eissing [Thu, 6 Jun 2024 10:40:38 +0000 (12:40 +0200)]
multi: prepare multi_wait() for future shutdown usage
- new struct curl_pollfds and struct curl_waitfds
- add structs and methods to init/add/cleanup an array of pollfd and
struct curl_waitfd. Use in multi_wait() and multi_waitfds() to
populate the sets for polling.
- place USE_WINSOCK WSAEventSelect() setting into a separate loop over
all collected pfds
Stefan Eissing [Fri, 7 Jun 2024 08:12:39 +0000 (10:12 +0200)]
connection: shutdown TLS (for FTP) better
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension
that enables a TCP connection to use different paths.
Multipath TCP has been used for several use cases. On smartphones, MPTCP
enables seamless handovers between cellular and Wi-Fi networks while
preserving established connections. This use-case is what pushed Apple
to use MPTCP since 2013 in multiple applications [2]. On dual-stack
hosts, Multipath TCP enables the TCP connection to automatically use the
best performing path, either IPv4 or IPv6. If one path fails, MPTCP
automatically uses the other path.
To benefit from MPTCP, both the client and the server have to support
it. Multipath TCP is a backward-compatible TCP extension that is enabled
by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...).
Multipath TCP is included in the Linux kernel since version 5.6 [3]. To
use it on Linux, an application must explicitly enable it when creating
the socket. No need to change anything else in the application.
This attached patch adds an --mptcp option which allows the creation of
an MPTCP socket instead of TCP on Linux. If Multipath TCP is not
supported on the system, an error will be reported. It is important to
note that if the end server doesn't support MPTCP, the connection will
continue after a seamless fallback to TCP.
Daniel Stenberg [Wed, 5 Jun 2024 12:08:02 +0000 (14:08 +0200)]
managen: warn on excessively long help texts
Help texts at 49 characters or longer get a warning displayed because
they make --help output uglier and we should make an effort to keep the
help texts short and succinct.
The warning is only for display, it does not break the build. That is
left for the future if necessary.
I picked 49 because the longest current text is 48.
Stefan Eissing [Tue, 4 Jun 2024 12:00:12 +0000 (14:00 +0200)]
vtls: new io_need flags for poll handling
- decouple need to recv/send from negotiation state, we need
this later in shutdown handling as well
- move ssl enums from urldata.h to vtls_int.h
- implement use of `connssl->io_need` in vtls.c. and all backends
Daniel Stenberg [Tue, 4 Jun 2024 15:00:05 +0000 (17:00 +0200)]
url: allow DoH transfers to override max connection limit
When reaching the set maximum limit of allowed connections, allow a new
connection anyway if the transfer is created for the (internal) purpose
of doing a DoH name resolve. Otherwise, unrelated "normal" transfers can
starve out new DoH requests making it impossible to name resolve for new
transfers.
Viktor Szakats [Mon, 3 Jun 2024 21:06:56 +0000 (23:06 +0200)]
windows: fix UWP builds, add GHA job
Add new job to test building for UWP (aka `CURL_WINDOWS_APP`).
Fix fallouts when building for UWP:
- rand: do not use `BCryptGenRandom()`.
- cmake: disable using win32 LDAP.
- cmake: disable telnet.
- version_win32: fix code before declaration.
- schannel: disable `HAS_MANUAL_VERIFY_API`.
- schannel: disable `SSLSUPP_PINNEDPUBKEY`
and make `schannel_checksum()` a stub.
Ref: e178fbd40a896f2098278ae61e1166c88e7b31d0 #1429
- schannel: make `cert_get_name_string()` a failing stub.
- system_win32: make `Curl_win32_impersonating()` a failing stub.
- system_win32: try to fix `Curl_win32_init()` (untested).
- threads: fix to use `CreateThread()`.
- src: disable searching `PATH` for the CA bundle.
- src: disable bold text support and capability detection.
- src: disable `getfiletime()`/`setfiletime()`.
- tests: make `win32_load_system_library()` a failing stub.
- tests/server/util: make it compile.
- tests/server/sockfilt: make it compile.
- tests/lib3026: fix to use `CreateThread()`.
See individual commits for build error details.
Some of these fixes may have better solutions, and some may not work
as expected. The goal of this patch is to make curl build for UWP.
Orgad Shaneh [Fri, 17 May 2024 11:44:44 +0000 (14:44 +0300)]
socket: support binding to interface *AND* IP
Introduce new notation for CURLOPT_INTERFACE / --interface:
ifhost!<interface>!<host>
Binding to an interface doesn't set the address, and an interface can
have multiple addresses.
When binding to an address (without interface), the kernel is free to
choose the route, and it can route through any device that can access
the target address, not necessarily the one with the chosen address.
Moreover, it is possible for different interfaces to have the same IP
address, on which case we need to provide a way to be more specific.
Factor out the parsing part of interface option, and add unit tests:
1663.
Andy Pan [Sat, 1 Jun 2024 18:26:11 +0000 (02:26 +0800)]
socketpair: add `eventfd` and use `SOCK_NONBLOCK` for `socketpair()`
Currently, we use `pipe` for `wakeup_create`, which requires ***two***
file descriptors. Furthermore, given its complexity inside, `pipe` is a
bit heavyweight for just a simple event wait/notify mechanism.
`eventfd` would be a more suitable solution for this kind of scenario,
kernel also advocates for developers to use `eventfd` instead of `pipe`
in some simple use cases:
Applications can use an eventfd file descriptor instead of a pipe
(see pipe(2) in all cases where a pipe is used simply to signal
events. The kernel overhead of an eventfd file descriptor is much
lower than that of a pipe, and only one file descriptor is required
(versus the two required for a pipe).
This change adds the new backend of `eventfd` for `wakeup_create` and
uses it where available, eliminating the overhead of `pipe`. Also, it
optimizes the `wakeup_create` to eliminate the system calls that make
file descriptors non-blocking by moving the logic of setting
non-blocking flags on file descriptors to `socketpair.c` and using
`SOCK_NONBLOCK` for `socketpair(2)`, `EFD_NONBLOCK` for `eventfd(2)`.
- Remove the locking callback code that demonstrates how to meet
requirements of threading libraries (mainly OpenSSL).
Locking callback code has not been needed for many years. According to
the documentation for OpenSSL and GnuTLS they are thread-safe by design,
assuming support for the underlying OS threading API is built-in.
Viktor Szakats [Tue, 4 Jun 2024 00:41:26 +0000 (02:41 +0200)]
lib: fix thread entry point to return `DWORD` on WinCE
We already do this in `tests/server/util.c`:
https://github.com/curl/curl/blob/97e5e37cc8269660bc5d4a1936f10f2390b97c5a/tests/server/util.c#L604-L606
and in `sockfilt.c`, `lib3026.c`.
Andy Pan [Sun, 2 Jun 2024 06:40:42 +0000 (14:40 +0800)]
socket: use SOCK_NONBLOCK to eliminate extra system call
Every time function `cf_socket_open()` is called to create a socket,
`curlx_nonblock()` is called to make that socket non-blocking. And
`curlx_nonblock()` will cost us 1 or 2 system calls (2 for `fcntl()`, 1
for `ioctl()`, etc.), meanwhile, tucking `SOCK_NONBLOCK` and
`SOCK_CLOEXEC` into the `type` argument for `socket()` is widely
supported across UNIX-like OS: Linux, *BSD, Solaris, etc. With that
ability, we can save 1 or 2 system calls on each socket.
Another change in this PR is to eliminate the redundant
`curlx_nonblock()` call on the socket in `cf_udp_setup_quic()` as that
socket created by `cf_socket_open()` is already non-blocking.
Viktor Szakats [Tue, 4 Jun 2024 00:46:05 +0000 (02:46 +0200)]
GHA: show cmake error log in Windows and non-native workflows
CMake configure doesn't fail often, but when it does, it helps to see
its `CMakeFiles/CMakeConfigureLog.yaml` output. This file is present
since CMake v3.26:
https://cmake.org/cmake/help/v3.26/manual/cmake-configure-log.7.html
(Older CMake versions save similar contend to
`CMakeFiles\CMakeOutput.log` and
`CMakeFiles\CMakeError.log`. This patch doesn't deal with that because
the workflows touched are all running a newer CMake.)
After this patch, we dump the content if cmake fails. Syncing this with
autotools, where we already did that.
Stefan Eissing [Fri, 31 May 2024 11:01:17 +0000 (13:01 +0200)]
mbedtls: v3.6.0 workarounds
- add special sauce to disable unwanted peer verification by mbedtls
when negotiating TLS v1.3
- add special sauce for MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET
return code on *writing* TLS data. We assume the data had not been
written and EAGAIN.
- return correct Curl error code when peer verification failed.
- disable test_08_05 with 50 HTTP/1.1 connections, as mbedtls reports a
memory allocation failed during handshake.
- bump CI mbedtls version to 3.6.0
Stefan Eissing [Mon, 27 May 2024 14:50:15 +0000 (16:50 +0200)]
gnutls: support CA caching
- similar to openssl, use a shared 'credentials' instance
among TLS connections with a plain configuration.
- different to openssl, a connection with a client certificate
is not eligible to sharing.
- document CURLOPT_CA_CACHE_TIMEOUT in man page
Andy Pan [Mon, 3 Jun 2024 07:15:55 +0000 (15:15 +0800)]
tcpkeepalive: support setting TCP keep-alive parameters on Solaris <11.4
Solaris didn't support TCP_KEEPIDLE and TCP_KEEPINTVL until 11.4,
before that it use TCP_KEEPALIVE_THRESHOLD and TCP_KEEPALIVE_ABORT_THRESHOLD
as the substitute. Therefore, for Solaris <11.4 we need to use this substitute
for setting TCP keep-alive parameters.
Viktor Szakats [Sun, 2 Jun 2024 16:22:48 +0000 (18:22 +0200)]
GHA: fix caching old mingw-w64 toolchains in the Windows workflow
- stop altering the `PATH` via `GITHUB_ENV`. This confused the
`actions/cache` post-job, which needs to run in the exact same
environment as its pre-job, to have a consistent cache entry "version"
hash. Altering the `PATH` via `GITHUB_ENV` spills into the the
post-job and breaks this hash. GHA doesn't reset the env automatically
and I have not found a way to do it manually.
- add double-quotes where missing.
- move cache directory under `USERPROFILE` to not rely on absolute
paths.
Andy Pan [Sat, 1 Jun 2024 10:36:04 +0000 (18:36 +0800)]
socket: change TCP keepalive from ms to seconds on DragonFly BSD
DragonFly BSD changed the time unit for TCP keep-alive from milliseconds
to seconds since v5.8, thus setting the keepalive options with
milliseconds with curl/libcurl will result in unexpected behaviors on
DragonFlyBSD 5.8+
Distinguish the DragonFly BSD versions and use the proper time units
accordingly.
Jan Venekamp [Wed, 29 May 2024 11:58:53 +0000 (13:58 +0200)]
sectransp: remove large cipher table
Previously a large table of ciphers was used to determine the default
ciphers and to lookup manually selected ciphers names.
With the lookup of the manually selected cipher names moved to
Curl_cipher_suite_walk_str() the large table is no longer needed for
that purpose.
The list of manually selected cipher can now be intersected with the
ciphers supported by Secure Transport (SSLGetSupportedCiphers()),
instead of using the fixed table for that.
The other use of the table was to filter the list of all supported
ciphers offered by Secure Transport to create a list of ciphers to
use by default, excluding ciphers in the table marked as weak.
Instead of using a complement based approach (exclude weak), switch
to using an intersection with a smaller list of ciphers deemed
appropriate.
The idea is better use of the build cache. Previously, they
independently create caches with the same key. Some of the caches
include source code and intermediate object files, which makes cache
quite large. In this commit, only built artifacts are cached, which
drastically reduces the cache size. OpenSSL v3, mod_h2 and quiche caches
still include all stuff, but they are left for the later improvement.
Because the contents of the cache have been changed, the cache keys are
also changed to include the word "http3".