Daniel Stenberg [Wed, 12 Jun 2024 06:55:07 +0000 (08:55 +0200)]
file: separate fake headers and body with a stand-alone CRLF
Instead of bolting on the extra CRLF to the final header - as that makes
the behavior inconsistent and not as documented. The final CRLF is now
also made unconditional, just like it is for HTTP.
Stefan Eissing [Mon, 10 Jun 2024 11:32:13 +0000 (13:32 +0200)]
lib: xfer_setup and non-blocking shutdown
- clarify Curl_xfer_setup() with RECV/SEND flags and different calls for
which socket they operate on. Add a shutdown flag for secondary
sockets
- change Curl_xfer_setup() calls to new functions
- implement non-blocking connection shutdown at the end of receiving or
sending a transfer
Daniel Stenberg [Mon, 10 Jun 2024 11:33:16 +0000 (13:33 +0200)]
tool_writeout: bsearch the variable name
As the list of variable names grows, doing a simple loop to find the
name get increasingly worse. This switches to a bsearch.
Also: do a case sensitive check for the variable name. The names have
not been documented to be case insensitive and there is no point in
having them so.
Stefan Eissing [Thu, 6 Jun 2024 10:40:38 +0000 (12:40 +0200)]
multi: prepare multi_wait() for future shutdown usage
- new struct curl_pollfds and struct curl_waitfds
- add structs and methods to init/add/cleanup an array of pollfd and
struct curl_waitfd. Use in multi_wait() and multi_waitfds() to
populate the sets for polling.
- place USE_WINSOCK WSAEventSelect() setting into a separate loop over
all collected pfds
Stefan Eissing [Fri, 7 Jun 2024 08:12:39 +0000 (10:12 +0200)]
connection: shutdown TLS (for FTP) better
This adds connection shutdown infrastructure and first use for FTP. FTP
data connections, when not encountering an error, are now shut down in a
blocking way with a 2sec timeout.
- add cfilter `Curl_cft_shutdown` callback
- keep a shutdown start timestamp and timeout at connectdata
- provide shutdown timeout default and member in
`data->set.shutdowntimeout`.
- provide methods for starting, interrogating and clearing
shutdown timers
- provide `Curl_conn_shutdown_blocking()` to shutdown the
`sockindex` filter chain in a blocking way. Use that in FTP.
- add `Curl_conn_cf_poll()` to wait for socket events during
shutdown of a connection filter chain.
This gets the monitoring sockets and events via the filters
"adjust_pollset()" methods. This gives correct behaviour when
shutting down a TLS connection through a HTTP/2 proxy.
- Implement shutdown for all socket filters
- for HTTP/2 and h2 proxying to send GOAWAY
- for TLS backends to the best of their capabilities
- for tcp socket filter to make a final, nonblocking
receive to avoid unwanted RST states
- add shutdown forwarding to happy eyeballers and
https connect ballers when applicable.
Multipath TCP (MPTCP), standardized in RFC8684 [1], is a TCP extension
that enables a TCP connection to use different paths.
Multipath TCP has been used for several use cases. On smartphones, MPTCP
enables seamless handovers between cellular and Wi-Fi networks while
preserving established connections. This use-case is what pushed Apple
to use MPTCP since 2013 in multiple applications [2]. On dual-stack
hosts, Multipath TCP enables the TCP connection to automatically use the
best performing path, either IPv4 or IPv6. If one path fails, MPTCP
automatically uses the other path.
To benefit from MPTCP, both the client and the server have to support
it. Multipath TCP is a backward-compatible TCP extension that is enabled
by default on recent Linux distributions (Debian, Ubuntu, Redhat, ...).
Multipath TCP is included in the Linux kernel since version 5.6 [3]. To
use it on Linux, an application must explicitly enable it when creating
the socket. No need to change anything else in the application.
This attached patch adds an --mptcp option which allows the creation of
an MPTCP socket instead of TCP on Linux. If Multipath TCP is not
supported on the system, an error will be reported. It is important to
note that if the end server doesn't support MPTCP, the connection will
continue after a seamless fallback to TCP.
Daniel Stenberg [Wed, 5 Jun 2024 12:08:02 +0000 (14:08 +0200)]
managen: warn on excessively long help texts
Help texts at 49 characters or longer get a warning displayed because
they make --help output uglier and we should make an effort to keep the
help texts short and succinct.
The warning is only for display, it does not break the build. That is
left for the future if necessary.
I picked 49 because the longest current text is 48.
Stefan Eissing [Tue, 4 Jun 2024 12:00:12 +0000 (14:00 +0200)]
vtls: new io_need flags for poll handling
- decouple need to recv/send from negotiation state, we need
this later in shutdown handling as well
- move ssl enums from urldata.h to vtls_int.h
- implement use of `connssl->io_need` in vtls.c. and all backends
Daniel Stenberg [Tue, 4 Jun 2024 15:00:05 +0000 (17:00 +0200)]
url: allow DoH transfers to override max connection limit
When reaching the set maximum limit of allowed connections, allow a new
connection anyway if the transfer is created for the (internal) purpose
of doing a DoH name resolve. Otherwise, unrelated "normal" transfers can
starve out new DoH requests making it impossible to name resolve for new
transfers.
Viktor Szakats [Mon, 3 Jun 2024 21:06:56 +0000 (23:06 +0200)]
windows: fix UWP builds, add GHA job
Add new job to test building for UWP (aka `CURL_WINDOWS_APP`).
Fix fallouts when building for UWP:
- rand: do not use `BCryptGenRandom()`.
- cmake: disable using win32 LDAP.
- cmake: disable telnet.
- version_win32: fix code before declaration.
- schannel: disable `HAS_MANUAL_VERIFY_API`.
- schannel: disable `SSLSUPP_PINNEDPUBKEY`
and make `schannel_checksum()` a stub.
Ref: e178fbd40a896f2098278ae61e1166c88e7b31d0 #1429
- schannel: make `cert_get_name_string()` a failing stub.
- system_win32: make `Curl_win32_impersonating()` a failing stub.
- system_win32: try to fix `Curl_win32_init()` (untested).
- threads: fix to use `CreateThread()`.
- src: disable searching `PATH` for the CA bundle.
- src: disable bold text support and capability detection.
- src: disable `getfiletime()`/`setfiletime()`.
- tests: make `win32_load_system_library()` a failing stub.
- tests/server/util: make it compile.
- tests/server/sockfilt: make it compile.
- tests/lib3026: fix to use `CreateThread()`.
See individual commits for build error details.
Some of these fixes may have better solutions, and some may not work
as expected. The goal of this patch is to make curl build for UWP.
Orgad Shaneh [Fri, 17 May 2024 11:44:44 +0000 (14:44 +0300)]
socket: support binding to interface *AND* IP
Introduce new notation for CURLOPT_INTERFACE / --interface:
ifhost!<interface>!<host>
Binding to an interface doesn't set the address, and an interface can
have multiple addresses.
When binding to an address (without interface), the kernel is free to
choose the route, and it can route through any device that can access
the target address, not necessarily the one with the chosen address.
Moreover, it is possible for different interfaces to have the same IP
address, on which case we need to provide a way to be more specific.
Factor out the parsing part of interface option, and add unit tests:
1663.
Andy Pan [Sat, 1 Jun 2024 18:26:11 +0000 (02:26 +0800)]
socketpair: add `eventfd` and use `SOCK_NONBLOCK` for `socketpair()`
Currently, we use `pipe` for `wakeup_create`, which requires ***two***
file descriptors. Furthermore, given its complexity inside, `pipe` is a
bit heavyweight for just a simple event wait/notify mechanism.
`eventfd` would be a more suitable solution for this kind of scenario,
kernel also advocates for developers to use `eventfd` instead of `pipe`
in some simple use cases:
Applications can use an eventfd file descriptor instead of a pipe
(see pipe(2) in all cases where a pipe is used simply to signal
events. The kernel overhead of an eventfd file descriptor is much
lower than that of a pipe, and only one file descriptor is required
(versus the two required for a pipe).
This change adds the new backend of `eventfd` for `wakeup_create` and
uses it where available, eliminating the overhead of `pipe`. Also, it
optimizes the `wakeup_create` to eliminate the system calls that make
file descriptors non-blocking by moving the logic of setting
non-blocking flags on file descriptors to `socketpair.c` and using
`SOCK_NONBLOCK` for `socketpair(2)`, `EFD_NONBLOCK` for `eventfd(2)`.
- Remove the locking callback code that demonstrates how to meet
requirements of threading libraries (mainly OpenSSL).
Locking callback code has not been needed for many years. According to
the documentation for OpenSSL and GnuTLS they are thread-safe by design,
assuming support for the underlying OS threading API is built-in.
Viktor Szakats [Tue, 4 Jun 2024 00:41:26 +0000 (02:41 +0200)]
lib: fix thread entry point to return `DWORD` on WinCE
We already do this in `tests/server/util.c`:
https://github.com/curl/curl/blob/97e5e37cc8269660bc5d4a1936f10f2390b97c5a/tests/server/util.c#L604-L606
and in `sockfilt.c`, `lib3026.c`.
Andy Pan [Sun, 2 Jun 2024 06:40:42 +0000 (14:40 +0800)]
socket: use SOCK_NONBLOCK to eliminate extra system call
Every time function `cf_socket_open()` is called to create a socket,
`curlx_nonblock()` is called to make that socket non-blocking. And
`curlx_nonblock()` will cost us 1 or 2 system calls (2 for `fcntl()`, 1
for `ioctl()`, etc.), meanwhile, tucking `SOCK_NONBLOCK` and
`SOCK_CLOEXEC` into the `type` argument for `socket()` is widely
supported across UNIX-like OS: Linux, *BSD, Solaris, etc. With that
ability, we can save 1 or 2 system calls on each socket.
Another change in this PR is to eliminate the redundant
`curlx_nonblock()` call on the socket in `cf_udp_setup_quic()` as that
socket created by `cf_socket_open()` is already non-blocking.
Viktor Szakats [Tue, 4 Jun 2024 00:46:05 +0000 (02:46 +0200)]
GHA: show cmake error log in Windows and non-native workflows
CMake configure doesn't fail often, but when it does, it helps to see
its `CMakeFiles/CMakeConfigureLog.yaml` output. This file is present
since CMake v3.26:
https://cmake.org/cmake/help/v3.26/manual/cmake-configure-log.7.html
(Older CMake versions save similar contend to
`CMakeFiles\CMakeOutput.log` and
`CMakeFiles\CMakeError.log`. This patch doesn't deal with that because
the workflows touched are all running a newer CMake.)
After this patch, we dump the content if cmake fails. Syncing this with
autotools, where we already did that.
Stefan Eissing [Fri, 31 May 2024 11:01:17 +0000 (13:01 +0200)]
mbedtls: v3.6.0 workarounds
- add special sauce to disable unwanted peer verification by mbedtls
when negotiating TLS v1.3
- add special sauce for MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET
return code on *writing* TLS data. We assume the data had not been
written and EAGAIN.
- return correct Curl error code when peer verification failed.
- disable test_08_05 with 50 HTTP/1.1 connections, as mbedtls reports a
memory allocation failed during handshake.
- bump CI mbedtls version to 3.6.0
Stefan Eissing [Mon, 27 May 2024 14:50:15 +0000 (16:50 +0200)]
gnutls: support CA caching
- similar to openssl, use a shared 'credentials' instance
among TLS connections with a plain configuration.
- different to openssl, a connection with a client certificate
is not eligible to sharing.
- document CURLOPT_CA_CACHE_TIMEOUT in man page
Andy Pan [Mon, 3 Jun 2024 07:15:55 +0000 (15:15 +0800)]
tcpkeepalive: support setting TCP keep-alive parameters on Solaris <11.4
Solaris didn't support TCP_KEEPIDLE and TCP_KEEPINTVL until 11.4,
before that it use TCP_KEEPALIVE_THRESHOLD and TCP_KEEPALIVE_ABORT_THRESHOLD
as the substitute. Therefore, for Solaris <11.4 we need to use this substitute
for setting TCP keep-alive parameters.
Viktor Szakats [Sun, 2 Jun 2024 16:22:48 +0000 (18:22 +0200)]
GHA: fix caching old mingw-w64 toolchains in the Windows workflow
- stop altering the `PATH` via `GITHUB_ENV`. This confused the
`actions/cache` post-job, which needs to run in the exact same
environment as its pre-job, to have a consistent cache entry "version"
hash. Altering the `PATH` via `GITHUB_ENV` spills into the the
post-job and breaks this hash. GHA doesn't reset the env automatically
and I have not found a way to do it manually.
- add double-quotes where missing.
- move cache directory under `USERPROFILE` to not rely on absolute
paths.
Andy Pan [Sat, 1 Jun 2024 10:36:04 +0000 (18:36 +0800)]
socket: change TCP keepalive from ms to seconds on DragonFly BSD
DragonFly BSD changed the time unit for TCP keep-alive from milliseconds
to seconds since v5.8, thus setting the keepalive options with
milliseconds with curl/libcurl will result in unexpected behaviors on
DragonFlyBSD 5.8+
Distinguish the DragonFly BSD versions and use the proper time units
accordingly.
Jan Venekamp [Wed, 29 May 2024 11:58:53 +0000 (13:58 +0200)]
sectransp: remove large cipher table
Previously a large table of ciphers was used to determine the default
ciphers and to lookup manually selected ciphers names.
With the lookup of the manually selected cipher names moved to
Curl_cipher_suite_walk_str() the large table is no longer needed for
that purpose.
The list of manually selected cipher can now be intersected with the
ciphers supported by Secure Transport (SSLGetSupportedCiphers()),
instead of using the fixed table for that.
The other use of the table was to filter the list of all supported
ciphers offered by Secure Transport to create a list of ciphers to
use by default, excluding ciphers in the table marked as weak.
Instead of using a complement based approach (exclude weak), switch
to using an intersection with a smaller list of ciphers deemed
appropriate.
The idea is better use of the build cache. Previously, they
independently create caches with the same key. Some of the caches
include source code and intermediate object files, which makes cache
quite large. In this commit, only built artifacts are cached, which
drastically reduces the cache size. OpenSSL v3, mod_h2 and quiche caches
still include all stuff, but they are left for the later improvement.
Because the contents of the cache have been changed, the cache keys are
also changed to include the word "http3".
Matt Jolly [Thu, 30 May 2024 04:47:11 +0000 (14:47 +1000)]
autoconf: remove 'deeper' checks for `AC_CHECK_FUNCS`
The net effect of the deeper checks is to raise implicit function decls
on modern compilers.
These checks appear to have been added ~20 years ago, relating to an
unverifiable claim about HP-UX. Autoconf support for the platform has
grown in leaps and bounds since.
It didn't cause a real problem here, but when investigating a FP this
came up. No evidence has been identified that this was actually broken
in the past, and there is no evidence that this is necessary now.
`-Werror=implicit-function-declarations` is enabled for both checks;
without a working prototype they will both fail regardless. In the
second case there will in fact never be a working prototype and
therefore it will always fail unconditionally.
`AC_CHECK_FUNCS` does effectively the same thing as the removed checks,
except it actually defines a dummy prototype to see if it links.
If `AC_CHECK_FUNCS` is broken on a given platform we have bigger
problems than trying to build cURL. This should also be faster.
Bug: https://bugs.gentoo.org/932827 Reviewed-By: Eli Schwartz <eschwartz93@gmail.com>
Closes #13830
- Save the last set SO_SNDBUF size to compare against so that we can
avoid setsockopt calls every second.
This is a follow-up to 0b520e12 which moved the SO_SNDBUF update check
into cf-socket. This change improves it further by making the function
names easier to understand and reducing the amount of setsockopt calls.
Adjusts the `make -j` flag to match the latest GitHub-hosted runner
hardware specs[^1]:
- `ubuntu-latest` on 4 CPU cores
- `macos-latest` on 3 CPU cores
The processor count is ideally obtained from `nproc`, but setting env
vars from the current CI yaml files is not possible because they expect
literal strings.
Stefan Eissing [Wed, 29 May 2024 15:13:34 +0000 (17:13 +0200)]
multi: fix multi_wait() timeout handling
- determine the actual poll timeout *after* all sockets
have been collected. Protocols and connection filters may
install new timeouts during collection.
- add debug logging to test1533 where the mistake was noticed
Reported-by: Matt Jolly
Fixes #13782
Closes #13825
Viktor Szakats [Tue, 28 May 2024 14:16:01 +0000 (16:16 +0200)]
lib: prefer `var = time(NULL)` over `time(&var)`
Following up on previous occurrences showing up as gcc warnings, replace
the remaining `time(&var)` calls with `var = time(NULL)`, though these
aren't specifically causing compiler warnings. These are in the TFTP
client code (`lib/tftp.c`), except one which is in a debug branch in
`lib/http_aws_sigv4.c`.
What's unexplainable is that this patch seems to mitigate TFTP tests
often hanging or going into an infinite loop on GHA windows workflows
with MSYS2, mingw-w64 and MSVC (Cygwin is unaffected):
https://github.com/curl/curl/pull/13599#issuecomment-2119372376
TFTP hangs did not entirely disappear though, so could be unrelated.
Stefan Eissing [Fri, 24 May 2024 08:09:32 +0000 (10:09 +0200)]
winsock: move SO_SNDBUF update into cf-socket
- Move the code that updates the SO_SNDBUF size for Windows to
cf_socket_send.
Prior to this change the code was in readwrite_upload but the socket
filter is the more appropriate place because it applies to all sends.
Background:
For Windows users SO_SNDBUF (the total per-socket buffer size reserved
by Winsock for sends) is updated dynamically by libcurl during the
transfer. This is because Windows does not do it automatically for
non-blocking sockets and without it the performance of large transfers
may suffer.
projects / thirdparty / curl.git / log
