Viktor Szakats [Thu, 17 Apr 2025 14:25:37 +0000 (16:25 +0200)]
runtests: add retry option to reduce flakiness
Add `--retry=<num>` option to tell runtests to retry the first `<num>`
tests that failed. Retries aren't run right away, but added to the end
of the test queue. Once all retry slots are used, test fail as normal.
In CI, typically a single test fails for flakiness, and rarely over 5.
Make the `ci-test` targets default to `--retry=5`.
Stefan Eissing [Tue, 15 Apr 2025 08:55:59 +0000 (10:55 +0200)]
quic: no local idle connection timeout, ngtcp2 keep-alive
Do not set a transport parameter idle timeout, meaning we have no such
thing from our side. The remote setting then applies.
In ngtcp2, set its "keep-alive" timer to prevent a possible remote idle
timeout to tear down the connection while we have active transfers on
that connection.
Viktor Szakats [Thu, 17 Apr 2025 17:21:38 +0000 (19:21 +0200)]
GHA/windows: add 2 basic MSVC jobs to restore some CI coverage
To keep testing these with MSVC:
- UWP !ssl
- arm64 build on the `windows-11-arm` runner
- examples
- OpenSSH-Windows (fix install on `windows-11-arm`)
- `windows-2025` runner
- cmake pre-fill checker
Surprise: UWP doesn't support SSPI, which is required by curl's Schannel
backend. Thus, no TLS support for this UWP build. It also suggests
the Schannel UWP mingw-w64 binaries may be broken and just a happy build
accident thanks to mingw-w64 headers being inaccurate.
Building zlib + libssh2 might actually take up to 2+ minutes with vcpkg,
instead of the previously estimated <1.5 minutes.
Viktor Szakats [Thu, 17 Apr 2025 14:59:04 +0000 (16:59 +0200)]
GHA/windows: restore a simple vcpkg job (zlib + libssh2)
To keep the basics tested. Building zlib and libssh2 takes <1.5 minutes.
So far it seems the "lost" vcpkg binary cache is not a vcpkg-tool
regression, but a GitHub server-side deprecation combined with lack of
replacement service or lack of vcpkg support for such service.
Stefan Eissing [Thu, 17 Apr 2025 09:05:28 +0000 (11:05 +0200)]
pytest: make test_07_22 more lenient to exit codes
Depending on timing when the server aborting the connection is detected,
the reported curl exit code may vary. Check for the possible set of
expected codes instead of a single one.
Stefan Eissing [Tue, 25 Mar 2025 08:47:40 +0000 (09:47 +0100)]
multi: do transfer book keeping using mid
Change multi's book keeping of transfers to no longer use lists, but a
special table and bitsets for unsigned int values.
`multi-xfers` is the `uint_tbl` where `multi_add_handle()` inserts a new
transfer which assigns it a unique identifier `mid`. Use bitsets to keep
track of transfers that are in state "process" or "pending" or
"msgsent".
Use sparse bitsets to replace `conn->easyq` and event handlings tracking
of transfers per socket. Instead of pointers, keep the mids involved.
Provide base data structures and document them in docs/internal:
* `uint_tbl`: a table of transfers with `mid` as lookup key,
handing out a mid for adds between 0 - capacity.
* `uint_bset`: a bitset keeping unsigned ints from 0 - capacity.
* `uint_spbset`: a sparse bitset for keeping a small number of
unsigned int values
* `uint_hash`: for associating `mid`s with a pointer.
This makes the `mid` the recommended way to refer to transfers inside
the same multi without risk of running into a UAF.
Modifying table and bitsets is safe while iterating over them. Overall
memory requirements are lower as with the double linked list apprach.
Stefan Eissing [Wed, 16 Apr 2025 14:16:26 +0000 (16:16 +0200)]
vquic: ngtcp2 + openssl support
With the new addition of OpenSSL QUIC API support and the support in
ngtcp2 main branch, make the necessary adjustments in curl to support
this combination.
- add support in configure.ac to detect the feature OPENSSL_QUIC_API2 in
openssl
- initialise ngtcp2 properly in this combination
- add a Curl_vquic_init() for global initialisation that ngtcp2 likes
for performance reasons
- add documentation on how to build in docs/HTTP3.md
- add CI testing in http3-linux.yml
Stefan Eissing [Wed, 16 Apr 2025 11:45:53 +0000 (13:45 +0200)]
async: DoH improvements
Adds a "meta_hash" to each easy handle for keeping special data during
operations. All meta data set needs to add its destructor callback, so
that meta data gets destroyed properly when the easy handle is cleaned
up or reset.
Add data->master_mid for "sub" transfers that belong to a "master" easy
handle. When a "sub" transfer is done, the corresponding "master" can
add a callback to be invoked. Used in DoH name resolution.
DoH: use easy meta hash to add internal structs for DoH name resolution.
One in each in each probe easy handle. When probes are done, response
data is copied from the probe to the initiating easy.
This allows DoH using transfers and their probes to be cleaned up in any
sequence correctly.
Fold DoH cleanup into the Curl_async_shutdown() and Curl_async_destroy()
functions.
Viktor Szakats [Mon, 14 Apr 2025 10:56:30 +0000 (12:56 +0200)]
cmake: enable `-Wall` for MSVC when `PICKY_COMPILER=ON`
Enable it for `_MSC_VER <= 1943`.
Omit it for not yet CI-tested MSVC versions, to avoid hitting unfixed
warnings emitted by future toolchain releases. It means we need
to explicitly opt-in to newer MSVC versions while fixing any new issues.
The newly enabled warnings did not reveal new issues. It hints that we
catch those with clang/gcc. Yet, these warnings may be useful for local
development done with MSVC.
Also:
- disable and document warnings that don't seem useful, unactionable,
or unfixable.
- disable and document warnings found in Windows SDK headers.
- tidy up a few comments, also to avoid
`-Wdocumentation-unknown-command`, part of llvm/clang `-Wall`.
```
lib\dynhds.h(159,29): error : unknown command tag name [-Werror,-Wdocumentation-unknown-command]
lib\ftp.c(337,15): error : unknown command tag name [-Werror,-Wdocumentation-unknown-command]
```
(This patch did not end up enabling `-Wall` for clang-cl.)
Viktor Szakats [Wed, 16 Apr 2025 02:18:33 +0000 (04:18 +0200)]
GHA: automatic vcpkg binary cache is broken, temporarily disable vcpkg jobs
In the last 1-2 days, curl's vcpkg binary cache entries stored by GHA
apparently disappeared, and each vcpkg job is now rebuilding everything
on every run. This takes up to 40 minutes per job. New cache entries
fail on upload with messages like:
```
Completed submission of nghttp2:arm64-android@1.65.0 to 0 binary cache(s) in 172 ms
```
(note the zero.)
This coincided with my adding support for windows-11-arm and touching
some jobs affected. Yet, unrelated. It also affects Android jobs. The root
cause is a runner image update bumping the vcpkg tool.
The latest vcpkg tool also has this issue.
As a temporary workaround, stop using vcpkg in CI.
Stefan Eissing [Fri, 11 Apr 2025 12:43:45 +0000 (14:43 +0200)]
asyn resolver code improvements
"asyn" is the internal name under which both c-ares and threaded
resolver operate. Make the naming more consistent. Implement the c-ares
resolver in `asyn-ares.*` and the threaded resolver in `asyn-thrdd.*`.
The common functions are in `asyn-base.c`.
When `CURLRES_ASYNCH` is defined, either of the two is used and
`data->state.async` exists. Members of that struct vary for the selected
implementation, but have the fields `hostname`, `port` and `ip_version`
always present. This are populated when the async resolving starts and
eliminate the need to pass them again when checking on the status and
processing the results of the resolving.
Add a `Curl_resolv_blocking()` to `hostip.h` that relieves FTP and SOCKS
from having to repeat the same code.
`Curl_resolv_check()` remains the function to check for status of
ongoing resolving. Now it also performs internally the check if the
needed DNS entry exists in the dnscache and if so, aborts the asnyc
operation. (libcurl right now does not check for duplicate resolve
attempts. an area for future improvements).
The number of functions in `asyn.h` has been reduced. There were subtle
difference in "cancel()" and "kill()" calls, both replaced by
`Curl_async_shutdown()` now. This changes behaviour for threaded
resolver insofar as the resolving thread is now always joined unless
`data->set.quick_exit` is set. Before this was only done on some code
paths. A future improvement would be a thread pool that keeps a limit
and also could handle joins more gracefully.
DoH, not previously tagged under "asny", has its struct `doh_probes` now
also in `data->state.async`, moved there from `data->req` because it
makes more sense. Further integration of DoH underneath the "asyn"
umbrella seems like a good idea.
Viktor Szakats [Tue, 15 Apr 2025 16:17:47 +0000 (18:17 +0200)]
GHA/windows: switch a job to ARM64 on native runner
Switch a build-only MSYS2/mingw-w64 job to ARM64 using the ARM64 Windows
runner. This avoids most downsides of native ARM64 builds. Side-effect
is switching this job from GCC to clang. This adds speed, which offsets
the slower MSYS2 install step.
The new ARM64 runner hits a bunch of corner cases and inefficiencies:
- MSYS2/Cygwin misses native ARM64 support and this isn't expected to
change. It means GH action installs x64 binaries on ARM64. vcpkg does
the same. It runs, but not native, so slower. It affects runtests and
possibly vcpkg build performance.
https://github.com/msys2/MSYS2-packages/discussions/2889
https://cygwin.com/pipermail/cygwin/2021-December/250156.html
- MSYS2 diffutils package missing for CLANGARM64.
- vcpkg boringssl build broken for ARM64.
Also:
- bump the vcpkg build time limit, as it seems the native x86_64 jobs
also can't fit into 35 minutes.
- prepare MSVC jobs for the ARM64 runner.
Joel Depooter [Mon, 24 Mar 2025 23:06:24 +0000 (16:06 -0700)]
schannel: handle pkcs12 client certificates which contain CA certificates
The SChannel code uses the CertFindCertificateInStore function to
retrieve the client certificate from a pkcs12 certificate store.
However, when called with the CERT_FIND_ANY flag, this function does not
provide any guarantees on the order in which certificates are retrieved.
If a pkcs12 file contains an entire certificate chain instead of a
single client certificate, the CertFindCertificateInStore function may
return the CA or an intermediate certificate instead of the desired
client certificate. Since there is no associated private key for such a
certificate, the TLS handshake fails.
With this change, we now pass the CERT_FIND_HAS_PRIVATE_KEY flag. This
ensures that the CertFindCertificateInStore function will return a
certificate which has a corresponding private key. This will stop the CA
and intermediate certificates from being selected. I don't think there
would be much use in a client certificate which has no associated
private key, so this should ensure the client certificate is selected. I
suppose it may be possible for a pkcs12 file to contain multiple
certificates with private keys and the new behaviour may not guarantee
which is selected. However, this is no worse that the previous behaviour
in which any certificate may been selected.
The CERT_FIND_HAS_PRIVATE_KEY is only available in Windows 8 / Server
2012 (aka Windows NT6.2). For older versions, we will fall back to using
the CERT_FIND_ANY flag.
http: in alt-svc negotiation only allow supported HTTP versions
Without this patch, the handling of the alt-svc header added via 279a4772ae67dd4d9770e11e60040f9113b1c345 in curl-8.13.0 attempts to
connect to alternative services via different HTTP versions, even if the
target HTTP version is not supported by curl (i.e., not enabled at
compile-time). If I understand the code and RFC 7838 correctly, then we
should only attempt to migrate to supported protocols. Therefore,
`allowed_apns` should only contain such protocols, and we need to guard
its modification with `ifdefs` for supported HTTP versions.
This was discovered in a downstream bug report in Alpine Linux [1] where
it was reported that a Matrix client (using libcurl) was defunct after
the upgrade to curl-8.13.0. Further debugging revealed that this was due
to the Matrix server sending a `alt-svc: h3=":443";` HTTP header,
causing curl to attempt migration to HTTP3 even though Alpine's curl
version is compiled without HTTP3 support.
I am not sure if this is the best place in the code to address this
or if the `allowed` bitmask shouldn't contain unsupported versions
in the first place. However, since there are existing `ifdefs` in
this function for source (not destination) ALP selection, it may
be a good fit to address this here.
Viktor Szakats [Tue, 15 Apr 2025 14:59:11 +0000 (16:59 +0200)]
GHA/windows: bump a job to windows-2025
- bump an MSYS2/mingw job to windows-2025 runner.
(MSVC is possible, but vcpkg needs to build for windows-2025, and
can't share these with windows-2022 builds, so not optimal for
a single canary job.)
- skip installing OpenSSH-Windows-builtin on windows-2025.
It's preinstalled:
```
ssh client found /c/Windows/System32/OpenSSH/ssh.exe is OpenSSH-Windows 9.5.0
ssh server found /c/Windows/System32/OpenSSH/sshd.exe is OpenSSH-Windows 9.5.0
```
Still older than the manual preview install (9.8.1), so keep using that.
Viktor Szakats [Mon, 14 Apr 2025 22:38:49 +0000 (00:38 +0200)]
cmake: append picky warnings to `CMAKE_REQUIRED_FLAGS` as string
Also:
- drop unnecessary type conversion. `CMAKE_REQUIRED_FLAGS` is already
space-separated.
https://cmake.org/cmake/help/latest/module/CheckCSourceCompiles.html
Viktor Szakats [Mon, 14 Apr 2025 21:47:30 +0000 (23:47 +0200)]
processhelp.pm: always call `taskkill` with `-f` (force)
In the hope this avoid a possible hang in `taskkill`.
To kill processes, `runtests` first tries to kill them gently (with
"TERM", or on Windows `taskkill`), then waits some time for them
to disappear and then kills them with `KILL`, or on Windows with
`taskkill -f`. This happens within `killpid()`.
This patch bumps the gentle phase to `taskkill -f`. On the obervation
that a non-forced `taskkill` may hang in cases:
msvc, CM x64-windows wolfssl +examples:
```
[...]
test 3006...[SMTP with multiple invalid (all) --mail-rcpt and --mail-rcpt-allowfails]
--p----e--- OK (1682 out of 1718, remaining: 00:04, took 0.524s, duration: 03:13)
test 3005...[SMTP with multiple and invalid (all but one) --mail-rcpt and --mail-rcpt-allowfails]
--p-u--e-Executing: 'taskkill -t -pid 1196 >nul 2>&1'
```
Ref: https://github.com/curl/curl/actions/runs/14445993473/job/40508986059?pr=17051#step:15:4176
Cancelling the job worked, resulting in a greyed out status, with the above
step and log entries lost.
If this change causes issues or does nothing at all, we may revert it
or limit it to CI runs.
Viktor Szakats [Mon, 14 Apr 2025 07:50:30 +0000 (09:50 +0200)]
cmake: prefer `COMPILE_OPTIONS` over `CMAKE_C_FLAGS` for custom C options
Also:
- pass `-D_GNU_SOURCE` via `COMPILE_DEFINITIONS`.
- make it explicit to pass these C flags to feature checks.
- update `_GNU_SOURCE` comment with `pipe2()`.
- enable `-pedantic-errors` picky option for GCC with CMake <3.23.
- drop redundant condition when stripping existing MSVC `/Wn` options.
CMake passes `CMAKE_C_FLAGS` to targets, feature checks and raw
`try_compile()` calls. With `COMPILE_OPTIONS`, this is limited to
targets, and we must explicitly pass them to feature checks. This
makes the build logic clearer, and offers more control. It also
reduces log noise by omitting these options from linker commands,
and from `CMAKE_C_FLAGS` dumps in feature checks.
Viktor Szakats [Sun, 23 Mar 2025 15:01:11 +0000 (16:01 +0100)]
runtests: split `SSH_PWD` into `SCP_PWD` and `SFTP_PWD`, and more
To allow configuring paths styles for SCP and SFTP servers separately.
- make `scp://` URLs use `%SCP_PWD` (was: `%SSH_PWD`).
- make `%SCP_PWD` equal to `%POSIX_PWD`.
To fix test 3022 with OpenSSH-Windows 9.8.0 server.
The fix works on a local machine. Remains broken in CI.
Before this patch, it was equal to `%FILE_PWD` when using
OpenSSH-Windows, otherwise it was `%POSIX_PWD`.
Notice that no matter what path-style we pass, test 3022
was and still is broken with earlier OpenSSH-Windows versions.
(as tested with 9.5.0, 9.5.0-beta20240403, 8.0.0.1)
- rename rest of `%SSH_PWD` uses to `%SFTP_PWD`.
- drop unused `%POSIX_PWD`.
- GHA/windows: test with OpenSSH-Windows server again.
In the LibreSSL MSVC job. This job is short enough to fit the slow
install of the built-in OpenSSH-Windows tools, if needed.
Viktor Szakats [Sun, 13 Apr 2025 10:01:00 +0000 (12:01 +0200)]
GHA/windows: add support for built-in OpenSSH-Windows
On the windows-2022 runner it installs these client/server versions:
```
ssh client found /c/Windows/System32/OpenSSH/ssh.exe is OpenSSH-Windows 9.5.0
ssh server found /c/Windows/System32/OpenSSH/sshd.exe is OpenSSH-Windows 8.1.0
```
Not currently enabled. Slight downside (when enabled) that Windows needs
over 1 minute to install these two tiny programs.
Viktor Szakats [Sat, 12 Apr 2025 15:44:43 +0000 (17:44 +0200)]
GHA/windows: bump Cygwin action, move package store to `D:`
- to benefit from the new download retry mechanism.
https://github.com/cygwin/cygwin-install-action/pull/26
- to use a new setting that not only moves the Cygwin install target
directory to the faster `D:` drive, but also the package download
directory. Expecting a little performance improvement from this for
the Cygwin install step.
https://github.com/cygwin/cygwin-install-action/commit/d3a7464b92425a95438e794ec49927871dde78d2
https://github.com/cygwin/cygwin-install-action/pull/27
Stefan Eissing [Fri, 11 Apr 2025 10:05:05 +0000 (12:05 +0200)]
cpool/cshutdown: force close connections under pressure
when CURLMOPT_MAX_HOST_CONNECTIONS or CURLMOPT_MAX_TOTAL_CONNECTIONS
limits are reached, force close connections in shutdown to go below
limit when possible.
Viktor Szakats [Thu, 10 Apr 2025 14:27:11 +0000 (16:27 +0200)]
GHA/macos: re-generate gcc-13 hacklayer on macos-15
Add workaround for an issue related to the gcc "hacklayer" after the
GitHub macos-15-arm64 runner bumped to 20250408.1231.
Fixes:
```
configure:5175: gcc-13 -o conftest --sysroot=/Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk -w conftest.c >&5
In file included from /Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk/usr/include/_stdio.h:71,
from /opt/homebrew/Cellar/gcc@13/13.3.0/lib/gcc/13/gcc/aarch64-apple-darwin24/13/include-fixed/stdio.h:75,
from conftest.c:9:
/Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk/usr/include/_stdio.h: In function 'fmemopen':
/Library/Developer/CommandLineTools/SDKs/MacOSX15.sdk/usr/include/_stdio.h:457:107: error: expected declaration specifiers before '__API_AVAILABLE_GET_MACRO_93585900'
457 | FILE *fmemopen(void * __restrict __buf _LIBC_SIZE(__size), size_t __size, const char * __restrict __mode) __API_AVAILABLE(macos(10.13), ios(11.0), tvos(11.0), watchos(4.0));
| ^~~~~~~~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/14378524390/job/40316589059?pr=17012#step:7:169
Assisted-by: Bo Anderson
Bug: https://github.com/curl/curl/pull/17012#issuecomment-2792572344
Bug: https://github.com/Homebrew/homebrew-core/issues/194778#issuecomment-2792601570
Viktor Szakats [Tue, 8 Apr 2025 22:10:30 +0000 (00:10 +0200)]
cmake: quotes, whitespace, use `VERSION_GREATER_EQUAL`
- `NOT` + `VERSION_LESS` -> `VERSION_GREATER_EQUAL`
Available since 3.7, which is the minimum required for curl:
https://cmake.org/cmake/help/latest/command/if.html#version-greater-equal
- make `CMAKE_REQUIRED_*` argument quotes consistent.
- make `CMAKE_REQUIRED_*` space alignment consistent.
- drop quote from version value for consistency with other cases.
- formatting
Daniel Stenberg [Tue, 8 Apr 2025 06:00:12 +0000 (08:00 +0200)]
lib: include files using known path
by including headers using "../[header]" when done from C files in
subdirectories, we do not need to specify the lib source dir as an
include path and we reduce the risk of header name collisions with
headers in the SDK using the same file names.
Johan Eliasson [Mon, 7 Apr 2025 20:20:29 +0000 (22:20 +0200)]
docs: fix incorrect shell substitution in docker run example command
Corrected the volume mount path in the Docker run example by replacing
`(pwd)` with the shell substitution syntax `$(pwd)`. This ensures the
current working directory is properly mounted into the container.
Viktor Szakats [Sun, 16 Mar 2025 13:19:38 +0000 (14:19 +0100)]
curl_multibyte: fixup low-level calls, include in unity builds
Also adjust `()` around low-level calls preventing macro overrides via
e.g. `memdebug.h`:
- add for `malloc` and `free`.
- drop for `_open`. (We do not override `_open` in curl.)
Tidy-up: also sync libcurlu custom macro order in cmake with autotools.
Andy Pan [Sun, 6 Apr 2025 12:37:10 +0000 (20:37 +0800)]
socketpair: support pipe2 where available
By replacing pipe with pipe2, it would save us 4 extra system calls of
setting O_NONBLOCK and O_CLOEXEC. This system call is widely supported
across UNIX-like OS's: Linux, *BSD, and SunOS derivatives - Solaris,
illumos, etc.
Daniel Stenberg [Sun, 6 Apr 2025 10:04:33 +0000 (12:04 +0200)]
make: clean tests better
1. 'make clean' in the root dir now also invokes 'make clean' in the
tests subdir so that it cleans up better recursively. The Makefile.am
does not list 'tests' as a normal subdir to avoid building that
directory for a normal make invoke.
2. 'make clean' in the tests/libtest and tests/unit directories now
*explicitly* remove the unity build executables even if this is not a
unit build. This, because those files may be leftovers from previous
builds and such leftovers can otherwise linger around and since
'runtests.pl' dynamically acts differently based on the mere *presence*
of those files, they can keep you fooled for a while until you
realize...
Andy Pan [Sat, 5 Apr 2025 15:55:32 +0000 (23:55 +0800)]
socket: use accept4 when available
Linux, *BSD, and Solaris support accept4 system call that enables the
caller to assign additional flags and save some extra system calls. It
can come in handy when O_NONBLOCK or/and FD_CLOEXEC is/are required on a
socket after being accepted.
scripts: completion.pl: sort the completion file for all shells
The reproducible builds effort in Debian has caught a regression in curl
8.13.0-rc1 but we were a bit slow to realize it. The ordering of the
completion file for fish is not deterministic so it can differ between
builds. Since there is no restriction about the order of the completion
file for fish, let's just sort it too.
Daniel Stenberg [Fri, 4 Apr 2025 12:34:10 +0000 (14:34 +0200)]
randdisable: build randomizer
This script makes a "random" build using configure and verifies that it
builds curl correctly. It randomly adds a number of the available
--disable-* flags to configure. When it detects a problem the script
stops, otherwise it continues trying more combinations.
Stefan Eissing [Fri, 4 Apr 2025 08:43:13 +0000 (10:43 +0200)]
http2: fix stream window size after unpausing
When pausing a HTTP/2 transfer, the stream's local window size
is reduced to 0 to prevent the server from sending further data
which curl cannot write out to the application.
When unpausing again, the stream's window size was not correctly
increased again. The attempt to trigger a window update was
ignored by nghttp2, the server never received it and the transfer
stalled.
Add a debug feature to allow use of small window sizes which
reproduces this bug in test_02_21.
Dan Fandrich [Fri, 4 Apr 2025 18:35:33 +0000 (11:35 -0700)]
tests: unify test case keywords
Unify the case, punctuation and name of test case keywords so they can
be more easily selected or skipped when desired. Add a few keywords that
were missing. Fix a couple of typos in test names.
projects / thirdparty / curl.git / log
