Viktor Szakats [Sun, 27 Apr 2025 10:26:41 +0000 (12:26 +0200)]
cmake: extend integration tests
- GHA: add cmake integration tests for Windows.
- make them run faster with prefill, unity, Ninja, omitting curl tool.
- also test static libcurl.
- add old-cmake support with auto-detection.
- auto-detect Ninja.
- run consumer test apps to see if they work.
- add support for Windows.
- make it more verbose.
- re-add `ExternalProject` cmake consumer test. It's broken.
- tidy up terminology.
To help anyone wanting to build/reproduce release candidates, this is
the set git tag naming scheme to use. Similar to, but different, than
the "normal" release tags to not be possible to mixup.
Viktor Szakats [Sat, 26 Apr 2025 07:34:55 +0000 (09:34 +0200)]
cmake: honor individual picky option overrides found in `CMAKE_C_FLAGS`
Also to sync up with similar `./configure` feature via
`CURL_ADD_COMPILER_WARNINGS()`.
Example: `-DCMAKE_C_FLAGS=-Wno-xor-used-as-pow`
It may be useful as a workaround if a specific build combination hits
a picky warning within curl's source code. If such happens, we do
appreciate a report to fix it in curl itself.
Viktor Szakats [Thu, 24 Apr 2025 23:45:23 +0000 (01:45 +0200)]
cmake: stop deleting `-W<n>` from `CMAKE_C_FLAGS` (MSVC)
1. `CMAKE_C_FLAGS` may apply to other projects, and deleting/altering it
may be unexpected.
2. We pass `-W4`/`-Wall` internally now, which do override custom
`-W<n>` options in all supported MSVC versions.
(as tested with Visual Studio generators)
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/51945416
Daniel Stenberg [Fri, 25 Apr 2025 06:16:13 +0000 (08:16 +0200)]
tests/buildinfo: former "disabled" now provides more info
This tool now contains ON/OFF information about features in the build.
This way, runtests gets both positive and negative feature presence with
this. Allows for more flexibility and avoids having to duplicate the
names.
Viktor Szakats [Thu, 24 Apr 2025 17:12:42 +0000 (19:12 +0200)]
GHA/windows: limit jobs to 15 minutes
They typically finish (well) within 10 minutes.
A notable exception was vcpkg jobs when a rebuild was triggered.
With caching lost and reducing them to short builds, this is not
an issue at the moment.
The advantage of shorter timeouts is hung/crashed jobs giving back
control earlier for a manual retry.
Viktor Szakats [Wed, 23 Apr 2025 08:26:38 +0000 (10:26 +0200)]
cmake: install shell completions for cross-builds
Also:
- omit auto-detecting `CURL_COMPLETION_FISH_DIR` via `pkg-config`
for cross-builds and when `CMAKE_INSTALL_PREFIX` is set.
- flatten nested `if`s.
Note:
On macOS with Homebrew, `pkg-config --variable completionsdir fish`
returns the version-specific Cellar path instead of the permanent path
`/opt/homebrew/share/fish/vendor_completions.d/`. This mimics what
autotools does, but may need further fixing, possibly upstream.
https://github.com/Homebrew/homebrew-core/blob/9c13e62b009b8e814fda180e0fcc5096318daf31/Formula/f/fish.rb
https://github.com/fish-shell/fish-shell/blob/ce631fd2fb1f5b63f5f0f1b4041a30dfad823d22/cmake/Install.cmake#L15-L21
Stefan Eissing [Wed, 23 Apr 2025 09:24:45 +0000 (11:24 +0200)]
multi: init_do(): check result
Calls to `Curl_init_do()` did not check on result and missed failures to
properly and completely initialize a transfer request.
The main cause of such an init failure is the need to rewind the
READFUNCTION without a SEEKFUNCTION registered. Check the failure to
"rewind" the upload data immediately make test cases 1576 and friends
fail.
Reported-by: Travis Lane
Fixes #17139
Closes #17150
Viktor Szakats [Wed, 23 Apr 2025 10:12:02 +0000 (12:12 +0200)]
GHA/windows: apply MSYS2 runtime downgrades to v3.5.x, leave v3.6.x as-is
windows-runners 20250420.1.0 come with msys2-runtime 3.6.x. It has
the perf regression issue fixed, so stop downgrading it.
This makes CI jobs settle on this version when supplied by
the runner image or the msys2/setup-msys2 action:
MINGW64_NT-10.0-20348 fv-az980-636 3.6.1-0cfedd4f.x86_64 2025-04-12 01:44 UTC x86_64 Msys
With 3.6.1, we've seen issues launching `perl.exe` before this patch:
https://github.com/curl/curl/discussions/14854#discussioncomment-12908214
https://github.com/curl/curl/discussions/14854#discussioncomment-12921007
I don't understand what this error says, why it happens in certain CMake
versions, and why a workaround is necessary for what seems like
a standard export/consume configuration. This patch is based on internet
suggestions and other projects ending up with this workaround.
Stefan Eissing [Tue, 22 Apr 2025 10:53:22 +0000 (12:53 +0200)]
http: fix HTTP/2 handling of TE request header using "trailers"
A "TE" request header is allowed in HTTP/2 when it only carries the
"trailers" value. RFC 9113 ch. 8.2.2. Check client supplied TE values
for the "trailers" token and only pass that one in a HTTP/2 request.
Add test_01_17 to verify.
Fixes #17122 Reported-by: epicmkirzinger on github
Closes #17128
Viktor Szakats [Sun, 20 Apr 2025 09:59:06 +0000 (11:59 +0200)]
GHA/linux: tidy up around wolfSSH
- fix filter expression for cache step.
It did not cause an issue because `wolfssh` is always paired with
a `wolfssl*`.
- build `wolfssh` against `wolfssl-opensslextra` (was: `wolfssl-all`).
It makes `wolfssh` builds `OPENSSL_COEXIST`-compatible, and clarifies
its use in the clang-tidy job. The earlier mixup didn't cause issues
because the clang-tidy job is compile-only (using their headers only.)
Useful side-effect is making the wolfssh build valgrind-tested.
Reported-by: bo0tzz on github
Ref: https://github.com/curl/curl/discussions/16970#discussioncomment-12752019
Viktor Szakats [Thu, 17 Apr 2025 14:25:37 +0000 (16:25 +0200)]
runtests: add retry option to reduce flakiness
Add `--retry=<num>` option to tell runtests to retry the first `<num>`
tests that failed. Retries aren't run right away, but added to the end
of the test queue. Once all retry slots are used, test fail as normal.
In CI, typically a single test fails for flakiness, and rarely over 5.
Make the `ci-test` targets default to `--retry=5`.
Stefan Eissing [Tue, 15 Apr 2025 08:55:59 +0000 (10:55 +0200)]
quic: no local idle connection timeout, ngtcp2 keep-alive
Do not set a transport parameter idle timeout, meaning we have no such
thing from our side. The remote setting then applies.
In ngtcp2, set its "keep-alive" timer to prevent a possible remote idle
timeout to tear down the connection while we have active transfers on
that connection.
Viktor Szakats [Thu, 17 Apr 2025 17:21:38 +0000 (19:21 +0200)]
GHA/windows: add 2 basic MSVC jobs to restore some CI coverage
To keep testing these with MSVC:
- UWP !ssl
- arm64 build on the `windows-11-arm` runner
- examples
- OpenSSH-Windows (fix install on `windows-11-arm`)
- `windows-2025` runner
- cmake pre-fill checker
Surprise: UWP doesn't support SSPI, which is required by curl's Schannel
backend. Thus, no TLS support for this UWP build. It also suggests
the Schannel UWP mingw-w64 binaries may be broken and just a happy build
accident thanks to mingw-w64 headers being inaccurate.
Building zlib + libssh2 might actually take up to 2+ minutes with vcpkg,
instead of the previously estimated <1.5 minutes.
Viktor Szakats [Thu, 17 Apr 2025 14:59:04 +0000 (16:59 +0200)]
GHA/windows: restore a simple vcpkg job (zlib + libssh2)
To keep the basics tested. Building zlib and libssh2 takes <1.5 minutes.
So far it seems the "lost" vcpkg binary cache is not a vcpkg-tool
regression, but a GitHub server-side deprecation combined with lack of
replacement service or lack of vcpkg support for such service.
Stefan Eissing [Thu, 17 Apr 2025 09:05:28 +0000 (11:05 +0200)]
pytest: make test_07_22 more lenient to exit codes
Depending on timing when the server aborting the connection is detected,
the reported curl exit code may vary. Check for the possible set of
expected codes instead of a single one.
Stefan Eissing [Tue, 25 Mar 2025 08:47:40 +0000 (09:47 +0100)]
multi: do transfer book keeping using mid
Change multi's book keeping of transfers to no longer use lists, but a
special table and bitsets for unsigned int values.
`multi-xfers` is the `uint_tbl` where `multi_add_handle()` inserts a new
transfer which assigns it a unique identifier `mid`. Use bitsets to keep
track of transfers that are in state "process" or "pending" or
"msgsent".
Use sparse bitsets to replace `conn->easyq` and event handlings tracking
of transfers per socket. Instead of pointers, keep the mids involved.
Provide base data structures and document them in docs/internal:
* `uint_tbl`: a table of transfers with `mid` as lookup key,
handing out a mid for adds between 0 - capacity.
* `uint_bset`: a bitset keeping unsigned ints from 0 - capacity.
* `uint_spbset`: a sparse bitset for keeping a small number of
unsigned int values
* `uint_hash`: for associating `mid`s with a pointer.
This makes the `mid` the recommended way to refer to transfers inside
the same multi without risk of running into a UAF.
Modifying table and bitsets is safe while iterating over them. Overall
memory requirements are lower as with the double linked list apprach.
Stefan Eissing [Wed, 16 Apr 2025 14:16:26 +0000 (16:16 +0200)]
vquic: ngtcp2 + openssl support
With the new addition of OpenSSL QUIC API support and the support in
ngtcp2 main branch, make the necessary adjustments in curl to support
this combination.
- add support in configure.ac to detect the feature OPENSSL_QUIC_API2 in
openssl
- initialise ngtcp2 properly in this combination
- add a Curl_vquic_init() for global initialisation that ngtcp2 likes
for performance reasons
- add documentation on how to build in docs/HTTP3.md
- add CI testing in http3-linux.yml
Stefan Eissing [Wed, 16 Apr 2025 11:45:53 +0000 (13:45 +0200)]
async: DoH improvements
Adds a "meta_hash" to each easy handle for keeping special data during
operations. All meta data set needs to add its destructor callback, so
that meta data gets destroyed properly when the easy handle is cleaned
up or reset.
Add data->master_mid for "sub" transfers that belong to a "master" easy
handle. When a "sub" transfer is done, the corresponding "master" can
add a callback to be invoked. Used in DoH name resolution.
DoH: use easy meta hash to add internal structs for DoH name resolution.
One in each in each probe easy handle. When probes are done, response
data is copied from the probe to the initiating easy.
This allows DoH using transfers and their probes to be cleaned up in any
sequence correctly.
Fold DoH cleanup into the Curl_async_shutdown() and Curl_async_destroy()
functions.
Viktor Szakats [Mon, 14 Apr 2025 10:56:30 +0000 (12:56 +0200)]
cmake: enable `-Wall` for MSVC when `PICKY_COMPILER=ON`
Enable it for `_MSC_VER <= 1943`.
Omit it for not yet CI-tested MSVC versions, to avoid hitting unfixed
warnings emitted by future toolchain releases. It means we need
to explicitly opt-in to newer MSVC versions while fixing any new issues.
The newly enabled warnings did not reveal new issues. It hints that we
catch those with clang/gcc. Yet, these warnings may be useful for local
development done with MSVC.
Also:
- disable and document warnings that don't seem useful, unactionable,
or unfixable.
- disable and document warnings found in Windows SDK headers.
- tidy up a few comments, also to avoid
`-Wdocumentation-unknown-command`, part of llvm/clang `-Wall`.
```
lib\dynhds.h(159,29): error : unknown command tag name [-Werror,-Wdocumentation-unknown-command]
lib\ftp.c(337,15): error : unknown command tag name [-Werror,-Wdocumentation-unknown-command]
```
(This patch did not end up enabling `-Wall` for clang-cl.)
Viktor Szakats [Wed, 16 Apr 2025 02:18:33 +0000 (04:18 +0200)]
GHA: automatic vcpkg binary cache is broken, temporarily disable vcpkg jobs
In the last 1-2 days, curl's vcpkg binary cache entries stored by GHA
apparently disappeared, and each vcpkg job is now rebuilding everything
on every run. This takes up to 40 minutes per job. New cache entries
fail on upload with messages like:
```
Completed submission of nghttp2:arm64-android@1.65.0 to 0 binary cache(s) in 172 ms
```
(note the zero.)
This coincided with my adding support for windows-11-arm and touching
some jobs affected. Yet, unrelated. It also affects Android jobs. The root
cause is a runner image update bumping the vcpkg tool.
The latest vcpkg tool also has this issue.
As a temporary workaround, stop using vcpkg in CI.
projects / thirdparty / curl.git / log
