Stefan Eissing [Mon, 29 Jul 2024 08:23:20 +0000 (10:23 +0200)]
connect: fix connection shutdown for event based processing
connections being shutdown would register sockets for events, but then
never remove these sockets again. Nor would the shutdown effectively
been performed.
- If a socket event involves a transfer, check if that is the
connection cache internal handle and run its multi_perform()
instead (the internal handle is used for all shutdowns).
- When a timer triggers for a transfer, check also if it is
about the connection cache internal handle.
- During processing shutdowns in the connection cache, assess
the shutdown timeouts. Register a Curl_expire() of the lowest
value for the cache's internal handle.
Reported-by: Gordon Parke
Fixes #14280
Closes #14296
Daniel Stenberg [Mon, 29 Jul 2024 08:17:08 +0000 (10:17 +0200)]
tests: provide FTP directory contents in the test file
Instead of providing a fixed single synthetic response in the test
server itself. To allow us to better use *different* directory listings
in different test cases. In this change, most listings remain the same
as before.
The wildcard match tests still use synthetic responses but we should fix
that as well.
Viktor Szakats [Sat, 20 Jul 2024 16:20:06 +0000 (18:20 +0200)]
runtests: fold timing details with GHA, sync `-r` tflags
- move timing details into a foldable group when run in GitHub Actions.
Spec:
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#grouping-log-lines
- enable `-r` (run time stats) option in autotools' `test-ci` target,
syncing it with cmake.
Stefan Eissing [Fri, 26 Jul 2024 08:38:45 +0000 (10:38 +0200)]
transfer: speed limiting fix for 32bit systems
When checking if a speed limit on receives applies, compare the receive
sizes using the large int type to prevent an overflow on systems where
size_t is 32bit.
- implement the socket hash user/reader/writer processing also
for connections that are being shut down by the connection cache.
- split out handling of current vs. last pollset socket event handling
into a function available in other code parts
- add `shutdown_poll` pollset to `connectdata` struct so that changes
in the pollset can be recorded during shutdown. (The internal handle
cannot keep it since it might be used for many connections)
Reported-by: calvin2021y on github
Fixes #14252
Closes #14257
Daniel Stenberg [Sat, 20 Jul 2024 21:21:16 +0000 (23:21 +0200)]
tool_cb_prg: output "flying saucers" with leading carriage return
Because that is how the progress-bar is output, so when the progress-bar
has been shown at least once and the information is reset, like for a
redirect, there might be a moment where the size goes from known to
unknown and then the flying saucerts are shown after a brief display of
the progress-bar.
It could previously cause accidental character leftovers on the right
side of the bar when using a narrow display.
Reported-by: Chris Webb
Fixes #14213
Closes #14246
md4: fix compilation with OpenSSL 1.x with md4 disabled
If OpenSSL 1.x is used, and it is configured with md4 disabled,
OPENSSL_NO_MD4 is defined in opensslconf.h, but this header was not
included before checking for this define.
Later in md4.c, openssl/md4.h is included, and it includes that header
indirectly, leading to inconsistency within md4.c.
Since the md4.h branch was taken, wincrypt.h (or others) is not
included, and later below the USE_WIN32_CRYPTO branch is taken, but the
types are not defined.
Viktor Szakats [Wed, 17 Jul 2024 23:09:04 +0000 (01:09 +0200)]
GHA/macos: improve, fix gcc/llvm, add new test matrix
This PR began as an attempt to drop GCC support, after repeated reports
on fallouts when trying to use it on macOS.
Then it transformed into a 3-week project turning up the issues causing
the fallouts, ending up including llvm and all available Xcode / macOS
SDK, macOS runner image, build tools and compiler vendors and versions.
Accumulating 400 sub-commits.
I developed and tested all fixes under this PR, then merged them as
separate patches.
This PR retained CI jobs updates, extensively reworking and extending
them: [1]
At first it seemed GCC and the Apple SDK is "naturally" growing more
incompatible, as Apple added further non-standard features to their
headers. This is partly true, but reality is more complicated.
Besides some issues local to curl, there were bugs in Apple SDK
headers, Homebrew GCC builds, feature missing in the old llvm version
pre-installed on GitHub CI runner images, and subtle incompatibilities
between GCC and llvm/clang when handling language extensions.
Resulting compiler errors seldom pointed to a useful direction, and
internet search was silent about these issues too. Thus, I had to peel
them off layer by layer, using trial and error, and by recognizing
patterns of failures accross 150-200 builds combinations. Exposing
configure logs, and curl_config.h in the CI logs helped too.
1. GCC header compatibility layer ("hack" as GCC calls it)
The toughest issue is GCC's built-in compatibility layer:
https://github.com/gcc-mirror/gcc/tree/master/fixincludes
This patch layer is further patched by a "Darwin compatibility" project
applied on top by Homebrew GCC via:
https://github.com/iains/gcc-12-branch
https://github.com/iains/gcc-13-branch
https://github.com/iains/gcc-14-branch
The hack layer is designed in a way that breaks more builds than it
fixes, esp. in context of GHA runners. The idea is to build GCC
specifically for the SDK for the target macOS version. The problem with
this approach is that the Xcode + SDK installed on the local/CI machine
often does not match with the SDK used on while building GCC on
Homebrew's build machines. In these cases the GCC compatibility layer
turns into an "uncompatibility" layer and consistently breaks builds.
curl cannot offer a fix for this, because the solution (I found) is to
patch the toolchain on the local machine. I implemented this for our CI
builds and curl-for-win. In other case the user must do this patching
manually, or choose a compatible GCC + Xcode/SDK combination.
An upstream fix doesn't seem trivial either, because the issue is
ingrained in the compatibility layer's design. Offering an `-fapplesdk`
(or recognizing `-target`) option and/or fixing them within the compiler
would seem like a more robust option, and also how mainline llvm solves
this.
Here's a table summarizing the GCC + SDK combinations and curl build
failures: [2]
More info: https://github.com/curl/curl/issues/10356#issuecomment-2222734103
A recent minor Homebrew GCC upgrade caused major breakage. The "Darwin
compatibility" patch applied to GCC implemented the `availability`
compiler attribute in GCC. Apple SDK detected this and enabled using
them, but as it turns out GCC accepts compiler attributes with slightly
different rules than llvm/clang, and how the Apple SDK uses them,
breaking builds.
Affected Homebrew GCC versions are: 12.4.0, 13.3.0 and 14.1.0.
Possibly tracked here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info: https://github.com/llvm/llvm-project/issues/81767
That applied to Homebrew GCC (12.4.0):
https://github.com/Homebrew/homebrew-core/commit/b904223d9893f62bec2a8f7483bf5992747fc6c7#diff-89dd0b4176eca7fcc24b591943509bf8a8d6ea904d71e5dfcd6b78fed62fc574R44-R48
Ref: #13700
More info: https://github.com/curl/curl/pull/14091#issuecomment-2222703468
Apple SDK expects certain macros predefined by the compiler. Missing
them may causes odd issues. Mainline llvm is keeping up with Apple
clang, but it needs a fresh version, while the one installed on GitHub
runners is old (v15). I patched these in `lib/curl_setup.h`.
Without certain predefined macros, SDK headers can take a codepath where
it mis-defines its own `TARGET_OS_OSX` macro, which make it break its
own headers later. I patched it in `lib/curl_setup.h`.
7. Differences between autotools and CMake dependency detection
Fixed it by improving detection of libidn2, with some more fixes
pending for the next feature window.
f43adc2c4978f7f82a359e89186e58a31d17b0ad #14137 cmake: detect `libidn2` also via `pkg-config`
Ref: #14136 cmake: detect `nghttp2` via `pkg-config`, enable by default
8. libidn2 detection bug with CMake
Fixed the root cause and also the trigger in the CI config.
curl source code assumes this is available to enable certain codepaths.
It's also intermixed with monotonic timer support.
14. Monotonic timer support with GCC
Detected by GCC, while it probably shouldn't be. llvm/clang detects it
depending on target OS version. I've been playing with this, but so far
without a conclusion or fix.
15. Runtime/test failures with GCC
I couldn't find the reason for most of this. A bunch of RTSP tests fail
with GCC. SecureTransport + HTTP/2 is failing a bunch of tests. With
OpenSSL it fails two of those. SecureTransport builds also fail one DoH
test.
16. Runtime/test failure in llvm/clang
AppleIDN support received a fix with two more remaining.
- #14185 runtests: fold test details for GitHub CI runs
- #14197 cmake: grab-bag of tidy-ups
- #14196 configure: limit `__builtin_available` test to Darwin
Summary:
In general GCC doesn't seem to be a good fit with curl and macOS for
now. These "lucky" combinations (GitHub Actions runner) will build out
of the box now: macos-14 + Xcode 15.0.1 + gcc-11, gcc-12, gcc-14. The
rest builds with the ugly workaround in place, but all this still leaves
some runtime issues.
More info and links in the commit messages and source code.
[1]: This PR:
- add info about target OS version requirements per feature, with OS
names and release years.
- stop using `-Wno-deprecated-declarations` to suppress warnings.
- use `LDFLAGS=-w` to suppress 'object file was built for newer macOS
version than being linked' warnings.
(there were tens of thousands of them in some jobs)
- allow overriding Xcode version in all jobs.
- improve job names.
- abbreviate CMake as CM, autotools as AM for more compact job names.
- shorten job names by using `!` instead of `no-` and `non-`.
- bump parellel tests to 10 (from 5).
- drop using `--enable-maintainer-mode` `./configure` option.
- add gcc-12 no-ssl, autotools job with tests, ignore failing test
results. (It's not yet clear why gcc-12 builds have different runtime
results than clang/llvm ones.)
- add comments with OS names and release years next to version numbers,
e.g. 10.15 # Catalina (2019)
- fix broken gcc-12 SecureTransport build.
- show compiler, Xcode, SDK, gcc hack SDK versions, Homebrew
preinstalled packages and C compiler predefined macros for each job.
Useful for debugging all the strange problems these builds might have.
- merge brew bundle and install steps.
- move step names to the top.
- dump configure log for both cmake and autotools also for successful
builds. Useful for debugging.
- dump curl_config.h in short (sorted #defines) and full form.
- add support for the mainline llvm compiler.
- set sysroot for gcc and llvm.
- add timeout for cmake jobs.
- add new job matrix: combinations
It supports building all possible compiler, runner image, Xcode/SDK
combinations, with cmake and autotools, target OS versions and with or
without SecureTransport. It's quick. GHA limits the maximum number of
matrix jobs at 256.
I used this as a test-rig to fix the macOS build fallouts with gcc and
llvm.
I settled with 16 jobs, trying to maximize fallout coverage.
- implement hack to make Homebrew gcc work with all available SDKs.
- add handy mini-table about Xcode / SDK versions, OS names, years for
each GHA images, with the defaults.
- add tests for cmake jobs.
- make cmake config hack to link GnuTLS less intrusive.
- stop ignoring test 1452, seems fine now.
- fix to enable libpsl in autotools builds.
- enable libpsl in cmake builds.
- add an llvm job with tests (both autotools and cmake).
- delete similar macOS jobs from Circle CI. GHA is now arm64 too.
[2]: Homebrew GCC vs GHA runner images vs curl builds:
```
macOS Xcode gcc gcc SDK hacks Xcode SDK SDK major Build Compile
(*def) (Homebrew) (CommandLineTools) versions error
-------- -------- ---------- ------------------ ---------- --------- ----- ---------------------
macos-12 13.1 GCC 11.4.0 MacOSX12 MacOSX12.0
macos-12 13.2.1 GCC 11.4.0 MacOSX12 MacOSX12.1
macos-12 13.3.1 GCC 11.4.0 MacOSX12 MacOSX12.3
macos-12 13.4.1 GCC 11.4.0 MacOSX12 MacOSX12.3
macos-12 14.0.1 GCC 11.4.0 MacOSX12 MacOSX12.3
macos-12 14.1 GCC 11.4.0 MacOSX12 MacOSX13.0 MISMATCH FAIL /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12 *14.2 GCC 11.4.0 MacOSX12 MacOSX13.1 MISMATCH FAIL /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13 14.1 GCC 11.4.0 MacOSX13 MacOSX13.0
macos-13 14.2 GCC 11.4.0 MacOSX13 MacOSX13.1
macos-13 14.3.1 GCC 11.4.0 MacOSX13 MacOSX13.3
macos-13 *15.0.1 GCC 11.4.0 MacOSX13 MacOSX14.0 MISMATCH FAIL /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13 15.1 GCC 11.4.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13 15.2 GCC 11.4.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-14 14.3.1 GCC 11.4.0 MacOSX14 MacOSX13.3 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 *15.0.1 GCC 11.4.0 MacOSX14 MacOSX14.0
macos-14 15.1 GCC 11.4.0 MacOSX14 MacOSX14.2
macos-14 15.2 GCC 11.4.0 MacOSX14 MacOSX14.2
macos-14 15.3 GCC 11.4.0 MacOSX14 MacOSX14.4
macos-14 15.4 GCC 11.4.0 MacOSX14 MacOSX14.5
macos-14 16.0 GCC 11.4.0 MacOSX14 MacOSX15.0 MISMATCH FAIL /opt/homebrew/Cellar/gcc@11/11.4.0/lib/gcc/11/gcc/aarch64-apple-darwin23/11/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12 13.1 GCC 12.4.0 MacOSX12 MacOSX12.0
macos-12 13.2.1 GCC 12.4.0 MacOSX12 MacOSX12.1
macos-12 13.3.1 GCC 12.4.0 MacOSX12 MacOSX12.3
macos-12 13.4.1 GCC 12.4.0 MacOSX12 MacOSX12.3
macos-12 14.0.1 GCC 12.4.0 MacOSX12 MacOSX12.3
macos-12 14.1 GCC 12.4.0 MacOSX12 MacOSX13.0 MISMATCH FAIL /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12 *14.2 GCC 12.4.0 MacOSX12 MacOSX13.1 MISMATCH FAIL /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13 14.1 GCC 12.4.0 MacOSX13 MacOSX13.0
macos-13 14.2 GCC 12.4.0 MacOSX13 MacOSX13.1
macos-13 14.3.1 GCC 12.4.0 MacOSX13 MacOSX13.3
macos-13 *15.0.1 GCC 12.4.0 MacOSX13 MacOSX14.0 MISMATCH FAIL /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13 15.1 GCC 12.4.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13 15.2 GCC 12.4.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-14 14.3.1 GCC 12.4.0 MacOSX14 MacOSX13.3 MISMATCH
macos-14 *15.0.1 GCC 12.4.0 MacOSX14 MacOSX14.0
macos-14 15.1 GCC 12.4.0 MacOSX14 MacOSX14.2
macos-14 15.2 GCC 12.4.0 MacOSX14 MacOSX14.2
macos-14 15.3 GCC 12.4.0 MacOSX14 MacOSX14.4
macos-14 15.4 GCC 12.4.0 MacOSX14 MacOSX14.5
macos-14 16.0 GCC 12.4.0 MacOSX14 MacOSX15.0 MISMATCH FAIL /opt/homebrew/Cellar/gcc@12/12.4.0/lib/gcc/12/gcc/aarch64-apple-darwin23/12/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12 13.1 GCC 13.3.0 MacOSX12 MacOSX12.0
macos-12 13.2.1 GCC 13.3.0 MacOSX12 MacOSX12.1
macos-12 13.3.1 GCC 13.3.0 MacOSX12 MacOSX12.3
macos-12 13.4.1 GCC 13.3.0 MacOSX12 MacOSX12.3
macos-12 14.0.1 GCC 13.3.0 MacOSX12 MacOSX12.3
macos-12 14.1 GCC 13.3.0 MacOSX12 MacOSX13.0 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-12 *14.2 GCC 13.3.0 MacOSX12 MacOSX13.1 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13 14.1 GCC 13.3.0 MacOSX13 MacOSX13.0
macos-13 14.2 GCC 13.3.0 MacOSX13 MacOSX13.1
macos-13 14.3.1 GCC 13.3.0 MacOSX13 MacOSX13.3
macos-13 *15.0.1 GCC 13.3.0 MacOSX13 MacOSX14.0 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13 15.1 GCC 13.3.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13 15.2 GCC 13.3.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 14.3.1 GCC 13.3.0 MacOSX14 MacOSX13.3 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 *15.0.1 GCC 13.3.0 MacOSX14 MacOSX14.0 FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 15.1 GCC 13.3.0 MacOSX14 MacOSX14.2 FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 15.2 GCC 13.3.0 MacOSX14 MacOSX14.2 FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 15.3 GCC 13.3.0 MacOSX14 MacOSX14.4
macos-14 15.4 GCC 13.3.0 MacOSX14 MacOSX14.5
macos-14 16.0 GCC 13.3.0 MacOSX14 MacOSX15.0 MISMATCH FAIL /opt/homebrew/Cellar/gcc@13/13.3.0/lib/gcc/13/gcc/aarch64-apple-darwin23/13/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12 13.1 GCC 14.1.0 MacOSX12 MacOSX12.0
macos-12 13.2.1 GCC 14.1.0 MacOSX12 MacOSX12.1
macos-12 13.3.1 GCC 14.1.0 MacOSX12 MacOSX12.3
macos-12 13.4.1 GCC 14.1.0 MacOSX12 MacOSX12.3
macos-12 14.0.1 GCC 14.1.0 MacOSX12 MacOSX12.3
macos-12 14.1 GCC 14.1.0 MacOSX12 MacOSX13.0 MISMATCH FAIL /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12 *14.2 GCC 14.1.0 MacOSX12 MacOSX13.1 MISMATCH FAIL /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13 14.1 GCC 14.1.0 MacOSX13 MacOSX13.0
macos-13 14.2 GCC 14.1.0 MacOSX13 MacOSX13.1
macos-13 14.3.1 GCC 14.1.0 MacOSX13 MacOSX13.3
macos-13 *15.0.1 GCC 14.1.0 MacOSX13 MacOSX14.0 MISMATCH FAIL /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-13 15.1 GCC 14.1.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-13 15.2 GCC 14.1.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-14 14.3.1 GCC 14.1.0 MacOSX14 MacOSX13.3 MISMATCH
macos-14 *15.0.1 GCC 14.1.0 MacOSX14 MacOSX14.0
macos-14 15.1 GCC 14.1.0 MacOSX14 MacOSX14.2
macos-14 15.2 GCC 14.1.0 MacOSX14 MacOSX14.2
macos-14 15.3 GCC 14.1.0 MacOSX14 MacOSX14.4
macos-14 15.4 GCC 14.1.0 MacOSX14 MacOSX14.5
macos-14 16.0 GCC 14.1.0 MacOSX14 MacOSX15.0 MISMATCH FAIL /opt/homebrew/Cellar/gcc/14.1.0_1/lib/gcc/current/gcc/aarch64-apple-darwin23/14/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
```
Source: https://github.com/curl/curl/actions/runs/9883956647/job/27299564218
Viktor Szakats [Wed, 17 Jul 2024 23:09:04 +0000 (01:09 +0200)]
GHA/macos: improve, fix gcc/llvm, add new test matrix
This PR began as an attempt to drop GCC support, after repeated reports
on fallouts when trying to use it on macOS.
Then it transformed into a 3-week project turning up the issues causing
the fallouts, ending up including llvm and all available Xcode / macOS
SDK, macOS runner image, build tools and compiler vendors and versions.
Accumulating 400 sub-commits.
I developed and tested all fixes under this PR, then merged them as
separate patches.
This PR retained CI jobs updates, extensively reworking and extending
them: [1]
At first it seemed GCC and the Apple SDK is "naturally" growing more
incompatible, as Apple added further non-standard features to their
headers. This is partly true, but reality is more complicated.
Besides some issues local to curl, there were bugs in Apple SDK
headers, Homebrew GCC builds, feature missing in the old llvm version
pre-installed on GitHub CI runner images, and subtle incompatibilities
between GCC and llvm/clang when handling language extensions.
Resulting compiler errors seldom pointed to a useful direction, and
internet search was silent about these issues too. Thus, I had to peel
them off layer by layer, using trial and error, and by recognizing
patterns of failures accross 150-200 builds combinations. Exposing
configure logs, and curl_config.h in the CI logs helped too.
1. GCC header compatibility layer ("hack" as GCC calls it)
The toughest issue is GCC's built-in compatibility layer:
https://github.com/gcc-mirror/gcc/tree/master/fixincludes
This patch layer is further patched by a "Darwin compatibility" project
applied on top by Homebrew GCC via:
https://github.com/iains/gcc-12-branch
https://github.com/iains/gcc-13-branch
https://github.com/iains/gcc-14-branch
The hack layer is designed in a way that breaks more builds than it
fixes, esp. in context of GHA runners. The idea is to build GCC
specifically for the SDK for the target macOS version. The problem with
this approach is that the Xcode + SDK installed on the local/CI machine
often does not match with the SDK used on while building GCC on
Homebrew's build machines. In these cases the GCC compatibility layer
turns into an "uncompatibility" layer and consistently breaks builds.
curl cannot offer a fix for this, because the solution (I found) is to
patch the toolchain on the local machine. I implemented this for our CI
builds and curl-for-win. In other case the user must do this patching
manually, or choose a compatible GCC + Xcode/SDK combination.
An upstream fix doesn't seem trivial either, because the issue is
ingrained in the compatibility layer's design. Offering an `-fapplesdk`
(or recognizing `-target`) option and/or fixing them within the compiler
would seem like a more robust option, and also how mainline llvm solves
this.
Here's a table summarizing the GCC + SDK combinations and curl build
failures: [2]
More info: https://github.com/curl/curl/issues/10356#issuecomment-2222734103
A recent minor Homebrew GCC upgrade caused major breakage. The "Darwin
compatibility" patch applied to GCC implemented the `availability`
compiler attribute in GCC. Apple SDK detected this and enabled using
them, but as it turns out GCC accepts compiler attributes with slightly
different rules than llvm/clang, and how the Apple SDK uses them,
breaking builds.
Affected Homebrew GCC versions are: 12.4.0, 13.3.0 and 14.1.0.
Possibly tracked here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info: https://github.com/llvm/llvm-project/issues/81767
That applied to Homebrew GCC (12.4.0):
https://github.com/Homebrew/homebrew-core/commit/b904223d9893f62bec2a8f7483bf5992747fc6c7#diff-89dd0b4176eca7fcc24b591943509bf8a8d6ea904d71e5dfcd6b78fed62fc574R44-R48
Ref: #13700
More info: https://github.com/curl/curl/pull/14091#issuecomment-2222703468
Apple SDK expects certain macros predefined by the compiler. Missing
them may causes odd issues. Mainline llvm is keeping up with Apple
clang, but it needs a fresh version, while the one installed on GitHub
runners is old (v15). I patched these in `lib/curl_setup.h`.
Without certain predefined macros, SDK headers can take a codepath where
it mis-defines its own `TARGET_OS_OSX` macro, which make it break its
own headers later. I patched it in `lib/curl_setup.h`.
7. Differences between autotools and CMake dependency detection
Fixed it by improving detection of libidn2, with some more fixes
pending for the next feature window.
f43adc2c4978f7f82a359e89186e58a31d17b0ad #14137 cmake: detect `libidn2` also via `pkg-config`
Ref: #14136 cmake: detect `nghttp2` via `pkg-config`, enable by default
8. libidn2 detection bug with CMake
Fixed the root cause and also the trigger in the CI config.
curl source code assumes this is available to enable certain codepaths.
It's also intermixed with monotonic timer support.
14. Monotonic timer support with GCC
Detected by GCC, while it probably shouldn't be. llvm/clang detects it
depending on target OS version. I've been playing with this, but so far
without a conclusion or fix.
15. Runtime/test failures with GCC
I couldn't find the reason for most of this. A bunch of RTSP tests fail
with GCC. SecureTransport + HTTP/2 is failing a bunch of tests. With
OpenSSL it fails two of those. SecureTransport builds also fail one DoH
test.
16. Runtime/test failure in llvm/clang
AppleIDN support received a fix with two more remaining.
Stefan Eissing [Thu, 18 Jul 2024 09:29:37 +0000 (11:29 +0200)]
lib: send eos flag
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.
Stefan Eissing [Wed, 17 Jul 2024 12:46:47 +0000 (14:46 +0200)]
doh: fix cleanup
When removing an easy handle that had DoH sub-easy handles going, those
were not removed from the multi handle. Their memory was reclaimed on
curl_easy_cleanup() of the owning handle, but multi still had them in
their list.
Add `Curl_doh_close()` and `Curl_doh_cleanup()` as common point for
handling the DoH resource management. Use the `multi` present in the doh
handles (if so), for removal, as the `data->multi` might already have
been NULLed at this time.
Alex Snast [Wed, 17 Jul 2024 11:06:06 +0000 (14:06 +0300)]
http/3: resume upload on ack if we have more data to send
Currently we're waiting for sendbuf_len_in_flight to hit zero before
resuming upload which means we're blocking and waiting for _all_ acks to
arrive before sending more data. This causes significant delays especially
when ack delay is used on the server side.
The fix addresses several issues in h3 over ngtcp2:
- On ack we now call nghttp3_conn_resume_stream() when we have more
data to send.
- upload_left was incorrectly computed on CF_CTRL_DATA_DONE_SEND as
we need to subtract the ammount of data we have in flight.
- Remove upload_blocked_len as we Curl_bufq_write call will do the
right thing when called from cf_ngtcp2_send.
Stefan Eissing [Wed, 17 Jul 2024 10:26:40 +0000 (12:26 +0200)]
pytests: scorecard upload tests
- add upload tests to scorecard, invoke with
> python3 tests/http/scorecard.py -u h1|h2|h3
- add a reverse proxy setup from Caddy to httpd for
upload tests since Caddy does not have other PUT/POST handling
- add caddy tests in test_08 for POST/PUT
- increase read buffer in mod_curltest for larger reads
Tal Regev [Mon, 15 Jul 2024 18:29:17 +0000 (21:29 +0300)]
GHA/windows: add MSVC wolfSSL job with test
Fix the file of wolfssl.c because of this warning/error:
```
curl\lib\vtls\wolfssl.c(1017,42): error C2220: the following warning is treated as an error [curl\bld\lib\libcurl_object.vcxproj]
curl\lib\vtls\wolfssl.c(1017,42): warning C4267: 'function': conversion from 'size_t' to 'unsigned long', possible loss of data [curl\bld\lib\libcurl_object.vcxproj]
```
`size_t` in MSVC is different. Change it to `unsigned long` because
`wolfSSL_ERR_error_string_n` last argument is defined as
`unsigned long`.
Viktor Szakats [Mon, 15 Jul 2024 22:41:32 +0000 (00:41 +0200)]
tests: sync feature names with `curl -V`
Some feature names used in tests had minor differences compared to
the well-known ones from `curl -V`. This patch syncs them to make test
results easier to grok.
Stefan Eissing [Mon, 15 Jul 2024 09:56:51 +0000 (11:56 +0200)]
test2600: disable on win32
- disbable this test on WIN32 platforms. It uses the file describtor '1'
as valid socket without events. Not portable.
- reduce trace output somewhat on other runs
Fixes #14177 Reported-by: Viktor Szakats
Closes #14191
Daniel Stenberg [Thu, 11 Jul 2024 13:54:25 +0000 (15:54 +0200)]
libcurl-docs: make option lists alpha-sorted
The man pages for curl_easy_getinfo, curl_easy_setopt and
curl_multi_setopt now feature the lists of options alphabetically
sorted. Test 1139 verify that they are.
The curl_multi_setopt page also got brief explanations of the listed
options.
Viktor Szakats [Sat, 13 Jul 2024 11:06:33 +0000 (13:06 +0200)]
cmake: fix builds with detected libidn2 lib but undetected header
It caused IDN to appear in `curl-config`, `libidn2` referenced from
`libcurl.pc`, fail to fallback to `pkg-config` detection. But libidn2
not actually used.
It came up in macOS CI builds after enabling cmake build tests. It
remained hidden for a while due to setting `-DUSE_APPLE_IDN=ON`.
(The half-detection of Homebrew libidn2 was the result of configuring
with `-DCMAKE_EXE_LINKER_FLAGS=-L$(brew --prefix)/lib`, to fix
linking GnuTLS that needs the `nettle` lib from the brew prefix.)
Viktor Szakats [Fri, 12 Jul 2024 21:31:21 +0000 (23:31 +0200)]
cmake: fix building `unit1600` due to missing `ssl/openssl.h`
In specific builds configs, cmake failed to build test `unit1600`,
due missing an OpenSSL (or wolfSSL) header.
The test code relies on `lib/curl_ntlm_core.h`, which in turn included
TLS library headers. But, dependency header directories are not setup
in cmake for tests, because they should not normally be needed.
The issue was hidden in most builds because TLS headers are usually
found under the system prefix. One counterexample is macOS + Homebrew
LibreSSL builds, where OpenSSL is purposefully unlinked from there to
avoid a mixup with LibreSSL that resides under its own prefix. It was
also hidden in autotools, possibly because it sets up header directories
globally, tests included.
The actual bug however is that `lib/curl_ntlm_core.h` should not include
TLS headers. None of its internal users need it, and `curl_ntlm_core.c`
included them already directly.
Fix it by deleting the TLS header includes from this internal header.
Fixes:
```
In file included from curl/tests/unit/unit1600.c:27:
curl/lib/curl_ntlm_core.h:32:12: fatal error: 'openssl/ssl.h' file not found
# include <openssl/ssl.h>
^~~~~~~~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/9912684737/job/27388041520#step:12:1694
Viktor Szakats [Sat, 6 Jul 2024 12:59:57 +0000 (14:59 +0200)]
sectransp: fix clang compiler warnings, stop silencing them
Fix `-Wpointer-bool-conversion` warnings with the method suggested by
both Apple clang and mainline llvm. This was already tried and dropped
in #1705 (in year 2017), but the issue reported there no longer
replicates.
Verified with Apple clang 14, llvm 15, llvm 18 and gcc 11, 14 that the
generated objects are bit by bit identical before and after this patch.
Also:
- stop silencing `-Wtautological-pointer-compare`. This warning don't
seem to be appearing anymore (with or without this patch), at least
with the tested compilers and SDKs (clang 13.1.6-16.0.0beta, llvm 15,
18, gcc 11, 14) and minimum macOS target of 10.8. Older targets fail
to build curl with SecureTransport.
- silence `-Wunreachable-code` for clang only. Previously I applied it
also to GCC, by mistake.
Ref: https://github.com/curl/curl/pull/12331/commits/8d7172d20a48ebc6c1b1d94a76e2c5fb19dd9bfa
Apple clang `-Wpointer-bool-conversion`:
```
curl/lib/vtls/sectransp.c:1103:6: error: address of function 'SSLCreateContext' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
if(SSLCreateContext) { /* use the newer API if available */
~~ ^~~~~~~~~~~~~~~~
curl/lib/vtls/sectransp.c:1103:6: note: prefix with the address-of operator to silence this warning
if(SSLCreateContext) { /* use the newer API if available */
^
&
```
Ref: https://github.com/curl/curl/actions/runs/9819538439/job/27113201384#step:8:382
llvm `-Wpointer-bool-conversion`:
```
curl/lib/vtls/sectransp.c:2663:8: error: address of function 'SSLCreateContext' will always evaluate to 'true' [-Werror,-Wpointer-bool-conversion]
if(SSLCreateContext)
~~ ^~~~~~~~~~~~~~~~
curl/lib/vtls/sectransp.c:2663:8: note: prefix with the address-of operator to silence this warning
if(SSLCreateContext)
^
&
```
Ref: https://github.com/curl/curl/actions/runs/9819538439/job/27113200291#step:8:417
gcc still needs `-Waddress` suppressed to avoid these:
```
curl/lib/vtls/n/sectransp.c: In function 'getsubject':
curl/lib/vtls/n/sectransp.c:379:6: warning: the address of 'SecCertificateCopyLongDescription' will always evaluate as 'true' [-Waddress]
379 | if(&SecCertificateCopyLongDescription)
| ^
[...]
```
Viktor Szakats [Fri, 12 Jul 2024 17:23:43 +0000 (19:23 +0200)]
CI/circleci: config tidy-ups, bump up test parallelism
- bump parallel test for Linux jobs. Credit-to: Dan Fandrich
Cherry-picked from #11510
- bump parallel test for macOS jobs.
- drop no longer necessary `-Wno-vla` option.
- fold long lines.
- drop `--enable-maintainer-mode` `./configure` option.
- replace a hard-coded prefix with `brew --prefix`.
- update documentation link.
- move `--enable-debug` in front.
- tidy up quotes.
Stephen Farrell [Wed, 10 Jul 2024 22:43:32 +0000 (23:43 +0100)]
doh: fix leak and zero-length HTTPS RR crash
This PR fixes a leak and a crash that can happen when curl encounters
bad HTTPS RR values in DNS. We're starting to do better testing of that
kind of thing and e.g. have published bad HTTPS RR values at
dodgy.test.defo.ie.
Viktor Szakats [Thu, 11 Jul 2024 17:35:29 +0000 (19:35 +0200)]
build: fix llvm 17 and older + macOS SDK 14.4 and newer
Fixup faulty target macro initialization in macOS SDK since v14.4 (as of
15.0 beta). The SDK target detection in `TargetConditionals.h` correctly
detects macOS, but fails to set the macro's old name `TARGET_OS_OSX`,
then continues to set it to a default value of 0. Other parts of the SDK
still rely on the old name, and with this inconsistency our builds fail
due to missing declarations. It happens when using mainline llvm older
than v18. Later versions fixed it by predefining these target macros,
avoiding the faulty dynamic detection. gcc is not affected (for now)
because it lacks the necessary dynamic detection features, so the SDK
falls back to a codepath that sets both the old and new macro to 1.
Also move the `TargetConditionals.h` include to the top of to make sure
including it also for c-ares builds, combined with SecureTransport or
other curl features that may call use an Apple SDK.
Before this patch, affected build combinations (e.g. in GHA runners,
llvm@15 + Xcode 15.3, 15.4, 16.0 with their default SDKs +
SecureTransport) fail with:
```
error: use of undeclared identifier 'noErr'
or 'SecCertificateCopyLongDescription'
or 'SecItemImportExportKeyParameters'
or 'SecExternalFormat'
or 'SecExternalItemType'
or 'SEC_KEY_IMPORT_EXPORT_PARAMS_VERSION'
```
Example:
```
curl/lib/vtls/sectransp.c:311:18: error: use of undeclared identifier 'noErr'
OSStatus rtn = noErr;
^
curl/lib/vtls/sectransp.c:379:7: error: use of undeclared identifier 'SecCertificateCopyLongDescription'
if(&SecCertificateCopyLongDescription)
^
curl/lib/vtls/sectransp.c:381:7: error: call to undeclared function 'SecCertificateCopyLongDescription'; ISO C99 and later do not support implicit function declarations [-Werror,-Wimplicit-function-declaration]
SecCertificateCopyLongDescription(NULL, cert, NULL);
^
curl/lib/vtls/sectransp.c:380:25: error: incompatible integer to pointer conversion assigning to 'CFStringRef' (aka 'const struct __CFString *') from 'int' [-Wint-conversion]
server_cert_summary =
^
[...]
```
Ref: https://github.com/curl/curl/actions/runs/9893867519/job/27330135969#step:10:22
Viktor Szakats [Tue, 9 Jul 2024 18:48:11 +0000 (20:48 +0200)]
macos: undo `availability` macro enabled by Homebrew gcc
Homebrew gcc builds starting with 12.4.0, 13.3.0 and 14.1.0 enabled
the `availability` attribute.
This broke builds because the way the Apple SDK uses attributes (when
available) are incompatible with how gcc accepts them. Causing these
errors:
```
error: attributes should be specified before the declarator in a function definition
error: expected ',' or '}' before
```
The project above is a Darwin gcc compatibility pack, that is applied
to Homebrew gcc builds.
This patch works by redefining the `availability` macro to an invalid
value, making `__has_attribute(availability)` checks fail, stopping
Apple SDK from inserting the incompatible attributes.
It also replaces the previous, local workaround for `lib/macos.c`.
Example with gcc 12.4.0 with macOS SDK 14.0 (Xcode 15.0.1):
```
In file included from <path-to-sdk>/MacOSX14.0.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CoreFoundation.h:54,
from <path-to-sdk>/MacOSX14.0.sdk/System/Library/Frameworks/SystemConfiguration.framework/Headers/SCDynamicStoreCopySpecific.h:30,
from /Users/runner/work/curl/curl/lib/macos.c:33,
from /Users/runner/work/curl/curl/build/lib/CMakeFiles/libcurl_shared.dir/Unity/unity_0_c.c:244:
<path-to-sdk>/MacOSX14.0.sdk/System/Library/Frameworks/CoreFoundation.framework/Headers/CFUserNotification.h:126:1: error: attributes should be specified before the declarator in a function definition
126 | CF_INLINE CFOptionFlags CFUserNotificationCheckBoxChecked(CFIndex i) API_AVAILABLE(macos(10.0)) API_UNAVAILABLE(ios, watchos, tvos) {return ((CFOptionFlags)(1UL << (8 + i)));}
| ^~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/9787982387/job/27025351601?pr=14096#step:7:18
The gcc vs. llvm/clang incompatibility possibly tracked here upstream:
https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info:
https://github.com/llvm/llvm-project/issues/81767
https://github.com/gcc-mirror/gcc/commit/8433baadec88e5f31fa141b6d78094e91256079d
https://discourse.llvm.org/t/changing-attribute-ast-printing-location-for-gcc-compatibility/73215
https://reviews.llvm.org/D159362
Viktor Szakats [Tue, 9 Jul 2024 10:11:16 +0000 (12:11 +0200)]
cmake: detect `libidn2` also via `pkg-config`
Also:
- GHA/non-native: install `pkg-config` to detect libidn2 with cmake
on NetBSD and FreeBSD.
- GHA/non-native: tidy-up `curl --version` command if here.
Viktor Szakats [Tue, 9 Jul 2024 01:32:08 +0000 (03:32 +0200)]
build: fix llvm 16 or older + Xcode 15 or newer, and gcc
Xcode v15 (2023) or newer requires the built-in macro
`__ENVIRONMENT_OS_VERSION_MIN_REQUIRED__`. This macro is missing from
mainline llvm versions released earlier. llvm v17 introduced it here:
https://github.com/llvm/llvm-project/commit/c8e2dd8c6f490b68e41fe663b44535a8a21dfeab
This patch defines the missing macro when the necessary conditions
align, by using the value via the macro's old name.
The issue affected SecureTransport builds: The SecureTransport code,
`lib/md4.c` and `lib/md5.c`.
Existing gcc versions (as of v14) also don't define this macro, so apply
the patch to it as well. Even though gcc is incompatible in other ways,
so this isn't fixing an actual curl build case that I could find yet.
GHA macOS runner images have llvm v15 pre-installed, which broke builds
when building with an affected Xcode:
```
curl/lib/md4.c:80:14: error: '__ENVIRONMENT_OS_VERSION_MIN_REQUIRED__' is not defined, evaluates to 0 [-Werror,-Wundef]
(__MAC_OS_X_VERSION_MIN_REQUIRED < 101500)) || \
^
/Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/AvailabilityInternal.h:40:53: note: expanded from macro '__MAC_OS_X_VERSION_MIN_REQUIRED'
#define __MAC_OS_X_VERSION_MIN_REQUIRED __ENVIRONMENT_OS_VERSION_MIN_REQUIRED__
^
In file included from curl/build/lib/CMakeFiles/libcurl_shared.dir/Unity/unity_0_c.c:250:
curl/lib/md5.c:75:14: error: '__ENVIRONMENT_OS_VERSION_MIN_REQUIRED__' is not defined, evaluates to 0 [-Werror,-Wundef]
(__MAC_OS_X_VERSION_MIN_REQUIRED < 101500)) || \
^
/Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX14.2.sdk/usr/include/AvailabilityInternal.h:40:53: note: expanded from macro '__MAC_OS_X_VERSION_MIN_REQUIRED'
#define __MAC_OS_X_VERSION_MIN_REQUIRED __ENVIRONMENT_OS_VERSION_MIN_REQUIRED__
^
2 errors generated.
```
Ref: https://github.com/curl/curl/actions/runs/9811974634/job/27095218578#step:4:20
Viktor Szakats [Sat, 6 Jul 2024 21:47:50 +0000 (23:47 +0200)]
configure: limit `SystemConfiguration` test to non-c-ares, IPv6 builds
The framework this check detects is necessary for the function
`SCDynamicStoreCopyProxies()` used in `lib/macos.c`. Non-c-ares,
IPv6-enabled builds touch this codepath.
Limit the feature check for builds that actually need it.
It brings this in sync with CMake which already worked this way.
Viktor Szakats [Sat, 6 Jul 2024 21:47:50 +0000 (23:47 +0200)]
configure: fix `SystemConfiguration` detection
Before this patch, `SystemConfiguration` detection failed due to this
error when compiling the detection snippet:
```
/Applications/Xcode_15.3.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/TargetConditionals.h:140:50: error: missing binary operator before token "("
140 | #if !defined(__has_extension) || !__has_extension(define_target_os_macros)
| ^
```
Ref: https://github.com/curl/curl/actions/runs/9821817534/job/27117929218#step:6:1079
It occured with gcc-11 when combined with macOS SDK 14.4 and 14.5
(default SDKs in Xcode 15.3 and 15.4 respectively). It did not happen
with earlier releases.
Despite the failure in `./configure`, `lib/macos.c` compiled with
Apple's `TargetConditionals.h` just fine.
Turns out that including the `sys/types.h` header before the SDK
header fixes the error and makes the detection snippet compile.
Alex Snast [Sun, 7 Jul 2024 09:18:28 +0000 (12:18 +0300)]
wolfssl: use larger error buffer when formatting errors
Currently we're using WOLFSSL_MAX_ERROR_SZ to define the error buffer
size, this value is user defined which means it can be overwritten with
-DWOLFSSL_MAX_ERROR_SZ=512 when building wolfssl and this overwrite is
not exported to the users of wolfssl.
Instead of relying on WOLFSSL_MAX_ERROR_SZ we'll just use a 256 bytes
error buffer and use wolfSSL_ERR_error_string_n to fill it thus dropping
the dependency on WOLFSSL_MAX_ERROR_SZ altogether.
projects / thirdparty / curl.git / log
