Stefan Eissing [Thu, 1 Aug 2024 14:16:55 +0000 (16:16 +0200)]
tests/http: configure test httpd to honor client cipher order
Let the client, e.g. curl, influence the cipher selected in a TLS
handshake. TLS backends have different preferences and honor that
in httpd the same as Caddy does.
Also makes for a more fair compare of different TLS backends.
Daniel Stenberg [Thu, 1 Aug 2024 11:51:43 +0000 (13:51 +0200)]
dist: fix reproducible build from release tarball
Make it possible to rebuild an identical copy from a release tarball. It
was previously only possible from a checked out git repository.
- add release-tools.sh to dist
- keep Makefile.dist around to include it in dist
- regenerate tool_huge.c with the new version in dist
- fix the dist CI job to not do make clean like before
Daniel Stenberg [Thu, 1 Aug 2024 09:16:21 +0000 (11:16 +0200)]
GHA: scan git repository and detect unvetted binary files
The only binary-looking files that are accepted in the git repository
need to match the checksums in the sha256sum file
".github/scripts/binarycheck.sums".
This is done to make sure that no one has planted any hidden (encrypted)
potentially dangerous payload in the tree.
Tal Regev [Wed, 31 Jul 2024 18:49:51 +0000 (21:49 +0300)]
GHA/windows: remove vcpkg bin path in MSVC jobs
- the path is wrong, because we compile on debug, and we are using
the release bin path.
- the path is not needed, cmake curl copy the needed dlls to the
compilation cmake folder where the curl exe is found.
Viktor Szakats [Fri, 19 Jul 2024 20:42:42 +0000 (22:42 +0200)]
GHA/windows: timeout earlier with hung tests
Finishing tests takes on average 10 or less minutes depending on
platform. Reduce job step timeouts to reflect that. It helps
concluding hung/failed tests earlier, which allows to retry them
earlier.
This makes it more difficult to tell from a job if it hung or not,
because we lose the long runtime as a telltale sign. Let's see how it
works out and adjust as necessary.
Viktor Szakats [Wed, 31 Jul 2024 09:53:31 +0000 (11:53 +0200)]
cmake: distcheck for files in CMake subdir
- add CMake option to verify if the `CMake/*.cmake`, `CMake/*.in` files
are listed as distributable in autotools' `EXTRA_DIST`. The check can
be enabled with `-DENABLE_DIST_TEST=ON` CMake option.
Tal Regev [Tue, 30 Jul 2024 03:07:50 +0000 (06:07 +0300)]
vtls: avoid forward declaration in MultiSSL builds
The MSVC compiler cannot have forward declaration with const and static
variable, causing this error:
```
curl\lib\vtls\vtls.c(417,44): warning C4132: 'Curl_ssl_multi': const object should be initialized
```
Viktor Szakats [Tue, 30 Jul 2024 08:52:32 +0000 (10:52 +0200)]
cmake: drop `if(PKG_CONFIG_FOUND)` guard for `pkg_check_modules()`
The oldest cmake supported by curl is v3.7.0, which already has such
guard (using `PKG_CONFIG_EXECUTABLE`) inside `pkg_check_modules()`. The
advantage of leaving that guard to CMake is that it will define/reset
all output variables, while the manual guard doesn't do this and also
leaves for example `NETTLE_FOUND` undefined.
Delete the single use of this guard from the recently added `nettle`
detection, where I included it by accident. Then possibly re-introduce
it universally if we find it useful after more evaluation.
Viktor Szakats [Sat, 20 Jul 2024 11:31:45 +0000 (13:31 +0200)]
GHA/non-native: improve, migrate x86_64 FreeBSD with tests from Cirrus CI
- run tests via `make test-ci` instead of `make check` with autotools.
- add `x86_64` job for FreeBSD, with tests.
It matches the existing Cirrus CI job, with these differences:
- finishes 3x faster (thanks to parallel tests enabled).
- librtmp is not enabled because it's slated for removal by FreeBSD.
(already past the removal deadline, thought the package still
installs.)
- DICT and TELNET servers fail to start. Couldn't figure out why.
It means skipping test 1450 and 1452.
- it runs more tests, e.g. websockets and ip6-localhost.
- no `pkg update -f`.
- it misses the `CRYPTOGRAPHY_DONT_BUILD_RUST=1`, `pkg delete curl`,
`chmod 777`, `sudo -u nobody` and `sysctl net.inet.tcp.blackhole`
tricks. The latter is the default in these runners, the others did
not affect results.
- set `-j0` for tests in the NetBSD job. Flaky otherwise.
Viktor Szakats [Thu, 25 Jul 2024 22:10:21 +0000 (00:10 +0200)]
macos: fix Apple SDK bug workaround for non-macOS targets
Turns out that MAC != OSX, despite what these names otherwise mean and
what's suggested by source code comments. "MAC" in fact means Darwin
(aka Apple), not macOS. "OSX" means macOS.
GitHub bumped the macos-14 runner default to Xcode 15.4, hitting the
llvm@15 incompatibility bug by default. Meaning the previous workaround
for the SDK bug is necessary.
This patch extend the workaround to not apply to mobile OS variants.
Stefan Eissing [Fri, 26 Jul 2024 13:07:52 +0000 (15:07 +0200)]
wolfssl: CA store share fix
When sharing the x509 store in wolfSSL, always use an explicitly
constructed one, as the SSLCTX might have "only" an internal one which
is not obeying reference count lifetimes.
Fixes #14278 Reported-by: Alex Snast
Closes #14279
curl: support __ss_family use on NonStop platforms
The definition of sockaddr_storage incorrectly specifies the ss_family
field as __ss_family. This fix conditionally allows builds to succeed on
all NonStop platforms.
Signed-off-by: Randall S. Becker <randall.becker@nexbridge.ca>
Closes #14273
Stefan Eissing [Mon, 29 Jul 2024 08:23:20 +0000 (10:23 +0200)]
connect: fix connection shutdown for event based processing
connections being shutdown would register sockets for events, but then
never remove these sockets again. Nor would the shutdown effectively
been performed.
- If a socket event involves a transfer, check if that is the
connection cache internal handle and run its multi_perform()
instead (the internal handle is used for all shutdowns).
- When a timer triggers for a transfer, check also if it is
about the connection cache internal handle.
- During processing shutdowns in the connection cache, assess
the shutdown timeouts. Register a Curl_expire() of the lowest
value for the cache's internal handle.
Reported-by: Gordon Parke
Fixes #14280
Closes #14296
Daniel Stenberg [Mon, 29 Jul 2024 08:17:08 +0000 (10:17 +0200)]
tests: provide FTP directory contents in the test file
Instead of providing a fixed single synthetic response in the test
server itself. To allow us to better use *different* directory listings
in different test cases. In this change, most listings remain the same
as before.
The wildcard match tests still use synthetic responses but we should fix
that as well.
Viktor Szakats [Sat, 20 Jul 2024 16:20:06 +0000 (18:20 +0200)]
runtests: fold timing details with GHA, sync `-r` tflags
- move timing details into a foldable group when run in GitHub Actions.
Spec:
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#grouping-log-lines
- enable `-r` (run time stats) option in autotools' `test-ci` target,
syncing it with cmake.
Stefan Eissing [Fri, 26 Jul 2024 08:38:45 +0000 (10:38 +0200)]
transfer: speed limiting fix for 32bit systems
When checking if a speed limit on receives applies, compare the receive
sizes using the large int type to prevent an overflow on systems where
size_t is 32bit.
- implement the socket hash user/reader/writer processing also
for connections that are being shut down by the connection cache.
- split out handling of current vs. last pollset socket event handling
into a function available in other code parts
- add `shutdown_poll` pollset to `connectdata` struct so that changes
in the pollset can be recorded during shutdown. (The internal handle
cannot keep it since it might be used for many connections)
Reported-by: calvin2021y on github
Fixes #14252
Closes #14257
Daniel Stenberg [Sat, 20 Jul 2024 21:21:16 +0000 (23:21 +0200)]
tool_cb_prg: output "flying saucers" with leading carriage return
Because that is how the progress-bar is output, so when the progress-bar
has been shown at least once and the information is reset, like for a
redirect, there might be a moment where the size goes from known to
unknown and then the flying saucerts are shown after a brief display of
the progress-bar.
It could previously cause accidental character leftovers on the right
side of the bar when using a narrow display.
Reported-by: Chris Webb
Fixes #14213
Closes #14246
md4: fix compilation with OpenSSL 1.x with md4 disabled
If OpenSSL 1.x is used, and it is configured with md4 disabled,
OPENSSL_NO_MD4 is defined in opensslconf.h, but this header was not
included before checking for this define.
Later in md4.c, openssl/md4.h is included, and it includes that header
indirectly, leading to inconsistency within md4.c.
Since the md4.h branch was taken, wincrypt.h (or others) is not
included, and later below the USE_WIN32_CRYPTO branch is taken, but the
types are not defined.
Viktor Szakats [Wed, 17 Jul 2024 23:09:04 +0000 (01:09 +0200)]
GHA/macos: improve, fix gcc/llvm, add new test matrix
This PR began as an attempt to drop GCC support, after repeated reports
on fallouts when trying to use it on macOS.
Then it transformed into a 3-week project turning up the issues causing
the fallouts, ending up including llvm and all available Xcode / macOS
SDK, macOS runner image, build tools and compiler vendors and versions.
Accumulating 400 sub-commits.
I developed and tested all fixes under this PR, then merged them as
separate patches.
This PR retained CI jobs updates, extensively reworking and extending
them: [1]
At first it seemed GCC and the Apple SDK is "naturally" growing more
incompatible, as Apple added further non-standard features to their
headers. This is partly true, but reality is more complicated.
Besides some issues local to curl, there were bugs in Apple SDK
headers, Homebrew GCC builds, feature missing in the old llvm version
pre-installed on GitHub CI runner images, and subtle incompatibilities
between GCC and llvm/clang when handling language extensions.
Resulting compiler errors seldom pointed to a useful direction, and
internet search was silent about these issues too. Thus, I had to peel
them off layer by layer, using trial and error, and by recognizing
patterns of failures accross 150-200 builds combinations. Exposing
configure logs, and curl_config.h in the CI logs helped too.
1. GCC header compatibility layer ("hack" as GCC calls it)
The toughest issue is GCC's built-in compatibility layer:
https://github.com/gcc-mirror/gcc/tree/master/fixincludes
This patch layer is further patched by a "Darwin compatibility" project
applied on top by Homebrew GCC via:
https://github.com/iains/gcc-12-branch
https://github.com/iains/gcc-13-branch
https://github.com/iains/gcc-14-branch
The hack layer is designed in a way that breaks more builds than it
fixes, esp. in context of GHA runners. The idea is to build GCC
specifically for the SDK for the target macOS version. The problem with
this approach is that the Xcode + SDK installed on the local/CI machine
often does not match with the SDK used on while building GCC on
Homebrew's build machines. In these cases the GCC compatibility layer
turns into an "uncompatibility" layer and consistently breaks builds.
curl cannot offer a fix for this, because the solution (I found) is to
patch the toolchain on the local machine. I implemented this for our CI
builds and curl-for-win. In other case the user must do this patching
manually, or choose a compatible GCC + Xcode/SDK combination.
An upstream fix doesn't seem trivial either, because the issue is
ingrained in the compatibility layer's design. Offering an `-fapplesdk`
(or recognizing `-target`) option and/or fixing them within the compiler
would seem like a more robust option, and also how mainline llvm solves
this.
Here's a table summarizing the GCC + SDK combinations and curl build
failures: [2]
More info: https://github.com/curl/curl/issues/10356#issuecomment-2222734103
A recent minor Homebrew GCC upgrade caused major breakage. The "Darwin
compatibility" patch applied to GCC implemented the `availability`
compiler attribute in GCC. Apple SDK detected this and enabled using
them, but as it turns out GCC accepts compiler attributes with slightly
different rules than llvm/clang, and how the Apple SDK uses them,
breaking builds.
Affected Homebrew GCC versions are: 12.4.0, 13.3.0 and 14.1.0.
Possibly tracked here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info: https://github.com/llvm/llvm-project/issues/81767
That applied to Homebrew GCC (12.4.0):
https://github.com/Homebrew/homebrew-core/commit/b904223d9893f62bec2a8f7483bf5992747fc6c7#diff-89dd0b4176eca7fcc24b591943509bf8a8d6ea904d71e5dfcd6b78fed62fc574R44-R48
Ref: #13700
More info: https://github.com/curl/curl/pull/14091#issuecomment-2222703468
Apple SDK expects certain macros predefined by the compiler. Missing
them may causes odd issues. Mainline llvm is keeping up with Apple
clang, but it needs a fresh version, while the one installed on GitHub
runners is old (v15). I patched these in `lib/curl_setup.h`.
Without certain predefined macros, SDK headers can take a codepath where
it mis-defines its own `TARGET_OS_OSX` macro, which make it break its
own headers later. I patched it in `lib/curl_setup.h`.
7. Differences between autotools and CMake dependency detection
Fixed it by improving detection of libidn2, with some more fixes
pending for the next feature window.
f43adc2c4978f7f82a359e89186e58a31d17b0ad #14137 cmake: detect `libidn2` also via `pkg-config`
Ref: #14136 cmake: detect `nghttp2` via `pkg-config`, enable by default
8. libidn2 detection bug with CMake
Fixed the root cause and also the trigger in the CI config.
curl source code assumes this is available to enable certain codepaths.
It's also intermixed with monotonic timer support.
14. Monotonic timer support with GCC
Detected by GCC, while it probably shouldn't be. llvm/clang detects it
depending on target OS version. I've been playing with this, but so far
without a conclusion or fix.
15. Runtime/test failures with GCC
I couldn't find the reason for most of this. A bunch of RTSP tests fail
with GCC. SecureTransport + HTTP/2 is failing a bunch of tests. With
OpenSSL it fails two of those. SecureTransport builds also fail one DoH
test.
16. Runtime/test failure in llvm/clang
AppleIDN support received a fix with two more remaining.
- #14185 runtests: fold test details for GitHub CI runs
- #14197 cmake: grab-bag of tidy-ups
- #14196 configure: limit `__builtin_available` test to Darwin
Summary:
In general GCC doesn't seem to be a good fit with curl and macOS for
now. These "lucky" combinations (GitHub Actions runner) will build out
of the box now: macos-14 + Xcode 15.0.1 + gcc-11, gcc-12, gcc-14. The
rest builds with the ugly workaround in place, but all this still leaves
some runtime issues.
More info and links in the commit messages and source code.
[1]: This PR:
- add info about target OS version requirements per feature, with OS
names and release years.
- stop using `-Wno-deprecated-declarations` to suppress warnings.
- use `LDFLAGS=-w` to suppress 'object file was built for newer macOS
version than being linked' warnings.
(there were tens of thousands of them in some jobs)
- allow overriding Xcode version in all jobs.
- improve job names.
- abbreviate CMake as CM, autotools as AM for more compact job names.
- shorten job names by using `!` instead of `no-` and `non-`.
- bump parellel tests to 10 (from 5).
- drop using `--enable-maintainer-mode` `./configure` option.
- add gcc-12 no-ssl, autotools job with tests, ignore failing test
results. (It's not yet clear why gcc-12 builds have different runtime
results than clang/llvm ones.)
- add comments with OS names and release years next to version numbers,
e.g. 10.15 # Catalina (2019)
- fix broken gcc-12 SecureTransport build.
- show compiler, Xcode, SDK, gcc hack SDK versions, Homebrew
preinstalled packages and C compiler predefined macros for each job.
Useful for debugging all the strange problems these builds might have.
- merge brew bundle and install steps.
- move step names to the top.
- dump configure log for both cmake and autotools also for successful
builds. Useful for debugging.
- dump curl_config.h in short (sorted #defines) and full form.
- add support for the mainline llvm compiler.
- set sysroot for gcc and llvm.
- add timeout for cmake jobs.
- add new job matrix: combinations
It supports building all possible compiler, runner image, Xcode/SDK
combinations, with cmake and autotools, target OS versions and with or
without SecureTransport. It's quick. GHA limits the maximum number of
matrix jobs at 256.
I used this as a test-rig to fix the macOS build fallouts with gcc and
llvm.
I settled with 16 jobs, trying to maximize fallout coverage.
- implement hack to make Homebrew gcc work with all available SDKs.
- add handy mini-table about Xcode / SDK versions, OS names, years for
each GHA images, with the defaults.
- add tests for cmake jobs.
- make cmake config hack to link GnuTLS less intrusive.
- stop ignoring test 1452, seems fine now.
- fix to enable libpsl in autotools builds.
- enable libpsl in cmake builds.
- add an llvm job with tests (both autotools and cmake).
- delete similar macOS jobs from Circle CI. GHA is now arm64 too.
[2]: Homebrew GCC vs GHA runner images vs curl builds:
```
macOS Xcode gcc gcc SDK hacks Xcode SDK SDK major Build Compile
(*def) (Homebrew) (CommandLineTools) versions error
-------- -------- ---------- ------------------ ---------- --------- ----- ---------------------
macos-12 13.1 GCC 11.4.0 MacOSX12 MacOSX12.0
macos-12 13.2.1 GCC 11.4.0 MacOSX12 MacOSX12.1
macos-12 13.3.1 GCC 11.4.0 MacOSX12 MacOSX12.3
macos-12 13.4.1 GCC 11.4.0 MacOSX12 MacOSX12.3
macos-12 14.0.1 GCC 11.4.0 MacOSX12 MacOSX12.3
macos-12 14.1 GCC 11.4.0 MacOSX12 MacOSX13.0 MISMATCH FAIL /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12 *14.2 GCC 11.4.0 MacOSX12 MacOSX13.1 MISMATCH FAIL /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13 14.1 GCC 11.4.0 MacOSX13 MacOSX13.0
macos-13 14.2 GCC 11.4.0 MacOSX13 MacOSX13.1
macos-13 14.3.1 GCC 11.4.0 MacOSX13 MacOSX13.3
macos-13 *15.0.1 GCC 11.4.0 MacOSX13 MacOSX14.0 MISMATCH FAIL /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13 15.1 GCC 11.4.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13 15.2 GCC 11.4.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-14 14.3.1 GCC 11.4.0 MacOSX14 MacOSX13.3 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 *15.0.1 GCC 11.4.0 MacOSX14 MacOSX14.0
macos-14 15.1 GCC 11.4.0 MacOSX14 MacOSX14.2
macos-14 15.2 GCC 11.4.0 MacOSX14 MacOSX14.2
macos-14 15.3 GCC 11.4.0 MacOSX14 MacOSX14.4
macos-14 15.4 GCC 11.4.0 MacOSX14 MacOSX14.5
macos-14 16.0 GCC 11.4.0 MacOSX14 MacOSX15.0 MISMATCH FAIL /opt/homebrew/Cellar/gcc@11/11.4.0/lib/gcc/11/gcc/aarch64-apple-darwin23/11/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12 13.1 GCC 12.4.0 MacOSX12 MacOSX12.0
macos-12 13.2.1 GCC 12.4.0 MacOSX12 MacOSX12.1
macos-12 13.3.1 GCC 12.4.0 MacOSX12 MacOSX12.3
macos-12 13.4.1 GCC 12.4.0 MacOSX12 MacOSX12.3
macos-12 14.0.1 GCC 12.4.0 MacOSX12 MacOSX12.3
macos-12 14.1 GCC 12.4.0 MacOSX12 MacOSX13.0 MISMATCH FAIL /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12 *14.2 GCC 12.4.0 MacOSX12 MacOSX13.1 MISMATCH FAIL /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13 14.1 GCC 12.4.0 MacOSX13 MacOSX13.0
macos-13 14.2 GCC 12.4.0 MacOSX13 MacOSX13.1
macos-13 14.3.1 GCC 12.4.0 MacOSX13 MacOSX13.3
macos-13 *15.0.1 GCC 12.4.0 MacOSX13 MacOSX14.0 MISMATCH FAIL /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13 15.1 GCC 12.4.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-13 15.2 GCC 12.4.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:103:1: error: unknown type name 'dispatch_queue_t'
macos-14 14.3.1 GCC 12.4.0 MacOSX14 MacOSX13.3 MISMATCH
macos-14 *15.0.1 GCC 12.4.0 MacOSX14 MacOSX14.0
macos-14 15.1 GCC 12.4.0 MacOSX14 MacOSX14.2
macos-14 15.2 GCC 12.4.0 MacOSX14 MacOSX14.2
macos-14 15.3 GCC 12.4.0 MacOSX14 MacOSX14.4
macos-14 15.4 GCC 12.4.0 MacOSX14 MacOSX14.5
macos-14 16.0 GCC 12.4.0 MacOSX14 MacOSX15.0 MISMATCH FAIL /opt/homebrew/Cellar/gcc@12/12.4.0/lib/gcc/12/gcc/aarch64-apple-darwin23/12/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12 13.1 GCC 13.3.0 MacOSX12 MacOSX12.0
macos-12 13.2.1 GCC 13.3.0 MacOSX12 MacOSX12.1
macos-12 13.3.1 GCC 13.3.0 MacOSX12 MacOSX12.3
macos-12 13.4.1 GCC 13.3.0 MacOSX12 MacOSX12.3
macos-12 14.0.1 GCC 13.3.0 MacOSX12 MacOSX12.3
macos-12 14.1 GCC 13.3.0 MacOSX12 MacOSX13.0 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-12 *14.2 GCC 13.3.0 MacOSX12 MacOSX13.1 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13 14.1 GCC 13.3.0 MacOSX13 MacOSX13.0
macos-13 14.2 GCC 13.3.0 MacOSX13 MacOSX13.1
macos-13 14.3.1 GCC 13.3.0 MacOSX13 MacOSX13.3
macos-13 *15.0.1 GCC 13.3.0 MacOSX13 MacOSX14.0 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13 15.1 GCC 13.3.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-13 15.2 GCC 13.3.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 14.3.1 GCC 13.3.0 MacOSX14 MacOSX13.3 MISMATCH FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 *15.0.1 GCC 13.3.0 MacOSX14 MacOSX14.0 FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 15.1 GCC 13.3.0 MacOSX14 MacOSX14.2 FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 15.2 GCC 13.3.0 MacOSX14 MacOSX14.2 FAIL /Users/runner/work/curl/curl/bld/lib/curl_config.h:792:19: error: two or more data types in declaration specifiers
macos-14 15.3 GCC 13.3.0 MacOSX14 MacOSX14.4
macos-14 15.4 GCC 13.3.0 MacOSX14 MacOSX14.5
macos-14 16.0 GCC 13.3.0 MacOSX14 MacOSX15.0 MISMATCH FAIL /opt/homebrew/Cellar/gcc@13/13.3.0/lib/gcc/13/gcc/aarch64-apple-darwin23/13/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
macos-12 13.1 GCC 14.1.0 MacOSX12 MacOSX12.0
macos-12 13.2.1 GCC 14.1.0 MacOSX12 MacOSX12.1
macos-12 13.3.1 GCC 14.1.0 MacOSX12 MacOSX12.3
macos-12 13.4.1 GCC 14.1.0 MacOSX12 MacOSX12.3
macos-12 14.0.1 GCC 14.1.0 MacOSX12 MacOSX12.3
macos-12 14.1 GCC 14.1.0 MacOSX12 MacOSX13.0 MISMATCH FAIL /Applications/Xcode_14.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-12 *14.2 GCC 14.1.0 MacOSX12 MacOSX13.1 MISMATCH FAIL /Applications/Xcode_14.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/os/object.h:275:1: error: expected ';' before 'extern'
macos-13 14.1 GCC 14.1.0 MacOSX13 MacOSX13.0
macos-13 14.2 GCC 14.1.0 MacOSX13 MacOSX13.1
macos-13 14.3.1 GCC 14.1.0 MacOSX13 MacOSX13.3
macos-13 *15.0.1 GCC 14.1.0 MacOSX13 MacOSX14.0 MISMATCH FAIL /Applications/Xcode_15.0.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-13 15.1 GCC 14.1.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.1.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-13 15.2 GCC 14.1.0 MacOSX13 MacOSX14.2 MISMATCH FAIL /Applications/Xcode_15.2.app/Contents/Developer/Platforms/MacOSX.platform/Developer/SDKs/MacOSX.sdk/usr/include/dispatch/queue.h:70:1: error: type defaults to 'int' in declaration of 'DISPATCH_DECL_FACTORY_CLASS_SWIFT' [-Wimplicit-int]
macos-14 14.3.1 GCC 14.1.0 MacOSX14 MacOSX13.3 MISMATCH
macos-14 *15.0.1 GCC 14.1.0 MacOSX14 MacOSX14.0
macos-14 15.1 GCC 14.1.0 MacOSX14 MacOSX14.2
macos-14 15.2 GCC 14.1.0 MacOSX14 MacOSX14.2
macos-14 15.3 GCC 14.1.0 MacOSX14 MacOSX14.4
macos-14 15.4 GCC 14.1.0 MacOSX14 MacOSX14.5
macos-14 16.0 GCC 14.1.0 MacOSX14 MacOSX15.0 MISMATCH FAIL /opt/homebrew/Cellar/gcc/14.1.0_1/lib/gcc/current/gcc/aarch64-apple-darwin23/14/include-fixed/stdio.h:83:8: error: unknown type name 'FILE'
```
Source: https://github.com/curl/curl/actions/runs/9883956647/job/27299564218
Viktor Szakats [Wed, 17 Jul 2024 23:09:04 +0000 (01:09 +0200)]
GHA/macos: improve, fix gcc/llvm, add new test matrix
This PR began as an attempt to drop GCC support, after repeated reports
on fallouts when trying to use it on macOS.
Then it transformed into a 3-week project turning up the issues causing
the fallouts, ending up including llvm and all available Xcode / macOS
SDK, macOS runner image, build tools and compiler vendors and versions.
Accumulating 400 sub-commits.
I developed and tested all fixes under this PR, then merged them as
separate patches.
This PR retained CI jobs updates, extensively reworking and extending
them: [1]
At first it seemed GCC and the Apple SDK is "naturally" growing more
incompatible, as Apple added further non-standard features to their
headers. This is partly true, but reality is more complicated.
Besides some issues local to curl, there were bugs in Apple SDK
headers, Homebrew GCC builds, feature missing in the old llvm version
pre-installed on GitHub CI runner images, and subtle incompatibilities
between GCC and llvm/clang when handling language extensions.
Resulting compiler errors seldom pointed to a useful direction, and
internet search was silent about these issues too. Thus, I had to peel
them off layer by layer, using trial and error, and by recognizing
patterns of failures accross 150-200 builds combinations. Exposing
configure logs, and curl_config.h in the CI logs helped too.
1. GCC header compatibility layer ("hack" as GCC calls it)
The toughest issue is GCC's built-in compatibility layer:
https://github.com/gcc-mirror/gcc/tree/master/fixincludes
This patch layer is further patched by a "Darwin compatibility" project
applied on top by Homebrew GCC via:
https://github.com/iains/gcc-12-branch
https://github.com/iains/gcc-13-branch
https://github.com/iains/gcc-14-branch
The hack layer is designed in a way that breaks more builds than it
fixes, esp. in context of GHA runners. The idea is to build GCC
specifically for the SDK for the target macOS version. The problem with
this approach is that the Xcode + SDK installed on the local/CI machine
often does not match with the SDK used on while building GCC on
Homebrew's build machines. In these cases the GCC compatibility layer
turns into an "uncompatibility" layer and consistently breaks builds.
curl cannot offer a fix for this, because the solution (I found) is to
patch the toolchain on the local machine. I implemented this for our CI
builds and curl-for-win. In other case the user must do this patching
manually, or choose a compatible GCC + Xcode/SDK combination.
An upstream fix doesn't seem trivial either, because the issue is
ingrained in the compatibility layer's design. Offering an `-fapplesdk`
(or recognizing `-target`) option and/or fixing them within the compiler
would seem like a more robust option, and also how mainline llvm solves
this.
Here's a table summarizing the GCC + SDK combinations and curl build
failures: [2]
More info: https://github.com/curl/curl/issues/10356#issuecomment-2222734103
A recent minor Homebrew GCC upgrade caused major breakage. The "Darwin
compatibility" patch applied to GCC implemented the `availability`
compiler attribute in GCC. Apple SDK detected this and enabled using
them, but as it turns out GCC accepts compiler attributes with slightly
different rules than llvm/clang, and how the Apple SDK uses them,
breaking builds.
Affected Homebrew GCC versions are: 12.4.0, 13.3.0 and 14.1.0.
Possibly tracked here: https://gcc.gnu.org/bugzilla/show_bug.cgi?id=108796
More info: https://github.com/llvm/llvm-project/issues/81767
That applied to Homebrew GCC (12.4.0):
https://github.com/Homebrew/homebrew-core/commit/b904223d9893f62bec2a8f7483bf5992747fc6c7#diff-89dd0b4176eca7fcc24b591943509bf8a8d6ea904d71e5dfcd6b78fed62fc574R44-R48
Ref: #13700
More info: https://github.com/curl/curl/pull/14091#issuecomment-2222703468
Apple SDK expects certain macros predefined by the compiler. Missing
them may causes odd issues. Mainline llvm is keeping up with Apple
clang, but it needs a fresh version, while the one installed on GitHub
runners is old (v15). I patched these in `lib/curl_setup.h`.
Without certain predefined macros, SDK headers can take a codepath where
it mis-defines its own `TARGET_OS_OSX` macro, which make it break its
own headers later. I patched it in `lib/curl_setup.h`.
7. Differences between autotools and CMake dependency detection
Fixed it by improving detection of libidn2, with some more fixes
pending for the next feature window.
f43adc2c4978f7f82a359e89186e58a31d17b0ad #14137 cmake: detect `libidn2` also via `pkg-config`
Ref: #14136 cmake: detect `nghttp2` via `pkg-config`, enable by default
8. libidn2 detection bug with CMake
Fixed the root cause and also the trigger in the CI config.
curl source code assumes this is available to enable certain codepaths.
It's also intermixed with monotonic timer support.
14. Monotonic timer support with GCC
Detected by GCC, while it probably shouldn't be. llvm/clang detects it
depending on target OS version. I've been playing with this, but so far
without a conclusion or fix.
15. Runtime/test failures with GCC
I couldn't find the reason for most of this. A bunch of RTSP tests fail
with GCC. SecureTransport + HTTP/2 is failing a bunch of tests. With
OpenSSL it fails two of those. SecureTransport builds also fail one DoH
test.
16. Runtime/test failure in llvm/clang
AppleIDN support received a fix with two more remaining.
Stefan Eissing [Thu, 18 Jul 2024 09:29:37 +0000 (11:29 +0200)]
lib: send eos flag
Adds a `bool eos` flag to send methods to indicate that the data is the
last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers are
not yet fully flushed.
Stefan Eissing [Wed, 17 Jul 2024 12:46:47 +0000 (14:46 +0200)]
doh: fix cleanup
When removing an easy handle that had DoH sub-easy handles going, those
were not removed from the multi handle. Their memory was reclaimed on
curl_easy_cleanup() of the owning handle, but multi still had them in
their list.
Add `Curl_doh_close()` and `Curl_doh_cleanup()` as common point for
handling the DoH resource management. Use the `multi` present in the doh
handles (if so), for removal, as the `data->multi` might already have
been NULLed at this time.
Alex Snast [Wed, 17 Jul 2024 11:06:06 +0000 (14:06 +0300)]
http/3: resume upload on ack if we have more data to send
Currently we're waiting for sendbuf_len_in_flight to hit zero before
resuming upload which means we're blocking and waiting for _all_ acks to
arrive before sending more data. This causes significant delays especially
when ack delay is used on the server side.
The fix addresses several issues in h3 over ngtcp2:
- On ack we now call nghttp3_conn_resume_stream() when we have more
data to send.
- upload_left was incorrectly computed on CF_CTRL_DATA_DONE_SEND as
we need to subtract the ammount of data we have in flight.
- Remove upload_blocked_len as we Curl_bufq_write call will do the
right thing when called from cf_ngtcp2_send.
Stefan Eissing [Wed, 17 Jul 2024 10:26:40 +0000 (12:26 +0200)]
pytests: scorecard upload tests
- add upload tests to scorecard, invoke with
> python3 tests/http/scorecard.py -u h1|h2|h3
- add a reverse proxy setup from Caddy to httpd for
upload tests since Caddy does not have other PUT/POST handling
- add caddy tests in test_08 for POST/PUT
- increase read buffer in mod_curltest for larger reads
Tal Regev [Mon, 15 Jul 2024 18:29:17 +0000 (21:29 +0300)]
GHA/windows: add MSVC wolfSSL job with test
Fix the file of wolfssl.c because of this warning/error:
```
curl\lib\vtls\wolfssl.c(1017,42): error C2220: the following warning is treated as an error [curl\bld\lib\libcurl_object.vcxproj]
curl\lib\vtls\wolfssl.c(1017,42): warning C4267: 'function': conversion from 'size_t' to 'unsigned long', possible loss of data [curl\bld\lib\libcurl_object.vcxproj]
```
`size_t` in MSVC is different. Change it to `unsigned long` because
`wolfSSL_ERR_error_string_n` last argument is defined as
`unsigned long`.
projects / thirdparty / curl.git / log
