Dan Fandrich [Tue, 20 Aug 2024 17:56:59 +0000 (10:56 -0700)]
build: improve compiler version detection portability
POSIX sed doesn't support extended regular expressions, so convert a
call to the basic format. This caused a problem on AIX. Also, use the
detected sed binary name instead of hard-coding one.
Jan Venekamp [Tue, 20 Aug 2024 00:53:19 +0000 (02:53 +0200)]
tests: improve test_17_07_ssl_ciphers
Change TLS proto version on the test httpd server to test setting
combinations of --tls13-ciphers and --ciphers.
To not let the changed config of the httpd server bleed into the next
test, clean and reload on each test. Because a reload is slow, only
do this if the config is different than the loaded config. For this
the httpd.reload_if_config_changed() method is added.
Overloading of autouse fixtures does not seem to work. For the test
httpd server to be reloaded with a clean config in test_18_methods,
to not be affected by the config changes in test_17_ssl_use, the two
class scope fixtures of test_18_methods are now combined.
Jan Venekamp [Mon, 19 Aug 2024 01:59:08 +0000 (03:59 +0200)]
mbedtls: no longer use MBEDTLS_SSL_VERIFY_OPTIONAL
With mbedTLS if the minimum version of TLS is set to 1.3,
MBEDTLS_SSL_VERIFY_OPTIONAL is not available in client mode. See:
https://github.com/Mbed-TLS/mbedtls/blob/2ca6c285/library/ssl_tls.c#L1357
Also, there might be plans to remove it completely in future mbedTLS
versions.
Switch to always use MBEDTLS_SSL_VERIFY_REQUIRED. If verifypeer or
verifyhost are disabled the corresponding error flags are cleared in the
verify callback function. That is also where verification errors are
logged.
Viktor Szakats [Mon, 19 Aug 2024 12:27:51 +0000 (14:27 +0200)]
cmake: migrate dependency detections to Find modules
For: libgsasl, libidn2, libssh, libuv.
The new Find modules retain using `pkg-config` natively, not as a "hint"
for the CMake-native detection. Of the pre-existing Find modules, only
FindNettle, and FindGSS (with customized code) work this way. Align
detection code for the new modules and add version detection for the
CMake-native paths.
Also, add CMake-native detection for `libgsasl`.
The remaining outlier in `CMakeLists.txt` is GnuTLS, which has
a CMake built-in Find module, but which lacks `pkg-config` support,
required for vcpkg. It remains unchanged.
Another part-outlier is `libssh`, which keeps requiring the trick
`find_package(libssh CONFIG QUIET)` for reasons I could not yet figure
out.
Viktor Szakats [Mon, 19 Aug 2024 23:13:14 +0000 (01:13 +0200)]
cmake: add `find_package()` missing from `USE_MSH3` option
The original patch added the Find module and CMake option. But the logic
missed a `find_package(MSH3)` call to use that Find module, leaving the
referenced `MSH3_INCLUDE_DIRS`, `MSH3_LIBRARIES` variables undefined.
Viktor Szakats [Fri, 16 Aug 2024 23:08:45 +0000 (01:08 +0200)]
cmake: limit `pkg-config` to UNIX and MSVC+vcpkg by default
Limits `pkg-config` to UNIX and MSVC with vcpkg, by default. Compared to
curl 8.9.1, this unlocks `pkg-config` on MSVC with vcpkg.
This condition might be updated in the future depending on where
`pkg-config` can be useful without breaking things. (e.g. to non-cross
MINGW, or all MINGW).
In the meantime everyone is free to override the default and test their
build with `pkg-config` by setting the `CURL_USE_PKGCONFIG=ON` CMake
option.
Our convention for naming Find modules (the part after the `Find`
prefix, also called as 'package name') is:
Always start with uppercase. Follow with lowercase, unless there is
a clear preference for a stylized name. E.g. the project itself uses it
that way with a matching `<Name>Config.cmake` file, or we use it that
way elsewhere, or the name is an acronym.
Viktor Szakats [Mon, 19 Aug 2024 10:31:11 +0000 (12:31 +0200)]
cmake: fix Find module and package names
- fix BearSSL warning about name mismatch.
- fix Nettle Find module not found on Linux.
- tidy-up: drop quotes from a package name.
Package names must match case-sensitively to work on all platforms:
- `find_package(<NAME> ...)` in `CMakeLists.txt`.
- `CMake/Find<NAME>.cmake` filenames.
- `find_package_handle_standard_args(<NAME> ...` in Find modules.
- `message(STATUS "Found <NAME> ...` in Find modules.
(to match the message shown by `find_package_handle_standard_args()`)
Daniel Stenberg [Mon, 19 Aug 2024 12:17:55 +0000 (14:17 +0200)]
build: use -Wno-format-overflow
-Wformat-overflow is not a warning that we want enabled as it does not
help us. It can only bring us false positives since it warns on bad uses
of sprintf and vsprintf ("that might overflow the destination buffer").
Two functions we explicitly ban in curl code.
The only way this flag triggers warnings in curl code is false positives
for functions we have marked with the CURL_PRINTF() macro.
Further: it seems -Wformat-trunaction option might in turn also enable
-Wformat-overflow, so if this second option is used, we need to
explicitly set -Wno-format-overflow - not just skip setting
-Wformat-overflow.
Reported-by: Viktor Szakats
Fixes #14168
Closes #14598
Viktor Szakats [Sat, 17 Aug 2024 22:10:39 +0000 (00:10 +0200)]
cmake/FindNettle: skip `pkg-config` for custom configs
If either `NETTLE_INCLUDE_DIR` or `NETTLE_LIBRARY` is set to customize
the `nettle` dependency, skip `pkg-config` and use the CMake-native
detection to honor these custom settings.
Viktor Szakats [Sat, 17 Aug 2024 21:39:49 +0000 (23:39 +0200)]
cmake: tidy up more in Find modules
- add `NAMES` where missing.
- document input variables (including deprecated ones.)
- comment cleanups.
- FindWolfSSL: drop stray `QUIET` from `pkg_check_modules()`.
(`QUIET` may be re-added for all modules in the future.)
Viktor Szakats [Sun, 18 Aug 2024 07:26:22 +0000 (09:26 +0200)]
appveyor: drop uploading artifacts
Uploading artifacts sometimes results in this error:
```
Uploading artifacts...
[1/1] _bld\src\curl.exe (2,022,912 bytes)...100%
Error uploading artifact to the storage: Remote server returned 503: Service Temporarily Unavailable
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/50424126/job/e4envval6xkicv1i#L123
The artifacts are also probably not useful to upload for every run. Also
note that they were missing external DLL dependencies.
Leave the logic there commented, to make it easy to enable as needed for
debugging or testing artifacts locally.
- move `m` library detection to wolfSSL Find module.
`m` is necessary for wolfSSL (wolfcrypt) library functions called by
`libngtcp2_crypto_wolfssl`.
Follow-up to 8577f4ca084b8a3926b869a48a29d41a810eceb5 #14343
- fix comment header about supported `COMPONENT` names.
Justin Maggard [Mon, 5 Aug 2024 17:21:35 +0000 (10:21 -0700)]
mbedtls: add more informative logging
After TLS handshare, indicate which TLS version was negotiated in
addition to the cipher in the handshake completed log message.
Also use the verify callback for certificate logging and collection.
This allows things to work even when MBEDTLS_SSL_KEEP_PEER_CERTIFICATE
is disabled in the mbedtls library.
And lastly, catch certificate validation errors later so we can give the
user more informative error messages that indicate what the failure was
from certificate validation.
Tested on both current LTS versions (2.28 and 3.6).
Stefan Eissing [Thu, 15 Aug 2024 11:16:53 +0000 (13:16 +0200)]
transfer: Curl_sendrecv() and event related improvements
- Renames Curl_readwrite() to Curl_sendrecv() to reflect that it
is mainly about talking to the server, not reads or writes to the
client. Add a `nowp` parameter since the single caller already
has this.
- Curl_sendrecv() now runs all possible operations whenever it is
called and either it had been polling sockets or the 'select_bits'
are set.
POLL_IN/POLL_OUT are not always directly related to send/recv
operations. Filters like HTTP/2, QUIC or TLS may monitor reverse
directions. If a transfer does not want to send (KEEP_SEND), it
will not do so, as before. Same for receives.
- Curl_update_timer() now checks the absolute timestamp of an expiry
and the last/new timeout to determine if the application needs
to stop/start/restart its timer. This fixes edge cases where
updates did not happen as they should have.
- improved --test-event curl_easy_perform() simulation to handle
situations where no sockets are registered but a timeout is
in place.
- fixed bug in events_socket() that complained about removing
a socket that was unknown, when indeed it had removed the socket
just before, only it was the last in the list
- fixed conncache's internal handle to carry the multi instance
(where the cache has one) so that operations on the closure handle
trigger event callbacks correctly.
- fixed conncache to not POLL_REMOVE a socket twice when a conneciton
was closed.
Viktor Szakats [Fri, 16 Aug 2024 13:04:25 +0000 (15:04 +0200)]
cmake: sync up version detection in Find modules
- use the same pattern across all Find modules:
- verify if the version header exists before reading it.
- use a single regex per lookup.
- sync regexes between Find modules.
- use generic temporary variable names.
- improve readability.
- make it simpler to transition to new CMake syntax in the future:
```cmake
file(STRINGS "${CARES_INCLUDE_DIR}/ares_version.h" _version_str REGEX "<...>")
unset(_version_str)
set(CARES_VERSION "${CMAKE_MATCH_1}")
```
Ref: https://cmake.org/cmake/help/latest/policy/CMP0159.html#policy:CMP0159
- fix zstd version detection to be CMake 3.7 compatible.
Required 3.9 before this patch, for the `CMAKE_MATCH_<n>` feature.
Follow-up to c5d506e9bbf0669d7605d1d00865ae7e229b2409 #12200
Viktor Szakats [Thu, 15 Aug 2024 08:47:54 +0000 (10:47 +0200)]
cmake: tidy-up continues
- move variable dump to a GHA foldable group.
- minimize scope for an include().
- rename `HIDES_CURL_PRIVATE_SYMBOLS` to `CURL_HIDES_PRIVATE_SYMBOLS`,
to keep it in the curl namespace.
- drop quotes from a version number.
- add missing `Makefile.inc` var refs to comment.
- FindNGTCP2: rename internal var to underscore/lowercase.
- FindBearSSL, FindGSS: whitespace.
Daniel Stenberg [Thu, 15 Aug 2024 08:04:03 +0000 (10:04 +0200)]
docs: remove ALTSVC.md, HSTS.md, HTTP2.md and PARALLEL-TRANSFERS.md
These are files and documentation for established functionality that
should by now be covered properly and completely in the standard
documentation and in everything curl. Having these extra files provides
duplicated information where they risk being out of sync.
Viktor Szakats [Thu, 15 Aug 2024 21:38:10 +0000 (23:38 +0200)]
cmake: add wolfSSH support
Enable with CMake option `-DCURL_USE_WOLFSSH=ON`. Customize with
`-DWOLFSSH_INCLUDE_DIR=<path-to-wolfssh>/include`,
`-DWOLFSSH_LIBRARY=<path-to-wolfssh>/lib/libwolfssh.a`.
Daniel Stenberg [Wed, 14 Aug 2024 11:36:20 +0000 (13:36 +0200)]
managen: wordwrap long example lines in ASCII output
The entire ASCII version of the manpage word wraps at a fixed column,
while example command lines can easily go wider than so.
This change now makes manage work on wrapping long example command lines
to make them look nicer. And also to avoid triggering the build error
caused by too long lines in the output.
Quoted lines cannot be wrapped, so managen now errors out if they are
"too long". With this addition, the 'maxline' script is removed as it is
no longer needed.
Daniel Stenberg [Thu, 15 Aug 2024 09:06:38 +0000 (11:06 +0200)]
multi: on socket callback error, remove socket hash entry nonetheless
Previously an error from the callback accidentally made libcurl keep the
hash entry which would lead to the entry remaining and then potentially
getting removed *again* which would lead to internal confusions.
Viktor Szakats [Wed, 14 Aug 2024 09:07:01 +0000 (11:07 +0200)]
cmake: fix and tidy up c-ares builds, enable in more CI jobs
- set `CARES_NO_DEPRECATED` to avoid c-ares deprecation warnings.
Like autotools does.
- drop unused c-ares header directory when building libtests and test
servers.
- disable TrackMemory (aka `CURLDEBUG`) feature for MSVC builds.
It fails on test 558 and 1330, saying that TrackMemory isn't working.
Left a FIXME about it.
- GHA/macos: enable c-ares in a cmake job.
- GHA/windows: enable c-ares in MSVC job.
Fixes #14202
- GHA/windows: add c-ares mingw autotools job with tests.
(move `--with-windows-unicode` option from 'default' job to
this one to keep the former "default".)
Put these tests on ignore for now:
```
FAIL 472: 'aws-sigv4 with query' HTTP, aws-sigv4
FAIL 1299: 'Send "OPTIONS *" with --request-target' HTTP, --request-target
FAIL 1613: 'Send "OPTIONS *" with --request-target to a proxy' HTTP, HTTP proxy, --request-target
```
Ref: https://github.com/curl/curl/actions/runs/10388126947/job/28765761441#step:15:5902
472: fails with mingw despite Unicode enabled.
1299, 1613: seem like a case of expanding '*' to a filename.
- sync cmake macOS `OS` value manually with the autotools one.
- stop exporting/subst-ing `HAVE_OPENSSL_QUIC` from autotools.
The variable was only used internally.
- exclude a dependency detection symbol.
- allow to run when the workflow itself was updated.
- simplify cmake command.
- fix indentation.
Tal Regev [Sat, 10 Aug 2024 08:15:25 +0000 (11:15 +0300)]
GHA/windows: add GnuTLS to MSVC jobs
- enable GnuTLS in MultiSSL job.
- add new GnuTLS job with HTTP/3 enabled.
Replacing the `!ssl` job.
Use `Release` mode to work around a vcpkg package issue, ending up
depending on both the debug and release versions of the gnutls DLL,
one of them miss to copy next to the binary.
- cmake: add `pkg-config` detection for GnuTLS.
Co-authored-by: Tal Regev
Fixes #14494
Closes #14495
Viktor Szakats [Tue, 13 Aug 2024 20:31:21 +0000 (22:31 +0200)]
cmake: tidy up Find modules
Smoothen out minor differences between Find modules.
- brotli, nghttp2: drop redundant `FOUND_VAR` specifiers from
`find_package_handle_standard_args()` calls.
This function sets both `<NAME_UPPER>_FOUND` and `<NAME>_FOUND`
by default.
- brotli: set result vars only when found.
- brotli: add missing `mark_as_advanced()` call.
- brotli: delete custom fail message.
- mbedtls, bearssl: use `REQUIRED_VARS` instead of `DEFAULT_MSG`.
- msh3, quiche: set `<NAME>_VERSION` (via pkg-config).
- wolfssl: also use `PC_WOLFSSL_INCLUDEDIR`, `PC_WOLFSSL_LIBDIR`
as hints.
- libpsl, libssh2, zstd: clear temporary variables used for version
detection.
- gss, msh3, nghttp2, nghttp3, ngtcp2, quiche, zstd: fix to apply
`mark_as_advanced()` to internal variables only.
Stefan Eissing [Mon, 12 Aug 2024 09:28:19 +0000 (11:28 +0200)]
urldata: introduce `data->mid`, a unique identifier inside a multi
`data->id` is unique in *most* situations, but not in all. If a libcurl
application uses more than one connection cache, they will overlap. This
is a rare situations, but libcurl apps do crazy things. However, for
informative things, like tracing, `data->id` is superior, since it
assigns new ids in curl's serial curl_easy_perform() use.
Introduce `data->mid` which is a unique identifer inside one multi
instance, assigned on multi_add_handle() and cleared on
multi_remove_handle().
Use the `mid` in DoH operations and also in h2/h3 stream hashes.
Viktor Szakats [Thu, 8 Aug 2024 12:42:40 +0000 (14:42 +0200)]
runtests: log ignored but passed tests
List tests explicitly ignored, but passing. It can be useful to catch
ignore rules that are no longer necessary because the tests are not
failing anymore.
To not clutter the default view, hide this information under a GitHub
Actions foldable group.
Viktor Szakats [Sun, 11 Aug 2024 22:57:41 +0000 (00:57 +0200)]
lib: prefer `CURL_SHA256_DIGEST_LENGTH` over the unprefixed name
Already used in `vtls.h`. Prefer this curl-namespaced name over the
unprefixed `SHA256_DIGEST_LENGTH`. The latter is also defined by TLS
backends with a potential to cause issues.
Also stop relying on externel headers setting this constant. It's
already defined in `vtls.h` on curl's behalf, do this also for `lib`.
Viktor Szakats [Sun, 11 Aug 2024 23:02:37 +0000 (01:02 +0200)]
lib: avoid macro collisions between wolfSSL and GnuTLS headers
Both of these projects define the same `SHA*` macros via headers
included by curl (in MultiSSL builds, possibly only in Unity builds),
causing redefinition compiler warnings.
Fix it by disabling compatibility macros in wolfSSL.
```
Building C object lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c.o
In file included from _bld/lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c:202:
In file included from lib/http_aws_sigv4.c:33:
In file included from lib/curl_sha256.h:40:
In file included from /usr/local/Cellar/wolfssl/5.7.2/include/wolfssl/openssl/sha.h:30:
/usr/local/Cellar/wolfssl/5.7.2/include/wolfssl/wolfcrypt/sha256.h:117:13: warning: 'SHA256_BLOCK_SIZE' macro redefined [-Wmacro-redefined]
#define SHA256_BLOCK_SIZE WC_SHA256_BLOCK_SIZE
^
/usr/local/Cellar/nettle/3.10/include/nettle/sha2.h:70:9: note: previous definition is here
#define SHA256_BLOCK_SIZE 64
^
In file included from _bld/lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c:202:
In file included from lib/http_aws_sigv4.c:33:
In file included from lib/curl_sha256.h:40:
In file included from /usr/local/Cellar/wolfssl/5.7.2/include/wolfssl/openssl/sha.h:30:
[...]
#define SHA256_DIGEST_SIZE WC_SHA256_DIGEST_SIZE
#define SHA224_BLOCK_SIZE WC_SHA224_BLOCK_SIZE
#define SHA224_DIGEST_SIZE WC_SHA224_DIGEST_SIZE
#define SHA512_BLOCK_SIZE WC_SHA512_BLOCK_SIZE
#define SHA512_DIGEST_SIZE WC_SHA512_DIGEST_SIZE
#define SHA384_BLOCK_SIZE WC_SHA384_BLOCK_SIZE
#define SHA384_DIGEST_SIZE WC_SHA384_DIGEST_SIZE
```
Viktor Szakats [Sun, 11 Aug 2024 20:26:21 +0000 (22:26 +0200)]
lib: fix building with wolfSSL without DES support
E.g. with Homebrew wolfssl 5.7.2 on macOS:
```
In file included from _bld/lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c:85:
lib/curl_ntlm_core.c:157:27: error: unknown type name 'DES_key_schedule'
DES_key_schedule DESKEYARG(ks))
^
lib/curl_ntlm_core.c:159:3: error: use of undeclared identifier 'DES_cblock'
DES_cblock key;
^
[...]
```