Viktor Szakats [Fri, 16 Aug 2024 23:08:45 +0000 (01:08 +0200)]
cmake: limit `pkg-config` to UNIX and MSVC+vcpkg by default
Limits `pkg-config` to UNIX and MSVC with vcpkg, by default. Compared to
curl 8.9.1, this unlocks `pkg-config` on MSVC with vcpkg.
This condition might be updated in the future depending on where
`pkg-config` can be useful without breaking things. (e.g. to non-cross
MINGW, or all MINGW).
In the meantime everyone is free to override the default and test their
build with `pkg-config` by setting the `CURL_USE_PKGCONFIG=ON` CMake
option.
Our convention for naming Find modules (the part after the `Find`
prefix, also called as 'package name') is:
Always start with uppercase. Follow with lowercase, unless there is
a clear preference for a stylized name. E.g. the project itself uses it
that way with a matching `<Name>Config.cmake` file, or we use it that
way elsewhere, or the name is an acronym.
Viktor Szakats [Mon, 19 Aug 2024 10:31:11 +0000 (12:31 +0200)]
cmake: fix Find module and package names
- fix BearSSL warning about name mismatch.
- fix Nettle Find module not found on Linux.
- tidy-up: drop quotes from a package name.
Package names must match case-sensitively to work on all platforms:
- `find_package(<NAME> ...)` in `CMakeLists.txt`.
- `CMake/Find<NAME>.cmake` filenames.
- `find_package_handle_standard_args(<NAME> ...` in Find modules.
- `message(STATUS "Found <NAME> ...` in Find modules.
(to match the message shown by `find_package_handle_standard_args()`)
Daniel Stenberg [Mon, 19 Aug 2024 12:17:55 +0000 (14:17 +0200)]
build: use -Wno-format-overflow
-Wformat-overflow is not a warning that we want enabled as it does not
help us. It can only bring us false positives since it warns on bad uses
of sprintf and vsprintf ("that might overflow the destination buffer").
Two functions we explicitly ban in curl code.
The only way this flag triggers warnings in curl code is false positives
for functions we have marked with the CURL_PRINTF() macro.
Further: it seems -Wformat-trunaction option might in turn also enable
-Wformat-overflow, so if this second option is used, we need to
explicitly set -Wno-format-overflow - not just skip setting
-Wformat-overflow.
Reported-by: Viktor Szakats
Fixes #14168
Closes #14598
Viktor Szakats [Sat, 17 Aug 2024 22:10:39 +0000 (00:10 +0200)]
cmake/FindNettle: skip `pkg-config` for custom configs
If either `NETTLE_INCLUDE_DIR` or `NETTLE_LIBRARY` is set to customize
the `nettle` dependency, skip `pkg-config` and use the CMake-native
detection to honor these custom settings.
Viktor Szakats [Sat, 17 Aug 2024 21:39:49 +0000 (23:39 +0200)]
cmake: tidy up more in Find modules
- add `NAMES` where missing.
- document input variables (including deprecated ones.)
- comment cleanups.
- FindWolfSSL: drop stray `QUIET` from `pkg_check_modules()`.
(`QUIET` may be re-added for all modules in the future.)
Viktor Szakats [Sun, 18 Aug 2024 07:26:22 +0000 (09:26 +0200)]
appveyor: drop uploading artifacts
Uploading artifacts sometimes results in this error:
```
Uploading artifacts...
[1/1] _bld\src\curl.exe (2,022,912 bytes)...100%
Error uploading artifact to the storage: Remote server returned 503: Service Temporarily Unavailable
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/50424126/job/e4envval6xkicv1i#L123
The artifacts are also probably not useful to upload for every run. Also
note that they were missing external DLL dependencies.
Leave the logic there commented, to make it easy to enable as needed for
debugging or testing artifacts locally.
- move `m` library detection to wolfSSL Find module.
`m` is necessary for wolfSSL (wolfcrypt) library functions called by
`libngtcp2_crypto_wolfssl`.
Follow-up to 8577f4ca084b8a3926b869a48a29d41a810eceb5 #14343
- fix comment header about supported `COMPONENT` names.
Justin Maggard [Mon, 5 Aug 2024 17:21:35 +0000 (10:21 -0700)]
mbedtls: add more informative logging
After TLS handshare, indicate which TLS version was negotiated in
addition to the cipher in the handshake completed log message.
Also use the verify callback for certificate logging and collection.
This allows things to work even when MBEDTLS_SSL_KEEP_PEER_CERTIFICATE
is disabled in the mbedtls library.
And lastly, catch certificate validation errors later so we can give the
user more informative error messages that indicate what the failure was
from certificate validation.
Tested on both current LTS versions (2.28 and 3.6).
Stefan Eissing [Thu, 15 Aug 2024 11:16:53 +0000 (13:16 +0200)]
transfer: Curl_sendrecv() and event related improvements
- Renames Curl_readwrite() to Curl_sendrecv() to reflect that it
is mainly about talking to the server, not reads or writes to the
client. Add a `nowp` parameter since the single caller already
has this.
- Curl_sendrecv() now runs all possible operations whenever it is
called and either it had been polling sockets or the 'select_bits'
are set.
POLL_IN/POLL_OUT are not always directly related to send/recv
operations. Filters like HTTP/2, QUIC or TLS may monitor reverse
directions. If a transfer does not want to send (KEEP_SEND), it
will not do so, as before. Same for receives.
- Curl_update_timer() now checks the absolute timestamp of an expiry
and the last/new timeout to determine if the application needs
to stop/start/restart its timer. This fixes edge cases where
updates did not happen as they should have.
- improved --test-event curl_easy_perform() simulation to handle
situations where no sockets are registered but a timeout is
in place.
- fixed bug in events_socket() that complained about removing
a socket that was unknown, when indeed it had removed the socket
just before, only it was the last in the list
- fixed conncache's internal handle to carry the multi instance
(where the cache has one) so that operations on the closure handle
trigger event callbacks correctly.
- fixed conncache to not POLL_REMOVE a socket twice when a conneciton
was closed.
Viktor Szakats [Fri, 16 Aug 2024 13:04:25 +0000 (15:04 +0200)]
cmake: sync up version detection in Find modules
- use the same pattern across all Find modules:
- verify if the version header exists before reading it.
- use a single regex per lookup.
- sync regexes between Find modules.
- use generic temporary variable names.
- improve readability.
- make it simpler to transition to new CMake syntax in the future:
```cmake
file(STRINGS "${CARES_INCLUDE_DIR}/ares_version.h" _version_str REGEX "<...>")
unset(_version_str)
set(CARES_VERSION "${CMAKE_MATCH_1}")
```
Ref: https://cmake.org/cmake/help/latest/policy/CMP0159.html#policy:CMP0159
- fix zstd version detection to be CMake 3.7 compatible.
Required 3.9 before this patch, for the `CMAKE_MATCH_<n>` feature.
Follow-up to c5d506e9bbf0669d7605d1d00865ae7e229b2409 #12200
Viktor Szakats [Thu, 15 Aug 2024 08:47:54 +0000 (10:47 +0200)]
cmake: tidy-up continues
- move variable dump to a GHA foldable group.
- minimize scope for an include().
- rename `HIDES_CURL_PRIVATE_SYMBOLS` to `CURL_HIDES_PRIVATE_SYMBOLS`,
to keep it in the curl namespace.
- drop quotes from a version number.
- add missing `Makefile.inc` var refs to comment.
- FindNGTCP2: rename internal var to underscore/lowercase.
- FindBearSSL, FindGSS: whitespace.
Daniel Stenberg [Thu, 15 Aug 2024 08:04:03 +0000 (10:04 +0200)]
docs: remove ALTSVC.md, HSTS.md, HTTP2.md and PARALLEL-TRANSFERS.md
These are files and documentation for established functionality that
should by now be covered properly and completely in the standard
documentation and in everything curl. Having these extra files provides
duplicated information where they risk being out of sync.
Viktor Szakats [Thu, 15 Aug 2024 21:38:10 +0000 (23:38 +0200)]
cmake: add wolfSSH support
Enable with CMake option `-DCURL_USE_WOLFSSH=ON`. Customize with
`-DWOLFSSH_INCLUDE_DIR=<path-to-wolfssh>/include`,
`-DWOLFSSH_LIBRARY=<path-to-wolfssh>/lib/libwolfssh.a`.
Daniel Stenberg [Wed, 14 Aug 2024 11:36:20 +0000 (13:36 +0200)]
managen: wordwrap long example lines in ASCII output
The entire ASCII version of the manpage word wraps at a fixed column,
while example command lines can easily go wider than so.
This change now makes manage work on wrapping long example command lines
to make them look nicer. And also to avoid triggering the build error
caused by too long lines in the output.
Quoted lines cannot be wrapped, so managen now errors out if they are
"too long". With this addition, the 'maxline' script is removed as it is
no longer needed.
Daniel Stenberg [Thu, 15 Aug 2024 09:06:38 +0000 (11:06 +0200)]
multi: on socket callback error, remove socket hash entry nonetheless
Previously an error from the callback accidentally made libcurl keep the
hash entry which would lead to the entry remaining and then potentially
getting removed *again* which would lead to internal confusions.
Viktor Szakats [Wed, 14 Aug 2024 09:07:01 +0000 (11:07 +0200)]
cmake: fix and tidy up c-ares builds, enable in more CI jobs
- set `CARES_NO_DEPRECATED` to avoid c-ares deprecation warnings.
Like autotools does.
- drop unused c-ares header directory when building libtests and test
servers.
- disable TrackMemory (aka `CURLDEBUG`) feature for MSVC builds.
It fails on test 558 and 1330, saying that TrackMemory isn't working.
Left a FIXME about it.
- GHA/macos: enable c-ares in a cmake job.
- GHA/windows: enable c-ares in MSVC job.
Fixes #14202
- GHA/windows: add c-ares mingw autotools job with tests.
(move `--with-windows-unicode` option from 'default' job to
this one to keep the former "default".)
Put these tests on ignore for now:
```
FAIL 472: 'aws-sigv4 with query' HTTP, aws-sigv4
FAIL 1299: 'Send "OPTIONS *" with --request-target' HTTP, --request-target
FAIL 1613: 'Send "OPTIONS *" with --request-target to a proxy' HTTP, HTTP proxy, --request-target
```
Ref: https://github.com/curl/curl/actions/runs/10388126947/job/28765761441#step:15:5902
472: fails with mingw despite Unicode enabled.
1299, 1613: seem like a case of expanding '*' to a filename.
- sync cmake macOS `OS` value manually with the autotools one.
- stop exporting/subst-ing `HAVE_OPENSSL_QUIC` from autotools.
The variable was only used internally.
- exclude a dependency detection symbol.
- allow to run when the workflow itself was updated.
- simplify cmake command.
- fix indentation.
Tal Regev [Sat, 10 Aug 2024 08:15:25 +0000 (11:15 +0300)]
GHA/windows: add GnuTLS to MSVC jobs
- enable GnuTLS in MultiSSL job.
- add new GnuTLS job with HTTP/3 enabled.
Replacing the `!ssl` job.
Use `Release` mode to work around a vcpkg package issue, ending up
depending on both the debug and release versions of the gnutls DLL,
one of them miss to copy next to the binary.
- cmake: add `pkg-config` detection for GnuTLS.
Co-authored-by: Tal Regev
Fixes #14494
Closes #14495
Viktor Szakats [Tue, 13 Aug 2024 20:31:21 +0000 (22:31 +0200)]
cmake: tidy up Find modules
Smoothen out minor differences between Find modules.
- brotli, nghttp2: drop redundant `FOUND_VAR` specifiers from
`find_package_handle_standard_args()` calls.
This function sets both `<NAME_UPPER>_FOUND` and `<NAME>_FOUND`
by default.
- brotli: set result vars only when found.
- brotli: add missing `mark_as_advanced()` call.
- brotli: delete custom fail message.
- mbedtls, bearssl: use `REQUIRED_VARS` instead of `DEFAULT_MSG`.
- msh3, quiche: set `<NAME>_VERSION` (via pkg-config).
- wolfssl: also use `PC_WOLFSSL_INCLUDEDIR`, `PC_WOLFSSL_LIBDIR`
as hints.
- libpsl, libssh2, zstd: clear temporary variables used for version
detection.
- gss, msh3, nghttp2, nghttp3, ngtcp2, quiche, zstd: fix to apply
`mark_as_advanced()` to internal variables only.
Stefan Eissing [Mon, 12 Aug 2024 09:28:19 +0000 (11:28 +0200)]
urldata: introduce `data->mid`, a unique identifier inside a multi
`data->id` is unique in *most* situations, but not in all. If a libcurl
application uses more than one connection cache, they will overlap. This
is a rare situations, but libcurl apps do crazy things. However, for
informative things, like tracing, `data->id` is superior, since it
assigns new ids in curl's serial curl_easy_perform() use.
Introduce `data->mid` which is a unique identifer inside one multi
instance, assigned on multi_add_handle() and cleared on
multi_remove_handle().
Use the `mid` in DoH operations and also in h2/h3 stream hashes.
Viktor Szakats [Thu, 8 Aug 2024 12:42:40 +0000 (14:42 +0200)]
runtests: log ignored but passed tests
List tests explicitly ignored, but passing. It can be useful to catch
ignore rules that are no longer necessary because the tests are not
failing anymore.
To not clutter the default view, hide this information under a GitHub
Actions foldable group.
Viktor Szakats [Sun, 11 Aug 2024 22:57:41 +0000 (00:57 +0200)]
lib: prefer `CURL_SHA256_DIGEST_LENGTH` over the unprefixed name
Already used in `vtls.h`. Prefer this curl-namespaced name over the
unprefixed `SHA256_DIGEST_LENGTH`. The latter is also defined by TLS
backends with a potential to cause issues.
Also stop relying on externel headers setting this constant. It's
already defined in `vtls.h` on curl's behalf, do this also for `lib`.
Viktor Szakats [Sun, 11 Aug 2024 23:02:37 +0000 (01:02 +0200)]
lib: avoid macro collisions between wolfSSL and GnuTLS headers
Both of these projects define the same `SHA*` macros via headers
included by curl (in MultiSSL builds, possibly only in Unity builds),
causing redefinition compiler warnings.
Fix it by disabling compatibility macros in wolfSSL.
```
Building C object lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c.o
In file included from _bld/lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c:202:
In file included from lib/http_aws_sigv4.c:33:
In file included from lib/curl_sha256.h:40:
In file included from /usr/local/Cellar/wolfssl/5.7.2/include/wolfssl/openssl/sha.h:30:
/usr/local/Cellar/wolfssl/5.7.2/include/wolfssl/wolfcrypt/sha256.h:117:13: warning: 'SHA256_BLOCK_SIZE' macro redefined [-Wmacro-redefined]
#define SHA256_BLOCK_SIZE WC_SHA256_BLOCK_SIZE
^
/usr/local/Cellar/nettle/3.10/include/nettle/sha2.h:70:9: note: previous definition is here
#define SHA256_BLOCK_SIZE 64
^
In file included from _bld/lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c:202:
In file included from lib/http_aws_sigv4.c:33:
In file included from lib/curl_sha256.h:40:
In file included from /usr/local/Cellar/wolfssl/5.7.2/include/wolfssl/openssl/sha.h:30:
[...]
#define SHA256_DIGEST_SIZE WC_SHA256_DIGEST_SIZE
#define SHA224_BLOCK_SIZE WC_SHA224_BLOCK_SIZE
#define SHA224_DIGEST_SIZE WC_SHA224_DIGEST_SIZE
#define SHA512_BLOCK_SIZE WC_SHA512_BLOCK_SIZE
#define SHA512_DIGEST_SIZE WC_SHA512_DIGEST_SIZE
#define SHA384_BLOCK_SIZE WC_SHA384_BLOCK_SIZE
#define SHA384_DIGEST_SIZE WC_SHA384_DIGEST_SIZE
```
Viktor Szakats [Sun, 11 Aug 2024 20:26:21 +0000 (22:26 +0200)]
lib: fix building with wolfSSL without DES support
E.g. with Homebrew wolfssl 5.7.2 on macOS:
```
In file included from _bld/lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c:85:
lib/curl_ntlm_core.c:157:27: error: unknown type name 'DES_key_schedule'
DES_key_schedule DESKEYARG(ks))
^
lib/curl_ntlm_core.c:159:3: error: use of undeclared identifier 'DES_cblock'
DES_cblock key;
^
[...]
```
Viktor Szakats [Sun, 11 Aug 2024 19:23:44 +0000 (21:23 +0200)]
vtls: fix static function name collisions between TLS backends
When using CMake Unity build.
- use unique name for `set_ssl_version_min_max()`
Fixes collision between GnuTLS, mbedTLS and SecureTransport.
```
lib\vtls\mbedtls.c(317,1): error C2084: function 'CURLcode set_ssl_version_min_max(Curl_easy *,ssl_peer *,ssl_primary_config *,const char **,const char *)' already has a body
lib\vtls\mbedtls.c(837,49): warning C4133: 'function': incompatible types - from 'Curl_cfilter *' to 'Curl_easy *'
lib\vtls\mbedtls.c(837,53): warning C4133: 'function': incompatible types - from 'Curl_easy *' to 'ssl_peer *'
lib\vtls\mbedtls.c(837,25): error C2198: 'set_ssl_version_min_max': too few arguments for call
```
- use unique name for `do_file_type()`
Fixes collision between GnuTLS, OpenSSL and wolfSSL.
```
lib\vtls\openssl.c(1053,12): error C2084: function 'gnutls_x509_crt_fmt_t do_file_type(const char *)' already has a body
```
Ref: https://github.com/curl/curl/actions/runs/10341162641/job/28622681573?pr=14484#step:10:31
Cherry-picked from #14495
Closes #14516
Viktor Szakats [Mon, 12 Aug 2024 00:43:52 +0000 (02:43 +0200)]
build: silence C4232 MSVC warnings in vcpkg ngtcp2 builds
Silence bogus MSVC warning C4232. Use the method already used
for similar cases earlier.
Also fixup existing suppressions to use pragma push/pop.
```
lib\vquic\curl_ngtcp2.c(709,40): error C2220: the following warning is treated as an error
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'client_initial': address of dllimport 'ngtcp2_crypto_client_initial_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'recv_crypto_data': address of dllimport 'ngtcp2_crypto_recv_crypto_data_cb' is not static, identity not guaran
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'encrypt': address of dllimport 'ngtcp2_crypto_encrypt_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'decrypt': address of dllimport 'ngtcp2_crypto_decrypt_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'hp_mask': address of dllimport 'ngtcp2_crypto_hp_mask_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'recv_retry': address of dllimport 'ngtcp2_crypto_recv_retry_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'update_key': address of dllimport 'ngtcp2_crypto_update_key_cb' is not static, identity not guaranteed
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'delete_crypto_aead_ctx': address of dllimport 'ngtcp2_crypto_delete_crypto_aead_ctx_cb' is not static, identit
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'delete_crypto_cipher_ctx': address of dllimport 'ngtcp2_crypto_delete_crypto_cipher_ctx_cb' is not static, ide
lib\vquic\curl_ngtcp2.c(709,40): warning C4232: nonstandard extension used: 'get_path_challenge_data': address of dllimport 'ngtcp2_crypto_get_path_challenge_data_cb' is not static, ident
```
Ref: https://github.com/curl/curl/actions/runs/10343459009/job/28627621355#step:10:30
Cherry-picked from #14495 Co-authored-by: Tal Regev
Ref: #14383
Closes #14510
Viktor Szakats [Mon, 12 Aug 2024 13:35:50 +0000 (15:35 +0200)]
cmake: add `CURL_USE_PKGCONFIG` option
Add option to control whether to use `pkg-config` to detect
dependencies. Curl's CMake uses `pkg-config` by default for all targets
except for MSVC without vcpkg.
With the CMake option `-DCURL_USE_PKGCONFIG=ON` you can override it to
use `pkg-config` always.
If `pkg-config` is causing issues, e.g. in cross-builds or other cases,
`-DCURL_USE_PKGCONFIG=OFF` disables all use of `pkg-config`.
Also add it to `curl-config.cmake`. Not yet used, but will be once curl
starts referencing any curl-specific `Find*` module from this public
script.
Viktor Szakats [Mon, 12 Aug 2024 15:48:59 +0000 (17:48 +0200)]
IDN: fix/extend/migrate test exclusion rules
- make sure to exclude failing tests when libidn2 is detected by
default.
- ignore test 1560 results. Seen to fail with libidn2.
I'm not sure why this test was not executed earlier:
https://github.com/curl/curl/actions/runs/10354610889/job/28660309355#step:13:3647
- runtests: recognize `libidn2` as a feature.
- move IDN test exclusions from GHA/windows to `tests/data/DISABLED`.
- GHA/windows: drop default `-DUSE_LIBIDN2=ON` cmake config.
Jan Venekamp [Fri, 9 Aug 2024 12:29:18 +0000 (14:29 +0200)]
docs: update CIPHERS.md
Give a more consice overview of curl's cipher options and cipher suites.
Removed long lists of ciphers that were directly copied from the SSL
backends' documentation. Instead present the user a more common aproach
acorss the SSL backends, with notes for backends that do not conform
and/or provide alternate means.
Provide a shorter list of cipher suites that is more relevant for modern
usage and should work mostly across all backends, provide a seperate
list with all cipher suites, and provide links to the SSL backends'
documentation for more information.
Viktor Szakats [Mon, 12 Aug 2024 17:28:06 +0000 (19:28 +0200)]
GHA/macos: drop gcc-11
No longer present in macos-12, macos-13 images:
https://github.com/actions/runner-images/blob/macos-12/20240811.1/images/macos/macos-12-Readme.md
https://github.com/actions/runner-images/blob/macos-13/20240811.1/images/macos/macos-13-Readme.md