]> git.ipfire.org Git - thirdparty/curl.git/log
thirdparty/curl.git
15 months agobuild: improve compiler version detection portability
Dan Fandrich [Tue, 20 Aug 2024 17:56:59 +0000 (10:56 -0700)] 
build: improve compiler version detection portability

POSIX sed doesn't support extended regular expressions, so convert a
call to the basic format. This caused a problem on AIX. Also, use the
detected sed binary name instead of hard-coding one.

15 months agoGHA/windows: add missing time limit for msys2 autotools test runs
Viktor Szakats [Tue, 20 Aug 2024 13:59:32 +0000 (15:59 +0200)] 
GHA/windows: add missing time limit for msys2 autotools test runs

The new mingw job (the first msys2/mingw job to run tests) is seen
hanging frequently.

Follow-up to 9f23c8f201f55f1a148b41b16a5e71f3385faa5e #14541

15 months agotests: add test_17_09_ssl_min_max
Jan Venekamp [Tue, 20 Aug 2024 00:53:26 +0000 (02:53 +0200)] 
tests: add test_17_09_ssl_min_max

Test setting all combinations of --tlsv1.x and --tls-max.

Closes #14590

15 months agotests: improve test_17_07_ssl_ciphers
Jan Venekamp [Tue, 20 Aug 2024 00:53:19 +0000 (02:53 +0200)] 
tests: improve test_17_07_ssl_ciphers

Change TLS proto version on the test httpd server to test setting
combinations of --tls13-ciphers and --ciphers.

To not let the changed config of the httpd server bleed into the next
test, clean and reload on each test. Because a reload is slow, only
do this if the config is different than the loaded config. For this
the httpd.reload_if_config_changed() method is added.

Overloading of autouse fixtures does not seem to work. For the test
httpd server to be reloaded with a clean config in test_18_methods,
to not be affected by the config changes in test_17_ssl_use, the two
class scope fixtures of test_18_methods are now combined.

Closes #14589

15 months agombedtls: no longer use MBEDTLS_SSL_VERIFY_OPTIONAL
Jan Venekamp [Mon, 19 Aug 2024 01:59:08 +0000 (03:59 +0200)] 
mbedtls: no longer use MBEDTLS_SSL_VERIFY_OPTIONAL

With mbedTLS if the minimum version of TLS is set to 1.3,
MBEDTLS_SSL_VERIFY_OPTIONAL is not available in client mode. See:
https://github.com/Mbed-TLS/mbedtls/blob/2ca6c285/library/ssl_tls.c#L1357
Also, there might be plans to remove it completely in future mbedTLS
versions.

Switch to always use MBEDTLS_SSL_VERIFY_REQUIRED. If verifypeer or
verifyhost are disabled the corresponding error flags are cleared in the
verify callback function. That is also where verification errors are
logged.

Closes #14591

15 months agoGHA: update github/codeql-action digest to 883d858
renovate[bot] [Mon, 19 Aug 2024 21:21:56 +0000 (21:21 +0000)] 
GHA: update github/codeql-action digest to 883d858

Closes #14608

15 months agocmake: migrate dependency detections to Find modules
Viktor Szakats [Mon, 19 Aug 2024 12:27:51 +0000 (14:27 +0200)] 
cmake: migrate dependency detections to Find modules

For: libgsasl, libidn2, libssh, libuv.

The new Find modules retain using `pkg-config` natively, not as a "hint"
for the CMake-native detection. Of the pre-existing Find modules, only
FindNettle, and FindGSS (with customized code) work this way. Align
detection code for the new modules and add version detection for the
CMake-native paths.

Also, add CMake-native detection for `libgsasl`.

The remaining outlier in `CMakeLists.txt` is GnuTLS, which has
a CMake built-in Find module, but which lacks `pkg-config` support,
required for vcpkg. It remains unchanged.

Another part-outlier is `libssh`, which keeps requiring the trick
`find_package(libssh CONFIG QUIET)` for reasons I could not yet figure
out.

Closes #14555

15 months agocmake: add `find_package()` missing from `USE_MSH3` option
Viktor Szakats [Mon, 19 Aug 2024 23:13:14 +0000 (01:13 +0200)] 
cmake: add `find_package()` missing from `USE_MSH3` option

The original patch added the Find module and CMake option. But the logic
missed a `find_package(MSH3)` call to use that Find module, leaving the
referenced `MSH3_INCLUDE_DIRS`, `MSH3_LIBRARIES` variables undefined.

Blind fix.

Follow-up to 37492ebbfa24ba4e700e6655b3dbc2bdd65c894a #8517

Closes #14609

15 months agocf-socket: prevent KEEPALIVE_FACTOR being set to 1000 for Windows
Sam Jessup [Mon, 19 Aug 2024 15:37:59 +0000 (17:37 +0200)] 
cf-socket: prevent KEEPALIVE_FACTOR being set to 1000 for Windows

Fixes #14368
Reported-by: feelingseas on github
Closes #14606

15 months agocurl: find curlrc in XDG_CONFIG_HOME without leading dot
Daniel Stenberg [Mon, 19 Aug 2024 21:22:56 +0000 (23:22 +0200)] 
curl: find curlrc in XDG_CONFIG_HOME without leading dot

If XDG_CONFIG_HOME is set, look for XDG_CONFIG_HOME/curlrc - without a
leading dot in the filename.

Fixes #12129
Reported-by: Jat Satiro
Closes #14600

15 months agoGHA/windows: unblock TFTP MQTT WebSockets SMTP FTP tests
Viktor Szakats [Mon, 19 Aug 2024 17:05:36 +0000 (19:05 +0200)] 
GHA/windows: unblock TFTP MQTT WebSockets SMTP FTP tests

Run them now. Also ignore results for now.

Closes #14607

15 months agocmake: limit `pkg-config` to UNIX and MSVC+vcpkg by default
Viktor Szakats [Fri, 16 Aug 2024 23:08:45 +0000 (01:08 +0200)] 
cmake: limit `pkg-config` to UNIX and MSVC+vcpkg by default

Limits `pkg-config` to UNIX and MSVC with vcpkg, by default. Compared to
curl 8.9.1, this unlocks `pkg-config` on MSVC with vcpkg.

This condition might be updated in the future depending on where
`pkg-config` can be useful without breaking things. (e.g. to non-cross
MINGW, or all MINGW).

In the meantime everyone is free to override the default and test their
build with `pkg-config` by setting the `CURL_USE_PKGCONFIG=ON` CMake
option.

Closes #14575

15 months agocmake: rename Find modules
Viktor Szakats [Mon, 19 Aug 2024 22:47:55 +0000 (00:47 +0200)] 
cmake: rename Find modules

- `FindCARES`   -> `FindCares`
- `FindLibPSL`  -> `FindLibpsl`
- `FindLibSSH2` -> `FindLibssh2`
- `FindQUICHE`  -> `FindQuiche`
- `Findrustls`  -> `FindRustls`

Our convention for naming Find modules (the part after the `Find`
prefix, also called as 'package name') is:

Always start with uppercase. Follow with lowercase, unless there is
a clear preference for a stylized name. E.g. the project itself uses it
that way with a matching `<Name>Config.cmake` file, or we use it that
way elsewhere, or the name is an acronym.

Ref: #14580

Closes #14601

15 months agocmake: fix Find module and package names
Viktor Szakats [Mon, 19 Aug 2024 10:31:11 +0000 (12:31 +0200)] 
cmake: fix Find module and package names

- fix BearSSL warning about name mismatch.
- fix Nettle Find module not found on Linux.
- tidy-up: drop quotes from a package name.

Package names must match case-sensitively to work on all platforms:
- `find_package(<NAME> ...)` in `CMakeLists.txt`.
- `CMake/Find<NAME>.cmake` filenames.
- `find_package_handle_standard_args(<NAME> ...` in Find modules.
- `message(STATUS "Found <NAME> ...` in Find modules.
  (to match the message shown by `find_package_handle_standard_args()`)

Closes #14599

15 months agotidy-up: spelling quiche and Rustls
Viktor Szakats [Mon, 19 Aug 2024 14:53:40 +0000 (16:53 +0200)] 
tidy-up: spelling quiche and Rustls

Closes #14605

15 months agotidy-up: adjust casing of project names (continued)
Viktor Szakats [Mon, 19 Aug 2024 13:08:37 +0000 (15:08 +0200)] 
tidy-up: adjust casing of project names (continued)

Replace remaining `LibSSH2` with `libssh2`.

Follow-up to 6343034dd12fabfa88c998eb1182cbc6394afe25 #14160

Closes #14602

15 months agopingpong: drain the input buffer when reading responses
Daniel Stenberg [Mon, 19 Aug 2024 10:00:15 +0000 (12:00 +0200)] 
pingpong: drain the input buffer when reading responses

As the data might be held by TLS buffers, leaving some and expecting to
get called again is error prone.

Reported-by: ralfjunker on github
Fixes #14201
Closes #14597

15 months agoKNOWN_BUGS: Heimdal memory leaks
Daniel Stenberg [Mon, 19 Aug 2024 14:39:44 +0000 (16:39 +0200)] 
KNOWN_BUGS: Heimdal memory leaks

Closes #14446
Closes #14604

15 months agobuild: use -Wno-format-overflow
Daniel Stenberg [Mon, 19 Aug 2024 12:17:55 +0000 (14:17 +0200)] 
build: use -Wno-format-overflow

-Wformat-overflow is not a warning that we want enabled as it does not
help us. It can only bring us false positives since it warns on bad uses
of sprintf and vsprintf ("that might overflow the destination buffer").
Two functions we explicitly ban in curl code.

The only way this flag triggers warnings in curl code is false positives
for functions we have marked with the CURL_PRINTF() macro.

Further: it seems -Wformat-trunaction option might in turn also enable
-Wformat-overflow, so if this second option is used, we need to
explicitly set -Wno-format-overflow - not just skip setting
-Wformat-overflow.

Reported-by: Viktor Szakats
Fixes #14168
Closes #14598

15 months agocmake/FindNettle: log message when found via `pkg-config`
Viktor Szakats [Wed, 14 Aug 2024 21:27:33 +0000 (23:27 +0200)] 
cmake/FindNettle: log message when found via `pkg-config`

The message mimics the CMake-native message (by
`find_package_handle_standard_args()`), with the header path and version number.

Closes #14596

15 months agocmake: adjust GSSAPI option description
Viktor Szakats [Mon, 19 Aug 2024 09:12:15 +0000 (11:12 +0200)] 
cmake: adjust GSSAPI option description

krb5 also builds with CMake, not only Heimdal.

Ref: 558814e16d84aa202c5ccc0c8108a9d728e77a58

Closes #14595

15 months agoCI/azure: disable parallel tests, allow IDN tests
Viktor Szakats [Mon, 19 Aug 2024 07:28:17 +0000 (09:28 +0200)] 
CI/azure: disable parallel tests, allow IDN tests

They started show the similar flakiness as the GHA ones after enabling
parallel tests (`-j2`) by default.

Example flaky run:
https://dev.azure.com/daniel0244/curl/_build/results?buildId=24763&view=results

Ubuntu:
```
FAIL 137: 'FTP download without size in RETR string' FTP, RETR, --data-binary
FAIL 336: 'FTP range download when SIZE doesn't work' FTP, PASV, TYPE A, RETR
FAIL 975: 'HTTP with auth redirected to FTP allowing auth to continue' HTTP, FTP, --location-trusted
FAIL 1378: 'FTP DL, file without Content-Disposition inside, using -o fname' FTP, RETR
```

MSYS2 mingw32:
```
FAIL 1501: 'FTP with multi interface and slow LIST response' FTP, RETR, multi, LIST, DELAY
```

MSYS2 mingw64:
```
FAIL 1501: 'FTP with multi interface and slow LIST response' FTP, RETR, multi, LIST, DELAY
```

Follow-up to 0324d557e4b4f754ea89636ea9164065f6446560 #11510

Closes #14593

15 months agocmake/FindNettle: skip `pkg-config` for custom configs
Viktor Szakats [Sat, 17 Aug 2024 22:10:39 +0000 (00:10 +0200)] 
cmake/FindNettle: skip `pkg-config` for custom configs

If either `NETTLE_INCLUDE_DIR` or `NETTLE_LIBRARY` is set to customize
the `nettle` dependency, skip `pkg-config` and use the CMake-native
detection to honor these custom settings.

Closes #14584

15 months agombedtls: fix setting tls version
Jan Venekamp [Mon, 19 Aug 2024 01:59:02 +0000 (03:59 +0200)] 
mbedtls: fix setting tls version

TLS max values lower than 1.2 were automatically set to 1.2. Other SSL
backends (that dropped TLS 1.0 and 1.1) do not do that.

Closes #14588

15 months agowolfssl: fix setting tls version
Jan Venekamp [Mon, 19 Aug 2024 02:00:43 +0000 (04:00 +0200)] 
wolfssl: fix setting tls version

The value CURL_SSLVERSION_TLSv1_0 was unsupported.

Closes #14587

15 months agorustls: fix setting tls version
Jan Venekamp [Mon, 19 Aug 2024 02:02:12 +0000 (04:02 +0200)] 
rustls: fix setting tls version

The value CURL_SSLVERSION_TLSv1_0 was unsupported.

Closes #14586

15 months agobearssl: fix setting tls version
Jan Venekamp [Mon, 19 Aug 2024 02:03:16 +0000 (04:03 +0200)] 
bearssl: fix setting tls version

Previously version_max was ignored.

Closes #14585

15 months agoRELEASE-NOTES: synced
Daniel Stenberg [Sun, 18 Aug 2024 20:57:25 +0000 (22:57 +0200)] 
RELEASE-NOTES: synced

15 months agocmake: fix `cmakelint` warnings
Viktor Szakats [Sun, 18 Aug 2024 11:59:31 +0000 (13:59 +0200)] 
cmake: fix `cmakelint` warnings

- keep line lengths below 132 characters.
- fix two "weird indentation" warnings.

Reported-by: Dan Fandrich
Bug: #14580

Closes #14583

15 months agocmake: tidy up more in Find modules
Viktor Szakats [Sat, 17 Aug 2024 21:39:49 +0000 (23:39 +0200)] 
cmake: tidy up more in Find modules

- add `NAMES` where missing.
- document input variables (including deprecated ones.)
- comment cleanups.
- FindWolfSSL: drop stray `QUIET` from `pkg_check_modules()`.
  (`QUIET` may be re-added for all modules in the future.)

Closes #14579

15 months agoappveyor: drop uploading artifacts
Viktor Szakats [Sun, 18 Aug 2024 07:26:22 +0000 (09:26 +0200)] 
appveyor: drop uploading artifacts

Uploading artifacts sometimes results in this error:
```
Uploading artifacts...
[1/1] _bld\src\curl.exe (2,022,912 bytes)...100%
Error uploading artifact to the storage: Remote server returned 503: Service Temporarily Unavailable
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/50424126/job/e4envval6xkicv1i#L123

The artifacts are also probably not useful to upload for every run. Also
note that they were missing external DLL dependencies.

Leave the logic there commented, to make it easy to enable as needed for
debugging or testing artifacts locally.

Closes #14581

15 months agocmake: tidy up around ngtcp2 and wolfSSL
Viktor Szakats [Fri, 16 Aug 2024 23:41:23 +0000 (01:41 +0200)] 
cmake: tidy up around ngtcp2 and wolfSSL

- fix to add the `m` library without path.
  Follow-up to 8577f4ca084b8a3926b869a48a29d41a810eceb5 #14343
Authored-by: Tal Regev
  Fixes #14549

- move `m` library detection to wolfSSL Find module.
  `m` is necessary for wolfSSL (wolfcrypt) library functions called by
  `libngtcp2_crypto_wolfssl`.
  Follow-up to 8577f4ca084b8a3926b869a48a29d41a810eceb5 #14343

- fix comment header about supported `COMPONENT` names.

- quote strings.

- lowercase local variables.

Closes #14576

15 months agocmake: do not unset the deprecated mixed-case variables
Viktor Szakats [Sat, 17 Aug 2024 20:45:38 +0000 (22:45 +0200)] 
cmake: do not unset the deprecated mixed-case variables

To avoid interference with the calling env.

(Keep unsetting for the DIRS/DIR cases in BearSSL and mbedTLS, because
the deprecated variables play a new role in the detection.)

Follow-up to 9fbda4ca75483ee0a43289526e88d8f1e8ca2a78 #14574

15 months agocmake: rename wolfSSL and zstd config variables to uppercase
Viktor Szakats [Fri, 16 Aug 2024 16:56:49 +0000 (18:56 +0200)] 
cmake: rename wolfSSL and zstd config variables to uppercase

To match with other config variables and other projects.

Rename these CMake configuration variables:
- `WolfSSL_INCLUDE_DIR` -> `WOLFSSL_INCLUDE_DIR`
- `WolfSSL_LIBRARY`     -> `WOLFSSL_LIBRARY`
- `Zstd_INCLUDE_DIR`    -> `ZSTD_INCLUDE_DIR`
- `Zstd_LIBRARY`        -> `ZSTD_LIBRARY`

The old values continue to work, with a warning suggesting the new name.

Also:
- add similar warnings for earlier renames for mbedTLS and BearSSL.
- rename internal variables `PC_Zstd_*` to uppercase.

Follow-up to db39c668a8e33e064b9eb20892cd027f46302f77 #14542

Closes #14574

15 months agolocation: fix typo
Daniel Stenberg [Sat, 17 Aug 2024 19:04:34 +0000 (21:04 +0200)] 
location: fix typo

Follow-up to 5fcf96930efc
Bug: https://github.com/curl/curl/pull/14471#pullrequestreview-2244131475
Reported-by: Joshix-1 on github
15 months agodocs: add description of effect of --location-trusted on cookie
XYenon [Fri, 9 Aug 2024 09:30:40 +0000 (17:30 +0800)] 
docs: add description of effect of --location-trusted on cookie

Closes #14471

15 months agodocs: improve cipher options documentation
Jan Venekamp [Fri, 16 Aug 2024 09:45:26 +0000 (11:45 +0200)] 
docs: improve cipher options documentation

Closes #14407

15 months agoGHA: update github/codeql-action digest to 429e197
renovate[bot] [Wed, 14 Aug 2024 16:54:43 +0000 (16:54 +0000)] 
GHA: update github/codeql-action digest to 429e197

Closes #14425

15 months agoSECURITY: mention OpenSSF best practices gold badge
Viktor Petersson [Tue, 30 Jul 2024 17:24:13 +0000 (18:24 +0100)] 
SECURITY: mention OpenSSF best practices gold badge

Closes #14319

15 months agombedtls: add more informative logging
Justin Maggard [Mon, 5 Aug 2024 17:21:35 +0000 (10:21 -0700)] 
mbedtls: add more informative logging

After TLS handshare, indicate which TLS version was negotiated in
addition to the cipher in the handshake completed log message.

Also use the verify callback for certificate logging and collection.
This allows things to work even when MBEDTLS_SSL_KEEP_PEER_CERTIFICATE
is disabled in the mbedtls library.

And lastly, catch certificate validation errors later so we can give the
user more informative error messages that indicate what the failure was
from certificate validation.

Tested on both current LTS versions (2.28 and 3.6).

Closes #14444

15 months agoGHA: update dependency gnutls/gnutls to v3.8.7
renovate[bot] [Thu, 15 Aug 2024 08:18:03 +0000 (08:18 +0000)] 
GHA: update dependency gnutls/gnutls to v3.8.7

Closes #14554

15 months agotransfer: Curl_sendrecv() and event related improvements
Stefan Eissing [Thu, 15 Aug 2024 11:16:53 +0000 (13:16 +0200)] 
transfer: Curl_sendrecv() and event related improvements

- Renames Curl_readwrite() to Curl_sendrecv() to reflect that it
  is mainly about talking to the server, not reads or writes to the
  client. Add a `nowp` parameter since the single caller already
  has this.
- Curl_sendrecv() now runs all possible operations whenever it is
  called and either it had been polling sockets or the 'select_bits'
  are set.
  POLL_IN/POLL_OUT are not always directly related to send/recv
  operations. Filters like HTTP/2, QUIC or TLS may monitor reverse
  directions. If a transfer does not want to send (KEEP_SEND), it
  will not do so, as before. Same for receives.
- Curl_update_timer() now checks the absolute timestamp of an expiry
  and the last/new timeout to determine if the application needs
  to stop/start/restart its timer. This fixes edge cases where
  updates did not happen as they should have.
- improved --test-event curl_easy_perform() simulation to handle
  situations where no sockets are registered but a timeout is
  in place.
- fixed bug in events_socket() that complained about removing
  a socket that was unknown, when indeed it had removed the socket
  just before, only it was the last in the list
- fixed conncache's internal handle to carry the multi instance
  (where the cache has one) so that operations on the closure handle
  trigger event callbacks correctly.
- fixed conncache to not POLL_REMOVE a socket twice when a conneciton
  was closed.

Closes #14561

15 months agocmake: sync up version detection in Find modules
Viktor Szakats [Fri, 16 Aug 2024 13:04:25 +0000 (15:04 +0200)] 
cmake: sync up version detection in Find modules

- use the same pattern across all Find modules:
  - verify if the version header exists before reading it.
  - use a single regex per lookup.
  - sync regexes between Find modules.
  - use generic temporary variable names.
  - improve readability.
  - make it simpler to transition to new CMake syntax in the future:
    ```cmake
    file(STRINGS "${CARES_INCLUDE_DIR}/ares_version.h" _version_str REGEX "<...>")
    unset(_version_str)
    set(CARES_VERSION "${CMAKE_MATCH_1}")
    ```
    Ref: https://cmake.org/cmake/help/latest/policy/CMP0159.html#policy:CMP0159

- fix zstd version detection to be CMake 3.7 compatible.
  Required 3.9 before this patch, for the `CMAKE_MATCH_<n>` feature.
  Follow-up to c5d506e9bbf0669d7605d1d00865ae7e229b2409 #12200

Follow-up to 4e2f3641f80e233738147d82218317f331369c50 #14548

Closes #14572

15 months agocmake: tidy-up continues
Viktor Szakats [Thu, 15 Aug 2024 08:47:54 +0000 (10:47 +0200)] 
cmake: tidy-up continues

- move variable dump to a GHA foldable group.
- minimize scope for an include().
- rename `HIDES_CURL_PRIVATE_SYMBOLS` to `CURL_HIDES_PRIVATE_SYMBOLS`,
  to keep it in the curl namespace.
- drop quotes from a version number.
- add missing `Makefile.inc` var refs to comment.
- FindNGTCP2: rename internal var to underscore/lowercase.
- FindBearSSL, FindGSS: whitespace.

Closes #14571

15 months agocmake: revert to `pkg_check_modules()`
Viktor Szakats [Fri, 16 Aug 2024 16:19:25 +0000 (18:19 +0200)] 
cmake: revert to `pkg_check_modules()`

Prefer `pkg_check_modules()` over `pkg_search_module()`.

`pkg_check_modules()` logs a line when there is a hit, and also warnings
if a sub-dependency is missing. In `QUIET` mode, both are silent.

The extra info is useful to see if a detection happened via
`pkg-config`.

Keep `pkg_search_module()` in `FindGSS`. We pass two dependencies
there and we want to keep stopping on the first one.

Partially reverts c2889a7b4180fc963ae30811f59ab547b6eb03cd #14388

Closes #14573

15 months agocmake: fixup variable reference in FindZstd
Viktor Szakats [Fri, 16 Aug 2024 17:09:08 +0000 (19:09 +0200)] 
cmake: fixup variable reference in FindZstd

Follow-up to 4e2f3641f80e233738147d82218317f331369c50 #14548

15 months agointernals/SPLAY.md: internal API documentation
Daniel Stenberg [Thu, 15 Aug 2024 14:18:30 +0000 (16:18 +0200)] 
internals/SPLAY.md: internal API documentation

Closes #14563

15 months agocurl: make the progress bar detect terminal width changes
Daniel Stenberg [Fri, 16 Aug 2024 09:15:36 +0000 (11:15 +0200)] 
curl: make the progress bar detect terminal width changes

And up the widest supported bar to 400 columns.

Fixes #14565
Reported-by: lolbinarycat on github
Closes #14570

15 months agocmake: add missing version detection to Find modules
Viktor Szakats [Tue, 13 Aug 2024 20:59:50 +0000 (22:59 +0200)] 
cmake: add missing version detection to Find modules

- use `pkg-config` version when available and where it wasn't yet used.

- add manual version detection for dependencies where this is possible
  (via a public header) and where it wasn't done yet.

Closes #14548

15 months agoGHA/windows: delete redundant options, tidy up
Viktor Szakats [Fri, 16 Aug 2024 08:42:45 +0000 (10:42 +0200)] 
GHA/windows: delete redundant options, tidy up

- delete redundant `-DCURL_USE_LIBPSL=ON`.

- delete redundant `-DUSE_NGHTTP2=ON`.
  Follow-up to 87aa4ebd821ebae0023df8658360c724efcf5e00 #14136

- reorder options.

Closes #14569

15 months agotidy-up: misc build, tests, `lib/macos.c`
Viktor Szakats [Thu, 15 Aug 2024 08:47:37 +0000 (10:47 +0200)] 
tidy-up: misc build, tests, `lib/macos.c`

- mkhelp.pl: drop unused assigment.
- Makefile.mk: update a comment.
- lib/macos.c: delete redundant block.
- fix two typos.

Closes #14558

15 months agoRELEASE-NOTES: synced
Daniel Stenberg [Fri, 16 Aug 2024 12:52:34 +0000 (14:52 +0200)] 
RELEASE-NOTES: synced

15 months agodocs: remove ALTSVC.md, HSTS.md, HTTP2.md and PARALLEL-TRANSFERS.md
Daniel Stenberg [Thu, 15 Aug 2024 08:04:03 +0000 (10:04 +0200)] 
docs: remove ALTSVC.md, HSTS.md, HTTP2.md and PARALLEL-TRANSFERS.md

These are files and documentation for established functionality that
should by now be covered properly and completely in the standard
documentation and in everything curl. Having these extra files provides
duplicated information where they risk being out of sync.

Closes #14553

15 months agorustls: add support for setting TLS version and ciphers
Jan Venekamp [Tue, 13 Aug 2024 20:01:07 +0000 (22:01 +0200)] 
rustls: add support for setting TLS version and ciphers

Add support for CURLOPT_SSLVERSION, CURLOPT_TLS13_CIPHERS and
CURLOPT_SSL_CIPHER_LIST.

Closes #14535

15 months agocmake: add wolfSSH support
Viktor Szakats [Thu, 15 Aug 2024 21:38:10 +0000 (23:38 +0200)] 
cmake: add wolfSSH support

Enable with CMake option `-DCURL_USE_WOLFSSH=ON`. Customize with
`-DWOLFSSH_INCLUDE_DIR=<path-to-wolfssh>/include`,
`-DWOLFSSH_LIBRARY=<path-to-wolfssh>/lib/libwolfssh.a`.

It requires the wolfSSL TLS backend.

Closes #14568

15 months agocmake: TLS 1.3 warning only for bearssl and sectranp
Jan Venekamp [Thu, 15 Aug 2024 18:11:35 +0000 (20:11 +0200)] 
cmake: TLS 1.3 warning only for bearssl and sectranp

Closes #14566

15 months agosplay: use access functions, add asserts, use Curl_timediff
Daniel Stenberg [Thu, 15 Aug 2024 14:13:23 +0000 (16:13 +0200)] 
splay: use access functions, add asserts, use Curl_timediff

- add set/get functions for the custom data in a tree node

- use Curl_timediff for time comparisons instead of the custom macro, as they
  do the exact same things.

- add asserts to catch mistakes better

- updated test 1309 accordingly

Closes #14562

15 months agoscorecard: tweak request measurements
Stefan Eissing [Thu, 15 Aug 2024 14:52:40 +0000 (16:52 +0200)] 
scorecard: tweak request measurements

Increase max-parallel up to 300, the curl max

Tweak output to just give the http response code

Closes #14564

15 months agodocs/internals: new subdirectory
Daniel Stenberg [Thu, 15 Aug 2024 12:05:23 +0000 (14:05 +0200)] 
docs/internals: new subdirectory

For libcurl internal APIs and concepts.

Closes #14552

15 months agotest1707: output diff more for debugging differences in CI outputs
Daniel Stenberg [Thu, 15 Aug 2024 22:19:00 +0000 (00:19 +0200)] 
test1707: output diff more for debugging differences in CI outputs

15 months agomanagen: wordwrap long example lines in ASCII output
Daniel Stenberg [Wed, 14 Aug 2024 11:36:20 +0000 (13:36 +0200)] 
managen: wordwrap long example lines in ASCII output

The entire ASCII version of the manpage word wraps at a fixed column,
while example command lines can easily go wider than so.

This change now makes manage work on wrapping long example command lines
to make them look nicer. And also to avoid triggering the build error
caused by too long lines in the output.

Quoted lines cannot be wrapped, so managen now errors out if they are
"too long". With this addition, the 'maxline' script is removed as it is
no longer needed.

Closes #14543

15 months agocmake: fix find rustls
Jan Venekamp [Thu, 15 Aug 2024 18:43:33 +0000 (20:43 +0200)] 
cmake: fix find rustls

Follow-up to 2784801977e81f68c6f87f9509e64f332d74acab #14545
Closes #14567

15 months agomulti: on socket callback error, remove socket hash entry nonetheless
Daniel Stenberg [Thu, 15 Aug 2024 09:06:38 +0000 (11:06 +0200)] 
multi: on socket callback error, remove socket hash entry nonetheless

Previously an error from the callback accidentally made libcurl keep the
hash entry which would lead to the entry remaining and then potentially
getting removed *again* which would lead to internal confusions.

This is an old issue (introduced in 2b3dd01b779e4eff6), caught by the new
asserts from c0233a35da81.

Closes #14557

15 months agolibcurl.pc: add reference to `libgsasl`
Viktor Szakats [Thu, 15 Aug 2024 08:41:02 +0000 (10:41 +0200)] 
libcurl.pc: add reference to `libgsasl`

Closes #14556

15 months agotidy-up: misc spelling (bit, ASCII)
Viktor Szakats [Thu, 15 Aug 2024 09:00:04 +0000 (11:00 +0200)] 
tidy-up: misc spelling (bit, ASCII)

Closes #14559

15 months agotests: move the disabling of 500 for hyper from CI to DISABLED
Daniel Stenberg [Thu, 15 Aug 2024 07:04:28 +0000 (09:04 +0200)] 
tests: move the disabling of 500 for hyper from CI to DISABLED

Follow-up to 136504195ae63

Closes #14551

15 months agocurl: fix the -w urle.* variables
Daniel Stenberg [Thu, 15 Aug 2024 09:43:59 +0000 (11:43 +0200)] 
curl: fix the -w urle.* variables

urle.scheme, urle.user, urle.password and urle.options mistakenly
operated on the original URL instead of the *effective* (last) URL.

Add test 474 to verify.

Reported-by: Gruber Glass
Fixes #14550
Closes #14560

15 months agocmake: show warning if libpsl is not found
Viktor Szakats [Tue, 13 Aug 2024 11:44:27 +0000 (13:44 +0200)] 
cmake: show warning if libpsl is not found

Also:
- explicitly disable libpsl in CI to avoid configure warning, where
  necessary.
- add TODO to make this warning an error (to match autotools.)

Follow-up to 2998874bb61ac6ef3b72d6a61467cd2aaf6e53ea #12661

Closes #14533

15 months agomime: avoid inifite loop in client reader
Stefan Eissing [Tue, 13 Aug 2024 11:34:54 +0000 (13:34 +0200)] 
mime: avoid inifite loop in client reader

Curl_mime_read() may go into an infinite loop when called with buffer
lengths <= 4. Some encoders, like base64, are not prepared for that.

In the client reader for mime data, skip such small reads. The upload
buffers will get flushed eventually and larger reads will happen again.

Improves robustness of test652 which triggered the loop on blocked
sends.

Closes #14532

15 months agocmake: fix and tidy up c-ares builds, enable in more CI jobs
Viktor Szakats [Wed, 14 Aug 2024 09:07:01 +0000 (11:07 +0200)] 
cmake: fix and tidy up c-ares builds, enable in more CI jobs

- set `CARES_NO_DEPRECATED` to avoid c-ares deprecation warnings.
  Like autotools does.

- drop unused c-ares header directory when building libtests and test
  servers.

- disable TrackMemory (aka `CURLDEBUG`) feature for MSVC builds.
  It fails on test 558 and 1330, saying that TrackMemory isn't working.
  Left a FIXME about it.

- GHA/macos: enable c-ares in a cmake job.

- GHA/windows: enable c-ares in MSVC job.
  Fixes #14202

- GHA/windows: add c-ares mingw autotools job with tests.
  (move `--with-windows-unicode` option from 'default' job to
  this one to keep the former "default".)
  Put these tests on ignore for now:
  ```
  FAIL 472: 'aws-sigv4 with query' HTTP, aws-sigv4
  FAIL 1299: 'Send "OPTIONS *" with --request-target' HTTP, --request-target
  FAIL 1613: 'Send "OPTIONS *" with --request-target to a proxy' HTTP, HTTP proxy, --request-target
  ```
  Ref: https://github.com/curl/curl/actions/runs/10388126947/job/28765761441#step:15:5902

  472: fails with mingw despite Unicode enabled.
  1299, 1613: seem like a case of expanding '*' to a filename.

- GHA/windows: enable c-ares in mingw cmake job.

Closes #14541

15 months agoGHA/configure-vs-cmake: add macOS build, fix issues
Viktor Szakats [Wed, 14 Aug 2024 15:30:47 +0000 (17:30 +0200)] 
GHA/configure-vs-cmake: add macOS build, fix issues

- sync cmake macOS `OS` value manually with the autotools one.
- stop exporting/subst-ing `HAVE_OPENSSL_QUIC` from autotools.
  The variable was only used internally.
- exclude a dependency detection symbol.
- allow to run when the workflow itself was updated.
- simplify cmake command.
- fix indentation.

Closes #14546

15 months agocmake: add missing `pkg-config` hints to Find modules
Viktor Szakats [Tue, 13 Aug 2024 09:20:00 +0000 (11:20 +0200)] 
cmake: add missing `pkg-config` hints to Find modules

- brotli, c-ares, libpsl, libssh2, mbedtls, rustls:
  Use `pkg-config` for path hints and version info. Syncing them up with
  the rest of Find modules.

- GHA/macos: force-disable libssh2 with cmake to sync with autotools.
  After this patch, cmake auto-detects libssh2 in this job.

Closes #14545

15 months agoRELEASE-NOTES: synced
Daniel Stenberg [Thu, 15 Aug 2024 07:09:41 +0000 (09:09 +0200)] 
RELEASE-NOTES: synced

15 months agogetinfo: add CURLINFO_POSTTRANSFER_TIME_T
Alex Snast [Mon, 15 Jul 2024 08:52:50 +0000 (11:52 +0300)] 
getinfo: add CURLINFO_POSTTRANSFER_TIME_T

Returns the time, in microseconds, from the start until the last byte is
sent by libcurl (i.e. the request is sent off).

Closes #14189

15 months agohash: provide asserts to verify API use
Daniel Stenberg [Mon, 12 Aug 2024 12:06:12 +0000 (14:06 +0200)] 
hash: provide asserts to verify API use

- converted the Curl_hash_count() macro to a function

- Discourage accessing struct fields directly

- Document the internal API in HASH.md

Closes #14503

15 months agoGHA/windows: enable HTTP/3 in wolfSSL MSVC job
Tal Regev [Sun, 4 Aug 2024 17:45:00 +0000 (20:45 +0300)] 
GHA/windows: enable HTTP/3 in wolfSSL MSVC job

Closes #14383

15 months agoGHA/windows: add GnuTLS to MSVC jobs
Tal Regev [Sat, 10 Aug 2024 08:15:25 +0000 (11:15 +0300)] 
GHA/windows: add GnuTLS to MSVC jobs

- enable GnuTLS in MultiSSL job.

- add new GnuTLS job with HTTP/3 enabled.
  Replacing the `!ssl` job.
  Use `Release` mode to work around a vcpkg package issue, ending up
  depending on both the debug and release versions of the gnutls DLL,
  one of them miss to copy next to the binary.

- cmake: add `pkg-config` detection for GnuTLS.

Co-authored-by: Tal Regev
Fixes #14494
Closes #14495

15 months agocmake: add rustls
Jan Venekamp [Tue, 13 Aug 2024 22:05:44 +0000 (00:05 +0200)] 
cmake: add rustls

Closes #14534

15 months agocmake: sync up result variable names in Find modules
Viktor Szakats [Wed, 14 Aug 2024 01:09:08 +0000 (03:09 +0200)] 
cmake: sync up result variable names in Find modules

- bearssl, c-ares, gss, libpsl, libssh2, mbedtls:
  Before this patch these Find modules returned results via
  `<NAME>_INCLUDE_DIR` and `<NAME>_LIBRARY`.

  This patch makes them return `<NAME>_INCLUDE_DIRS` (note the `S`)
  and `<NAME>_LIBRARIES` like other modules already did.

- bearssl, mbedtls:
  Before this patch these Find modules allowed custom configuration
  via `<NAME>_INCLUDE_DIRS` (note the `S`).

  This patch makes them accept `<NAME>_INCLUDE_DIR`, like the rest of
  the modules did.

  Deprecate the old variables, but keep accepting them for
  compatibility.

- bearssl: add missing `mark_as_advanced()` call.

Closes #14542

15 months agocmake: tidy up Find modules
Viktor Szakats [Tue, 13 Aug 2024 20:31:21 +0000 (22:31 +0200)] 
cmake: tidy up Find modules

Smoothen out minor differences between Find modules.

- brotli, nghttp2: drop redundant `FOUND_VAR` specifiers from
  `find_package_handle_standard_args()` calls.
  This function sets both `<NAME_UPPER>_FOUND` and `<NAME>_FOUND`
  by default.

- brotli: set result vars only when found.

- brotli: add missing `mark_as_advanced()` call.

- brotli: delete custom fail message.

- mbedtls, bearssl: use `REQUIRED_VARS` instead of `DEFAULT_MSG`.

- msh3, quiche: set `<NAME>_VERSION` (via pkg-config).

- wolfssl: also use `PC_WOLFSSL_INCLUDEDIR`, `PC_WOLFSSL_LIBDIR`
  as hints.

- libpsl, libssh2, zstd: clear temporary variables used for version
  detection.

- gss, msh3, nghttp2, nghttp3, ngtcp2, quiche, zstd: fix to apply
  `mark_as_advanced()` to internal variables only.

Closes #14538

15 months agocmake: update list of "advanced" variables
Viktor Szakats [Wed, 14 Aug 2024 08:11:03 +0000 (10:11 +0200)] 
cmake: update list of "advanced" variables

To hide them from the CMake GUI by default.

Closes #14540

15 months agosmtp: add tracing feature
Stefan Eissing [Tue, 13 Aug 2024 11:49:07 +0000 (13:49 +0200)] 
smtp: add tracing feature

Add `smtp` as tracing feature, use CURL_TRC_SMTP() in code to
trace operations.

Closes #14531

15 months agoTODO: mqtt and gopher test fails on network blocks
Stefan Eissing [Tue, 13 Aug 2024 09:09:30 +0000 (11:09 +0200)] 
TODO: mqtt and gopher test fails on network blocks

Add descriptions of the test failures and how to reproduce
them to the TODO for both protocols.

Closes #14528

15 months agotest649: improve robustness
Stefan Eissing [Tue, 13 Aug 2024 08:53:24 +0000 (10:53 +0200)] 
test649: improve robustness

Remove check of upload data before the error is triggered as this
may vary with network conditions.

Closes #14526

15 months agotest587: improve robustness
Stefan Eissing [Tue, 13 Aug 2024 08:39:48 +0000 (10:39 +0200)] 
test587: improve robustness

Remove check of server output as upload may abort before request
could fully be sent, so server output may be completely missing.

Test already used a 1 second delay to mitigate timing. This change
makes timing no longer an issue.

Closes #14525

15 months agotest httpd, tweak cipher list
Stefan Eissing [Mon, 12 Aug 2024 11:53:04 +0000 (13:53 +0200)] 
test httpd, tweak cipher list

Configure the AES 256 instead of the AES 128 cipher in the test httpd to
make scorecard testing between httpd and caddy more comparable.

Adapt test_17 expectations, now that AES 128 can no longer, but 256 can
now be negotiated.

Closes #14502

15 months agognutls/wolfssl: improve error message when certificate fails
Stefan Eissing [Mon, 12 Aug 2024 10:21:38 +0000 (12:21 +0200)] 
gnutls/wolfssl: improve error message when certificate fails

Give more detailed reasons for certificate failures where available in
gnutls and wolfssi to allow user to understand the cause of the failure.

Closes #14501

15 months agohyper: call Curl_req_set_upload_done()
Stefan Eissing [Wed, 14 Aug 2024 08:22:20 +0000 (10:22 +0200)] 
hyper: call Curl_req_set_upload_done()

Hyper implementation was missing the call to Curl_req_set_upload_done()
as it works differently than out normal request upload handling.

Closes #14539

15 months agourldata: introduce `data->mid`, a unique identifier inside a multi
Stefan Eissing [Mon, 12 Aug 2024 09:28:19 +0000 (11:28 +0200)] 
urldata: introduce `data->mid`, a unique identifier inside a multi

`data->id` is unique in *most* situations, but not in all. If a libcurl
application uses more than one connection cache, they will overlap. This
is a rare situations, but libcurl apps do crazy things. However, for
informative things, like tracing, `data->id` is superior, since it
assigns new ids in curl's serial curl_easy_perform() use.

Introduce `data->mid` which is a unique identifer inside one multi
instance, assigned on multi_add_handle() and cleared on
multi_remove_handle().

Use the `mid` in DoH operations and also in h2/h3 stream hashes.

Reported-by: 罗朝辉
Fixes #14414
Closes #14499

15 months agotool_paramhlp: bump maximum post data size in memory to 16GB
Daniel Stenberg [Tue, 13 Aug 2024 07:12:18 +0000 (09:12 +0200)] 
tool_paramhlp: bump maximum post data size in memory to 16GB

- stick to 2GB for 32bit systems.

Reported-by: Tim Yuer
Fixes #14521
Closes #14523

15 months agocmake: sync up formatting in Find modules
Viktor Szakats [Tue, 13 Aug 2024 08:45:37 +0000 (10:45 +0200)] 
cmake: sync up formatting in Find modules

- lowercase internal variable names (FindGSS)
- comments
- whitespace

Closes #14527

15 months agoruntests: log ignored but passed tests
Viktor Szakats [Thu, 8 Aug 2024 12:42:40 +0000 (14:42 +0200)] 
runtests: log ignored but passed tests

List tests explicitly ignored, but passing. It can be useful to catch
ignore rules that are no longer necessary because the tests are not
failing anymore.

To not clutter the default view, hide this information under a GitHub
Actions foldable group.

Closes #14457

15 months agoGHA/macos: disable AppleIDN for autotools in combinations jobs
Viktor Szakats [Tue, 13 Aug 2024 11:28:48 +0000 (13:28 +0200)] 
GHA/macos: disable AppleIDN for autotools in combinations jobs

Without libidn2, AppleIDN gets enabled by default in autotools.

To remain minimal and match cmake.

Follow-up to 8de8fe8c98309c8b6183b22cc2e209ce4648173b #14401

15 months agotests: don't mangle output if hostname or type unknown
Dan Fandrich [Tue, 13 Aug 2024 08:11:33 +0000 (01:11 -0700)] 
tests: don't mangle output if hostname or type unknown

If either of these couldn't be determined, the output would be missing a
newline. Make sure the line ends even in the case of an error.

15 months agocurl_sha512_256: fix symbol collisions with nettle library
Viktor Szakats [Sun, 11 Aug 2024 19:44:19 +0000 (21:44 +0200)] 
curl_sha512_256: fix symbol collisions with nettle library

`SHA512_256_BLOCK_SIZE`, `SHA512_256_DIGEST_SIZE` macros were both
defined within curl and also in the nettle library required by GnuTLS.

Fix it by namespacing the curl macros.

Cherry-picked from #14495
Closes #14514

15 months agolib: prefer `CURL_SHA256_DIGEST_LENGTH` over the unprefixed name
Viktor Szakats [Sun, 11 Aug 2024 22:57:41 +0000 (00:57 +0200)] 
lib: prefer `CURL_SHA256_DIGEST_LENGTH` over the unprefixed name

Already used in `vtls.h`. Prefer this curl-namespaced name over the
unprefixed `SHA256_DIGEST_LENGTH`. The latter is also defined by TLS
backends with a potential to cause issues.

Also stop relying on externel headers setting this constant. It's
already defined in `vtls.h` on curl's behalf, do this also for `lib`.

Cherry-picked from #14495
Closes #14513

15 months agolib: avoid macro collisions between wolfSSL and GnuTLS headers
Viktor Szakats [Sun, 11 Aug 2024 23:02:37 +0000 (01:02 +0200)] 
lib: avoid macro collisions between wolfSSL and GnuTLS headers

Both of these projects define the same `SHA*` macros via headers
included by curl (in MultiSSL builds, possibly only in Unity builds),
causing redefinition compiler warnings.

Fix it by disabling compatibility macros in wolfSSL.

```
Building C object lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c.o
In file included from _bld/lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c:202:
In file included from lib/http_aws_sigv4.c:33:
In file included from lib/curl_sha256.h:40:
In file included from /usr/local/Cellar/wolfssl/5.7.2/include/wolfssl/openssl/sha.h:30:
/usr/local/Cellar/wolfssl/5.7.2/include/wolfssl/wolfcrypt/sha256.h:117:13: warning: 'SHA256_BLOCK_SIZE' macro redefined [-Wmacro-redefined]
    #define SHA256_BLOCK_SIZE  WC_SHA256_BLOCK_SIZE
            ^
/usr/local/Cellar/nettle/3.10/include/nettle/sha2.h:70:9: note: previous definition is here
#define SHA256_BLOCK_SIZE 64
        ^
In file included from _bld/lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c:202:
In file included from lib/http_aws_sigv4.c:33:
In file included from lib/curl_sha256.h:40:
In file included from /usr/local/Cellar/wolfssl/5.7.2/include/wolfssl/openssl/sha.h:30:
[...]
    #define SHA256_DIGEST_SIZE WC_SHA256_DIGEST_SIZE
    #define SHA224_BLOCK_SIZE  WC_SHA224_BLOCK_SIZE
    #define SHA224_DIGEST_SIZE WC_SHA224_DIGEST_SIZE
    #define SHA512_BLOCK_SIZE  WC_SHA512_BLOCK_SIZE
    #define SHA512_DIGEST_SIZE WC_SHA512_DIGEST_SIZE
    #define SHA384_BLOCK_SIZE  WC_SHA384_BLOCK_SIZE
    #define SHA384_DIGEST_SIZE WC_SHA384_DIGEST_SIZE
```

Cherry-picked from #14495
Closes #14511

15 months agocmake: update `curl-config.cmake.in` template var list [ci skip]
Viktor Szakats [Tue, 13 Aug 2024 08:01:41 +0000 (10:01 +0200)] 
cmake: update `curl-config.cmake.in` template var list [ci skip]

Omit those defined by CMake.

Follow-up to a298df7f475f95996dcabc9e5a6ab31225ae6afa #14462

15 months agolib: fix building with wolfSSL without DES support
Viktor Szakats [Sun, 11 Aug 2024 20:26:21 +0000 (22:26 +0200)] 
lib: fix building with wolfSSL without DES support

E.g. with Homebrew wolfssl 5.7.2 on macOS:
```
In file included from _bld/lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c:85:
lib/curl_ntlm_core.c:157:27: error: unknown type name 'DES_key_schedule'
                          DES_key_schedule DESKEYARG(ks))
                          ^
lib/curl_ntlm_core.c:159:3: error: use of undeclared identifier 'DES_cblock'
  DES_cblock key;
  ^
[...]
```

Earlier patch addressing this for OpenSSL:
802d8644500f5b18c895b77a23d85e029766d65e #7808

Cherry-picked from #14495
Closes #14512

15 months agosha256: fix symbol collision between nettle (GnuTLS) and OpenSSL codepath
Viktor Szakats [Sun, 11 Aug 2024 19:43:46 +0000 (21:43 +0200)] 
sha256: fix symbol collision between nettle (GnuTLS) and OpenSSL codepath

Fixes:
```
C:\vcpkg\installed\x64-windows\include\nettle\sha2.h(75,8): error C2011: 'sha256_ctx': 'struct' type redefinition
```
Ref: https://github.com/curl/curl/actions/runs/10341162641/job/28622681573?pr=14484#step:10:30

Cherry-picked from #14495
Closes #14515