]> git.ipfire.org Git - thirdparty/curl.git/log
thirdparty/curl.git
15 months agowolfssl: improve shutdown handling
Stefan Eissing [Sun, 4 Aug 2024 11:22:03 +0000 (13:22 +0200)] 
wolfssl: improve shutdown handling

Improve handling of shutdown when sending gets blocked.

Add workaround for <https://github.com/wolfSSL/wolfssl/issues/7784>
where wolfSSL keeps on adding close notify messages to its outgoing
buffer on ever attempt.

Closes #14376

15 months agoopenssl: improve shutdown handling
Stefan Eissing [Sun, 4 Aug 2024 11:08:51 +0000 (13:08 +0200)] 
openssl: improve shutdown handling

Make sure that `io_need` is cleared and set at the filter operations.
Add some more tracing for shutdown situations.

Improve shutdown handling for blocked sends. OpenSSL is a bit tricksy
here that it only reports WANT_WRITE on SSL_shutdown(), but never on
SSL_read() on blocked sends. So we need to use both.

At last, set SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER when available since we
are not always retrying sends from the very same address, as testing
showed.

Closes #14375

15 months agobearssl: improve shutdown handling
Stefan Eissing [Sun, 4 Aug 2024 10:56:59 +0000 (12:56 +0200)] 
bearssl: improve shutdown handling

- replace own "adjust_pollset()" with vtls' standard one
- set io_need always accordingly
- handle EAGAIN on shutdown

Closes #14374

15 months agoconfigure: fail if PSL is not disabled but not found
Daniel Stenberg [Sun, 4 Aug 2024 14:38:11 +0000 (16:38 +0200)] 
configure: fail if PSL is not disabled but not found

Regression since 9b3f67e (shipped in 8.7.0)
Reported-by: Ryan Carsten Schmidt
Fixes #14373
Assisted-by: Viktor Szakats
Closes #14379

15 months agoKNOWN_BUGS: mention AppleIDN and WinIDN test problems
Daniel Stenberg [Sun, 4 Aug 2024 21:20:07 +0000 (23:20 +0200)] 
KNOWN_BUGS: mention AppleIDN and WinIDN test problems

Closes #14176
Closes #14387

15 months agotool_operhlp: fix "potentially uninitialized local variable 'pc' used"
Tal Regev [Sun, 4 Aug 2024 22:03:09 +0000 (01:03 +0300)] 
tool_operhlp: fix  "potentially uninitialized local variable 'pc' used"

A false positive by MSVC.

Closes #14389

15 months agosigpipe: init the struct so that first apply ignores
Daniel Stenberg [Sun, 4 Aug 2024 22:17:17 +0000 (00:17 +0200)] 
sigpipe: init the struct so that first apply ignores

Initializes 'no_signal' to TRUE, so that a call to sigpipe_apply() after
init ignores the signal (unless CURLOPT_NOSIGNAL) is set.

I have read the existing code multiple times now and I think it gets the
initial state reversed this missing to ignore.

Regression from 17e6f06ea37136c36d27

Reported-by: Rasmus Thomsen
Fixes #14344
Closes #14390

15 months agowolfssl: add support for ssl cert blob / ssl key blob options
Alex Snast [Tue, 25 Jun 2024 16:34:24 +0000 (09:34 -0700)] 
wolfssl: add support for ssl cert blob / ssl key blob options

wolfSSL supports setting certificates/private keys from memory blobs
which allow us to implement both CURLOPT_SSLCERT_BLOB and
CURLOPT_SSLKEY_BLOB options.

Closes #14018

15 months agocmake: add support for versioned symbols option
Viktor Szakats [Sun, 4 Aug 2024 11:52:44 +0000 (13:52 +0200)] 
cmake: add support for versioned symbols option

Implement the `--enable-versioned-symbols` feature available in
`./configure` for CMake.

Enable with `-DCURL_LIBCURL_VERSIONED_SYMBOLS=ON`. Customize the version
prefix with `-DCURL_LIBCURL_VERSIONED_SYMBOLS_PREFIX=MYPREFIX_`. By
default the prefix matches what autotools uses.

When enabled, the feature is detected and a warning shown if
unavailable. (E.g. on Apple and Windows, it isn't.)

Included `HIDDEN {};` to match autotools, though I don't know if it's
necessary, useful or making any difference.

Differences from the autotools implementation:
- soversion is dynamic instead of hard-coded.
- omits referencing non-curl symbols.
- allows prefix/flavour override.
- more universal feature detection.
- doesn't rely on the in-repo `lib/libcurl.vers.in` file.

Also:
- add mbedTLS and BearSSL versioned symbol prefix support to autotools.
- enable this option in an old-linux job.

Follow-up to 7cc2e8b349df28d55d5f40bfae323485df9f0cf2
Fixes #14349
Closes #14378

15 months agoeasy: fix curl_easy_upkeep for shared connection caches
Jay Satiro [Wed, 10 Jan 2024 08:53:23 +0000 (03:53 -0500)] 
easy: fix curl_easy_upkeep for shared connection caches

- Determine which connection cache is or will be used by the easy handle
  and perform connection upkeep on that cache.

This change allows curl_easy_upkeep to be effective on easy handles that
are using a shared connection cache, either from a user created shared
connection cache or a user created multi which has its own shared
connection cache.

Prior to this change curl_easy_upkeep would upkeep the connection cache
for the easy handle only if that cache was from the multi owned by the
easy handle (ie curl_easy_perform was previously called and there's a
connection cache exclusive to the easy handle in
data->multi_easy->conn_cache).

Ref: https://curl.se/mail/lib-2024-01/0016.html

Closes https://github.com/curl/curl/pull/12677

15 months agovtls: stop offering alpn http/1.1 for http2-prior-knowledge
Jay Satiro [Tue, 23 Jul 2024 21:34:41 +0000 (17:34 -0400)] 
vtls: stop offering alpn http/1.1 for http2-prior-knowledge

- For HTTPS if http2-prior-knowledge is set then only offer h2 (HTTP/2)
  alpn to the server for protocol negotiation.

Prior to this change both HTTP/2 ("h2") and HTTP/1.1 ("http/1.1") were
offered for ALPN when http2-prior-knowledge was set.

CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE (tool: --http2-prior-knowledge) is
meant to send non-TLS HTTP requests HTTP/2 when it is known the server
supports them. However when HTTPS is used then it attempts to first
negotiate the connection with ALPN. In that case the user likely does
not want to offer http/1.1 to the server as an acceptable protocol.

Reported-by: kit-ty-kate@users.noreply.github.com
Fixes https://github.com/curl/curl/issues/9963
Closes https://github.com/curl/curl/pull/14266

15 months agocurl: add --skip-existing
Daniel Stenberg [Sun, 4 Aug 2024 14:14:24 +0000 (16:14 +0200)] 
curl: add --skip-existing

With this option, the entire download is skipped if the selected target
filename already exists when the opertion is about to begin.

Test 994, 995 and 996 verify.

Ref: #11012
Closes #13993

15 months agorevert "tests/http: configure test httpd to honor client cipher order"
Jan Venekamp [Sun, 4 Aug 2024 15:00:30 +0000 (17:00 +0200)] 
revert "tests/http: configure test httpd to honor client cipher order"

revert f6cb3c63 #14338

Setting SSLHonorCipherOrder to on means it honors the server cipher
order. From the documentation: "When choosing a cipher during an SSLv3
or TLSv1 handshake, normally the client's preference is used. If this
directive is enabled, the server's preference will be used instead."

Also the commit inhibits test_17_07_ssl_ciphers. The test tries to
tests if all the ciphers specified, and only those, are properly set
in curl. For that to work we need have cases where some or all ciphers
do no intersect with the cipher-set of the server. We need to be able
to assert a failed connection based on a cipher set mismatch.

That is why a restricted set of ciphers is used on the server. This
set is so chosen that it contains the well known most secure ciphers.
Except with the slower aes256 variant intentionally left out, to be
able to test above described.

As test_17_07_ssl_ciphers is currently the only test that tests the
functioning of the --ciphers and --tls13-ciphers options, it is
important that its coverage is as good as possible.

Closes #14381

15 months agoGHA/windows: add mbedTLS MSVC job
Tal Regev [Tue, 16 Jul 2024 18:43:55 +0000 (21:43 +0300)] 
GHA/windows: add mbedTLS MSVC job

Ref: 92e28f2897dc4adf0014af197e88fd8e1ccc6aeb #14228
Closes #14203

15 months agotidy-up: OS names
Viktor Szakats [Fri, 2 Aug 2024 23:09:57 +0000 (01:09 +0200)] 
tidy-up: OS names

Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.

Mostly OS names and a few more.

Also a couple of other minor text fixups.

Closes #14360

15 months agodist: add missing `docs/examples/CMakeLists.txt`
Viktor Szakats [Sun, 4 Aug 2024 14:56:58 +0000 (16:56 +0200)] 
dist: add missing `docs/examples/CMakeLists.txt`

Follow-up to dfdd978f7c60224dffe2aac25b436dc0a5cd0186 #13491
Closes #14380

15 months agoRELEASE-NOTES: synced
Daniel Stenberg [Sun, 4 Aug 2024 14:26:21 +0000 (16:26 +0200)] 
RELEASE-NOTES: synced

15 months agomaketgz: accept option to include latest commit hash
Daniel Stenberg [Sat, 3 Aug 2024 22:03:41 +0000 (00:03 +0200)] 
maketgz: accept option to include latest commit hash

If the second argument to the script is "commit", then this will
generate a file named `docs/tarball-commit.txt` that contains the latest
commit hash (git rev-parse HEAD) at the time the script runs.

Doing this breaks the reproducibility so it will not be used for "real"
releases but is meant for automated daily snapshots and similar.

Reported-by: Dan Fandrich
Fixes #14363
Closes #14369

15 months agocurl: --help [option] displays documentation for given cmdline option
Daniel Stenberg [Sat, 3 Aug 2024 18:24:12 +0000 (20:24 +0200)] 
curl: --help [option] displays documentation for given cmdline option

Since the documentation text blob might be gzipped, it needs to search
for what to output in a streaming manner. It then first searches for
"\nALL OPTIONS".

Then, it looks for the start to display at "\n    -[option]" and stops
again at "\n    -". Except for the last option in the man page, which
ends at "\nFILES" - the subtitle for the section following all options
in the manpage.

Test 1707 to 1710 verify

Closes #13997

15 months agotool_operate: support --dump-header % to direct to stderr
Daniel Stenberg [Sun, 4 Aug 2024 14:01:41 +0000 (16:01 +0200)] 
tool_operate: support --dump-header % to direct to stderr

Similar to how --trace and --trace-ascii already do it.

Added test 1489 to verify

Closes #13992

15 months agotool_operate: for -O, use "default" as filename when the URL has none
Daniel Stenberg [Sat, 3 Aug 2024 18:08:27 +0000 (20:08 +0200)] 
tool_operate: for -O, use "default" as filename when the URL has none

... or pick the last directory part from the path if available.

Instead of returning error.

Add test 690 and 691 to verify. Test 76 and 2036 no longer apply.

Closes #13988

15 months agodoh-url.md: point out DOH server IP pinning
David Sardari [Sun, 4 Aug 2024 11:35:06 +0000 (13:35 +0200)] 
doh-url.md: point out DOH server IP pinning

Closes #14377
Signed-off-by: David Sardari <d@duxsco.de>
15 months agotests: fixup `tests/data/Makefile.am` references
Viktor Szakats [Sun, 4 Aug 2024 10:25:45 +0000 (12:25 +0200)] 
tests: fixup `tests/data/Makefile.am` references

Follow-up to f5b826532f2c564ef240df0ba2f3287d521df711 #14357
Reported-by: Stefan Eissing
Fixes #14371
Closes #14372

15 months agoGHA/non-native: ignore FreeBSD FTP test results
Viktor Szakats [Sun, 4 Aug 2024 00:08:06 +0000 (02:08 +0200)] 
GHA/non-native: ignore FreeBSD FTP test results

They are flaky.

15 months agopytests: add tests for HEAD requests in all HTTP versions
Stefan Eissing [Sat, 3 Aug 2024 10:46:28 +0000 (12:46 +0200)] 
pytests: add tests for HEAD requests in all HTTP versions

Closes #14367

15 months agocmake: tidy-ups
Viktor Szakats [Fri, 12 Jul 2024 19:10:57 +0000 (21:10 +0200)] 
cmake: tidy-ups

- tidy-up comments.
- use lowercase, underscore prefixed names for internal variables.
- use `IN LISTS` and `IN ITEMS` in `foreach()` loops.
- rename variable name `OUTPUT` to a more distinctive one.
- tidy-up `STREQUAL` syntax.
- delete commented code.
- indent/whitespace.

Closes #14197

15 months agoRELEASE-NOTES: synced
Daniel Stenberg [Sat, 3 Aug 2024 18:38:54 +0000 (20:38 +0200)] 
RELEASE-NOTES: synced

15 months agodocs/cmdline-opts: update see-also to use show-headers
Daniel Stenberg [Sat, 3 Aug 2024 18:29:30 +0000 (20:29 +0200)] 
docs/cmdline-opts: update see-also to use show-headers

Since --include is now deprecated since 82c53f821fa3fd4

15 months agogetparam: make --rate accept "number of units"
Daniel Stenberg [Sat, 20 Jul 2024 15:54:44 +0000 (17:54 +0200)] 
getparam: make --rate accept "number of units"

Do no more than 5 transfers per 15 seconds with "5/15s" or limit it to 3
transfers per 4 hours with "3/4h" etc.

Previously it would always only work with a single time unit.

Ref: #14242
Closes #14245

15 months agoGHA/windows: move Cygwin into its own workflow
Viktor Szakats [Sat, 3 Aug 2024 10:01:28 +0000 (12:01 +0200)] 
GHA/windows: move Cygwin into its own workflow

Cygwin runs stable but slow, while native Windows in the same workflow
runs fast but unreliable (hangs). GHA requires all jobs to finish before
allowing to re-run failed ones. Before this patch this meant waiting for
the slow Cygwin jobs to re-run hung native jobs. After this patch Cygwin
jobs run in their own workflow, allowing to re-run broken Windows jobs
earlier.

Follow-up to 0d125287296857fc4a931a0c20d0ffcb1f731807 #14236
Closes #14366

15 months agotool_getparam: make --show-headers the same as --include
Daniel Stenberg [Wed, 10 Jul 2024 14:30:17 +0000 (16:30 +0200)] 
tool_getparam: make --show-headers the same as --include

Simply a name alias that better explains what the option does.

Closes #13987

15 months agocfilters: send flush
Stefan Eissing [Thu, 25 Jul 2024 11:10:01 +0000 (13:10 +0200)] 
cfilters: send flush

Since data can be held in connection filter buffers when sending gives
EAGAIN, add methods to query this and perform flushing of those buffers.

The transfer loop will continue sending until all upload data is
processed and the connection is flushed.

- add `CF_QUERY_SEND_PENDING` to query filters
- add `CF_CTRL_DATA_SEND_FLUSH` to flush filters
- change `Curl_req_want_send()` to query the connection
  if it needs flushing
- use `Curl_req_want_send()` to determine the POLLOUT
  in the PERFORMING multi state
- implement flush handling in the HTTP/2 connection filter

Closes #14271

15 months agolib: add eos flag to send methods
Stefan Eissing [Thu, 18 Jul 2024 09:29:37 +0000 (11:29 +0200)] 
lib: add eos flag to send methods

Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.

This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.

Closes #14220

15 months agovtls: init ssl peer only once
Stefan Eissing [Thu, 11 Jul 2024 10:44:15 +0000 (12:44 +0200)] 
vtls: init ssl peer only once

- check that `struct ssl_peer` is only initialized once
- fix vtls peer init to run only once
- check in peer init that hostname is not empty, fail otherwise

Closes #14152

15 months agourl: dns_entry related improvements
Stefan Eissing [Fri, 12 Jul 2024 10:46:50 +0000 (12:46 +0200)] 
url: dns_entry related improvements

Replace Curl_resolv_unlock() with Curl_resolv_unlink():

-replace inuse member with refcount in Curl_dns_entry

- pass Curl_dns_entry ** to unlink, so it gets always cleared

- solve potential (but unlikley) UAF in FTP's handling of looked up
  Curl_dns_entry. Esp. do not use addr information after unlinking an entry.
  In reality, the unlink will not free memory, as the dns entry is still
  referenced by the hostcache. But this is not safe and relying on no other
  code pruning the cache in the meantime.

- pass permanent flag when adding a dns entry instead of fixing timestamp
  afterwards.

url.c: fold several static *resolve_* functions into one.

Closes #14195

15 months agoCurl_rand_bytes to control env override
Stefan Eissing [Tue, 23 Jul 2024 10:21:51 +0000 (12:21 +0200)] 
Curl_rand_bytes to control env override

- in DEBUGBUILD, all specifying if true random numbers
  are desired or simulated ones via CURL_ENTROPY
- allows to use randoms in other DEBUG checks to not
  interfere with the CURL_ENTROPY
- without this change, any Curl_rand() use will alter
  results of some AUTHENTICATION methods like DIGEST

Closes #14264

15 months agoCI: enable parallel testing in CI builds
Dan Fandrich [Thu, 22 Jun 2023 17:03:16 +0000 (10:03 -0700)] 
CI: enable parallel testing in CI builds

The test-ci target now uses 2 processes by default, but the amount of
parallelism is tuned for each CI service and build environment based on
results of a number of test runs.  Some CI services use super-
oversubscribed build machines that can barely run the curl tests
already with no parallelism without frequently failing with
timing-induced failures. These continue to be run without parallelism.
Other services provide two fast, unloaded cores and these run with 14
processes, which is a good default for this kind of environment.

Here's a summary of the number of test processes by CI service:

  Appveyor - 2 (Windows MSVC), 1 (others)
  Azure - 2
  Circle CI - 14
  Cirrus - 28 (macOS), 14 (Linux), 7 (FreeBSD), 5 (macOS torture), 2 (Windows)
  GitHub Actions - 3 (macOS), 2 (Linux)

Some of these are a bit conservative to keep timing-induced flakiness down.

The net result is that the first test results should arrive only
3 minutes after a commit submission.

Changes merged via separate commits:
2a7c8b27fdd266894fe24d6d0f5d853ea843dff4 #14171
72341068a2d8f1b94a26add16830c725cc4054e6
efce544418971b064d9fc8183a6caa582c54559b #14244
c6cf411bacf97f230db160e543d0cc3bbe5b9aba

Ref: #10818
Closes #11510

15 months agoCI: realign cmake build settings (for nghttp2, libidn2)
Viktor Szakats [Sat, 3 Aug 2024 08:01:49 +0000 (10:01 +0200)] 
CI: realign cmake build settings (for nghttp2, libidn2)

- nghttp2 is now on by default, adjust build configuration accordingly.
  Follow-up to 87aa4ebd821ebae0023df8658360c724efcf5e00 #14136

- GHA/macos: disable libidn2 in combination builds, syncing with
  autotools. Otherwise it's now auto-detected via pkg-config by default.
  Follow-up to f43adc2c4978f7f82a359e89186e58a31d17b0ad #14137

Closes #14364

15 months agocurl: support embedding a CA bundle
Viktor Szakats [Sat, 29 Jun 2024 01:30:14 +0000 (03:30 +0200)] 
curl: support embedding a CA bundle

Add the ability to embed a CA bundle into the curl binary. It is used
when no other runtime or build-time option set one.

This helps curl-for-win macOS and Linux builds to run standalone, and
also helps Windows builds to avoid picking up the CA bundle from an
arbitrary (possibly world-writable) location (though this behaviour is
not currently disablable).

Usage:
- cmake: `-DCURL_CA_EMBED=/path/to/curl-ca-bundle.crt`
- autotools: `--with-ca-embed=/path/to/curl-ca-bundle.crt`
- Makefile.mk: `CURL_CA_EMBED=/path/to/curl-ca-bundle.crt`

Also add new command-line option `--dump-ca-embed` to dump the embedded
CA bundle to standard output.

Closes #14059

15 months agocmake: detect `nghttp2` via `pkg-config`, enable by default
Viktor Szakats [Tue, 9 Jul 2024 09:39:48 +0000 (11:39 +0200)] 
cmake: detect `nghttp2` via `pkg-config`, enable by default

- also detect nghttp2 via `pkg-config` to match nghttp3 detection
  and autotools.

- enable nghttp2 by default to match autotools.

Cherry-picked from #14097
Closes #14136

15 months agocmake: drop unused internal variable
Viktor Szakats [Fri, 2 Aug 2024 22:23:11 +0000 (00:23 +0200)] 
cmake: drop unused internal variable

Closes #14361

15 months agovtls: fix MSVC 'cast truncates constant value' warning
Tal Regev [Thu, 1 Aug 2024 21:07:12 +0000 (00:07 +0300)] 
vtls: fix MSVC 'cast truncates constant value' warning

Fixes:
```
curl\lib\vtls\cipher_suite.c(193,3): error C2220: the following warning is treated as an error
curl\lib\vtls\cipher_suite.c(193,3): warning C4310: cast truncates constant value
```

Closes #14341

15 months agoci: Update actions/upload-artifact digest to 89ef406
renovate[bot] [Fri, 2 Aug 2024 21:06:28 +0000 (21:06 +0000)] 
ci: Update actions/upload-artifact digest to 89ef406

Closes #14359

15 months agocmake: drop reference to undefined variable
Viktor Szakats [Fri, 2 Aug 2024 13:05:35 +0000 (15:05 +0200)] 
cmake: drop reference to undefined variable

Undefined since the initial commit adding CMake support.

Closes #14358

15 months agocmake: drop no-op `tests/data/CMakeLists.txt`
Viktor Szakats [Fri, 2 Aug 2024 11:41:36 +0000 (13:41 +0200)] 
cmake: drop no-op `tests/data/CMakeLists.txt`

It's a no-op since
d162fca69a22d199c0369c5ce3d66a9531752ba2 #9333 (2022-08-18).

Also revert 476499c75c83ffa3d65da398b5759a5f2fdb980d that is
no longer necessary: move `Makefile.inc` back into `Makefile.am`.

Closes #14357

15 months agocmake: drop custom `CMakeOutput.log`/`CMakeError.log` logs
Viktor Szakats [Fri, 2 Aug 2024 11:38:14 +0000 (13:38 +0200)] 
cmake: drop custom `CMakeOutput.log`/`CMakeError.log` logs

CMake writes this information into `CMakeConfigureLog.yaml`
automatically.

Closes #14356

15 months agox509asn1: raise size limit for x509 certification information
Stefan Eissing [Fri, 2 Aug 2024 10:50:07 +0000 (12:50 +0200)] 
x509asn1: raise size limit for x509 certification information

Raise the limit for certification information from 10 thousand to 100
thousand bytes. Certificates can be larger than 10k.

Change the infof() debug output to add '...' at the end when the max
limit it can handle is exceeded.

Reported-by: Sergio Durigan Junior
Fixes #14352
Closes #14354

15 months agoGHA/distcheck: add a reproducible release check
Daniel Stenberg [Fri, 2 Aug 2024 06:58:05 +0000 (08:58 +0200)] 
GHA/distcheck: add a reproducible release check

15 months agoverify-release: shell script that verifies a release tarball
Daniel Stenberg [Fri, 2 Aug 2024 06:46:06 +0000 (08:46 +0200)] 
verify-release: shell script that verifies a release tarball

This script remakes a provided curl release and verifies that the newly
built version is identical to the original file.

Due to bugs in releases up to and including curl 8.9.1, it does not work
on tarballs generated before commit 754acd1a9dc6.

Closes #14350

15 months agoMakefile: remove 'scripts' duplicate from DIST_SUBDIRS
Daniel Stenberg [Fri, 2 Aug 2024 07:49:31 +0000 (09:49 +0200)] 
Makefile: remove 'scripts' duplicate from DIST_SUBDIRS

Also fix the .dist replacing by avoiding all Makefiles because it
otherwise also went into the temporary release folder and got confused
about the Makefile.dist in there.

15 months agodmaketgz: only run 'make distclean' if Makefile exists
Daniel Stenberg [Fri, 2 Aug 2024 07:16:31 +0000 (09:16 +0200)] 
dmaketgz: only run 'make distclean' if Makefile exists

... so that we can avoid the build failure if we run this in a clean
checkout.

Also remove -it from the docker invoke since it is not interactive and
it needs no TTY. They made the job fail in the CI.

15 months agoautotools: fix typo in tests/data target
Viktor Szakats [Fri, 2 Aug 2024 11:33:59 +0000 (13:33 +0200)] 
autotools: fix typo in tests/data target

Ref: c1b369fd4c2d538c89d6f9342092bead2628ff5c (2002-12-12)
Closes #14355

15 months agoGHA/non-native: reduce FreeBSD test parallelism to -j8 [ci skip]
Viktor Szakats [Fri, 2 Aug 2024 12:32:25 +0000 (14:32 +0200)] 
GHA/non-native: reduce FreeBSD test parallelism to -j8 [ci skip]

To reduce flakiness.

15 months agotests: gitignore newly generated files
Stefan Eissing [Fri, 2 Aug 2024 07:58:52 +0000 (09:58 +0200)] 
tests: gitignore newly generated files

Closes #14351

15 months agoprogress: ratelimit/progress tweaks
Stefan Eissing [Thu, 1 Aug 2024 11:05:49 +0000 (13:05 +0200)] 
progress: ratelimit/progress tweaks

- multi.c: when ratelimiting a transfer stops (MSTATE_RATELIMITING ->
  MSTATE_PERFORMING), run the MSTATE_PERFORMING state right away

- urldata.h: factor out upload and download progress counters into a
  struct, use that for passing these to progress update functions

- progress.c/getinfo.c: change names of moved progress counters

- progress.c: use new structs and a helper struct to factor repeated
  calculation into static helpers

Closes #14335

15 months agohttp2: improve rate limiting of downloads
Stefan Eissing [Thu, 1 Aug 2024 07:53:50 +0000 (09:53 +0200)] 
http2: improve rate limiting of downloads

Set the initial stream window size to 64KB and increase that to the 10MB
we used to start with on the first server reply, unless a rate limit is
in effect.

Continously monitory changes to the transfers rate limit and adjust the
stream window size accordingly. `max_recv_speed` is a transfer propert
that can be changed during processing by a callback.

Closes #14326

15 months agoGHA: update awslabs/aws-lc to v1.33.0
renovate[bot] [Fri, 2 Aug 2024 02:32:54 +0000 (02:32 +0000)] 
GHA: update awslabs/aws-lc to v1.33.0

Closes #14347

15 months agotests/http: configure test httpd to honor client cipher order
Stefan Eissing [Thu, 1 Aug 2024 14:16:55 +0000 (16:16 +0200)] 
tests/http: configure test httpd to honor client cipher order

Let the client, e.g. curl, influence the cipher selected in a TLS
handshake. TLS backends have different preferences and honor that
in httpd the same as Caddy does.

Also makes for a more fair compare of different TLS backends.

Closes #14338

15 months agodist: fix reproducible build from release tarball
Daniel Stenberg [Thu, 1 Aug 2024 11:51:43 +0000 (13:51 +0200)] 
dist: fix reproducible build from release tarball

Make it possible to rebuild an identical copy from a release tarball. It
was previously only possible from a checked out git repository.

- add release-tools.sh to dist
- keep Makefile.dist around to include it in dist
- regenerate tool_huge.c with the new version in dist
- fix the dist CI job to not do make clean like before

Closes #14336

15 months agocmake: add gnutls to multissl feature
Tal Regev [Thu, 1 Aug 2024 20:13:42 +0000 (23:13 +0300)] 
cmake: add gnutls to multissl feature

Closes #14345

15 months agocurl: allow 500MB data URL encode strings
Daniel Stenberg [Thu, 1 Aug 2024 14:45:50 +0000 (16:45 +0200)] 
curl: allow 500MB data URL encode strings

Previously it would bail out of the generated data reached 8MB in
memory.

Reported-by: Antoine du Hamel
Fixes #14337
Closes #14340

15 months agoescape: allow curl_easy_escape to generate 3*input length output
Daniel Stenberg [Thu, 1 Aug 2024 14:42:58 +0000 (16:42 +0200)] 
escape: allow curl_easy_escape to generate 3*input length output

Instead of capping it to the 3 * CURL_MAX_INPUT_LENGTH. To allow users
to URL encode larger chunks of data.

Closes #14339

15 months agoCHANGES: rename to CHANGES.md, no longer generated
Daniel Stenberg [Thu, 1 Aug 2024 07:58:31 +0000 (09:58 +0200)] 
CHANGES: rename to CHANGES.md, no longer generated

- scripts/log2changes.pl was not included in release tarballs, which broke
  reproducible builds

- since log2changes uses git to generate the contents, it makes it difficult
  to generate the same contents later (it would need to be fixed)

- the CHANGES file has outlived its purpose. the main changes are in the
  RELEASE-NOTES, the rest are better tracked directly using git or on GitHub

- put a fixed CHANGES.md in there instead pointing out where the info lives
  now

Closes #14331

15 months agoRELEASE-NOTES: synced
Daniel Stenberg [Thu, 1 Aug 2024 11:36:28 +0000 (13:36 +0200)] 
RELEASE-NOTES: synced

Plus add a link to https://curl.se/changes.html

15 months agoGHA: scan git repository and detect unvetted binary files
Daniel Stenberg [Thu, 1 Aug 2024 09:16:21 +0000 (11:16 +0200)] 
GHA: scan git repository and detect unvetted binary files

The only binary-looking files that are accepted in the git repository
need to match the checksums in the sha256sum file
".github/scripts/binarycheck.sums".

This is done to make sure that no one has planted any hidden (encrypted)
potentially dangerous payload in the tree.

Closes #14333

15 months agoGHA/windows: drop FTP tests
Viktor Szakats [Wed, 31 Jul 2024 19:53:15 +0000 (21:53 +0200)] 
GHA/windows: drop FTP tests

Try reducing hangs.

Closes #14330

15 months agoGHA/windows: remove vcpkg bin path in MSVC jobs
Tal Regev [Wed, 31 Jul 2024 18:49:51 +0000 (21:49 +0300)] 
GHA/windows: remove vcpkg bin path in MSVC jobs

- the path is wrong, because we compile on debug, and we are using
  the release bin path.
- the path is not needed, cmake curl copy the needed dlls to the
  compilation cmake folder where the curl exe is found.

Closes #14329

15 months agoGHA/windows: timeout earlier with hung tests
Viktor Szakats [Fri, 19 Jul 2024 20:42:42 +0000 (22:42 +0200)] 
GHA/windows: timeout earlier with hung tests

Finishing tests takes on average 10 or less minutes depending on
platform. Reduce job step timeouts to reflect that. It helps
concluding hung/failed tests earlier, which allows to retry them
earlier.

This makes it more difficult to tell from a job if it hung or not,
because we lose the long runtime as a telltale sign. Let's see how it
works out and adjust as necessary.

Also fix a comment while here.

Closes #14236

15 months agoINSTALL.md: MultiSSL and QUIC are mutually exclusive
Daniel Stenberg [Wed, 31 Jul 2024 16:40:44 +0000 (18:40 +0200)] 
INSTALL.md: MultiSSL and QUIC are mutually exclusive

Reported-by: Matt Jolly
Fixes #14308
Closes #14328

15 months agolib: convert some debugf()s into traces
Stefan Eissing [Wed, 31 Jul 2024 09:36:08 +0000 (11:36 +0200)] 
lib: convert some debugf()s into traces

Use CURL_TRC_CF() for some useful tracing information instead of
DEBUGF().

Closes #14322

15 months agocmake: distcheck for files in CMake subdir
Viktor Szakats [Wed, 31 Jul 2024 09:53:31 +0000 (11:53 +0200)] 
cmake: distcheck for files in CMake subdir

- add CMake option to verify if the `CMake/*.cmake`, `CMake/*.in` files
  are listed as distributable in autotools' `EXTRA_DIST`. The check can
  be enabled with `-DENABLE_DIST_TEST=ON` CMake option.

- add CI job to that effect.

Ref: #14320
Closes #14323

15 months agolibcurl.pc: add `Cflags.private`
Viktor Szakats [Wed, 31 Jul 2024 08:49:38 +0000 (10:49 +0200)] 
libcurl.pc: add `Cflags.private`

Ref: https://manpages.debian.org/bookworm/pkgconf/pc.5.en.html#PROPERTY_KEYWORDS
Ref: https://www.msys2.org/docs/pkgconfig/#cflagsprivate-static-libraries

Closes #14321

15 months agodist: add missing `FindNettle.cmake`
Viktor Szakats [Wed, 31 Jul 2024 08:02:13 +0000 (10:02 +0200)] 
dist: add missing `FindNettle.cmake`

Follow-up to 669ce42275635dc1f881dab3dfc9a55c9ab49b21 #14285
Reported-by: Christoph Reiter
Bug: https://github.com/curl/curl/pull/14285#issuecomment-2259880050
Closes #14320

15 months agotests: provide docs a as curldown, not nroff
Daniel Stenberg [Wed, 31 Jul 2024 10:56:33 +0000 (12:56 +0200)] 
tests: provide docs a as curldown, not nroff

As runtests.md and testcurl.md. Very few people actually need these as
manpages anyway.

With this, we have no more nroff formatted documents in git.

Closes #14324

15 months agoRELEASE-NOTES: synced
Daniel Stenberg [Wed, 31 Jul 2024 08:48:48 +0000 (10:48 +0200)] 
RELEASE-NOTES: synced

Bump to 8.10.0

15 months agorustls: make all tests pass
Joe Birr-Pixton [Tue, 30 Jul 2024 13:23:50 +0000 (14:23 +0100)] 
rustls: make all tests pass

- supports IP addresses in certs since 0.21
- Remove workaround, and re-enable tests

Closes #14317

15 months agoGHA/windows: enable MulitSSL in an MSVC job
Tal Regev [Wed, 31 Jul 2024 05:18:18 +0000 (08:18 +0300)] 
GHA/windows: enable MulitSSL in an MSVC job

Ref: 98da147b18fa4963efb425df8400ca40e7526646 #14305
Closes #14276

15 months agoRELEASE-NOTES: synced curl-8_9_1
Daniel Stenberg [Wed, 31 Jul 2024 06:50:04 +0000 (08:50 +0200)] 
RELEASE-NOTES: synced

15 months agoTHANKS: add names from the 8.9.1 release notes
Daniel Stenberg [Wed, 31 Jul 2024 06:50:04 +0000 (08:50 +0200)] 
THANKS: add names from the 8.9.1 release notes

Also remove duplicates

15 months agox509asn1: unittests and fixes for gtime2str
Stefan Eissing [Tue, 30 Jul 2024 14:40:48 +0000 (16:40 +0200)] 
x509asn1: unittests and fixes for gtime2str

Fix issues in GTime2str() and add unit test cases to verify correct
behaviour.

Follow-up to 3c914bc6801

Closes #14316

15 months agovtls: avoid forward declaration in MultiSSL builds
Tal Regev [Tue, 30 Jul 2024 03:07:50 +0000 (06:07 +0300)] 
vtls: avoid forward declaration in MultiSSL builds

The MSVC compiler cannot have forward declaration with const and static
variable, causing this error:
```
curl\lib\vtls\vtls.c(417,44): warning C4132: 'Curl_ssl_multi': const object should be initialized
```

Ref: #14276
Closes #14305

15 months agotidy-up: URL updates (one more)
Viktor Szakats [Tue, 30 Jul 2024 19:35:59 +0000 (21:35 +0200)] 
tidy-up: URL updates (one more)

Follow-up to 767d5811b5c783b42cea999dd42ecf0453085d17 #14318

15 months agotidy-up: URL updates
Viktor Szakats [Tue, 30 Jul 2024 16:05:02 +0000 (18:05 +0200)] 
tidy-up: URL updates

Closes #14318

15 months agocmake: drop `if(PKG_CONFIG_FOUND)` guard for `pkg_check_modules()`
Viktor Szakats [Tue, 30 Jul 2024 08:52:32 +0000 (10:52 +0200)] 
cmake: drop `if(PKG_CONFIG_FOUND)` guard for `pkg_check_modules()`

The oldest cmake supported by curl is v3.7.0, which already has such
guard (using `PKG_CONFIG_EXECUTABLE`) inside `pkg_check_modules()`. The
advantage of leaving that guard to CMake is that it will define/reset
all output variables, while the manual guard doesn't do this and also
leaves for example `NETTLE_FOUND` undefined.

Delete the single use of this guard from the recently added `nettle`
detection, where I included it by accident. Then possibly re-introduce
it universally if we find it useful after more evaluation.

Follow-up to 669ce42275635dc1f881dab3dfc9a55c9ab49b21 #14285
Closes #14309

15 months agomailmap: dedupe an author showing up twice in shortlog -s
Daniel Stenberg [Tue, 30 Jul 2024 12:25:30 +0000 (14:25 +0200)] 
mailmap: dedupe an author showing up twice in shortlog -s

15 months agomisc: cleanup after removing years from copyright
Ivan Kuchin [Tue, 30 Jul 2024 10:39:50 +0000 (12:39 +0200)] 
misc: cleanup after removing years from copyright

- remove leftover copyright years from few test files
- fix email in copyright lines
- consistent format of copyright lines

Closes #14312

15 months agowolfssl: avoid calling get_cached_x509_store if store is uncachable
Alex Snast [Tue, 30 Jul 2024 05:12:26 +0000 (08:12 +0300)] 
wolfssl: avoid calling get_cached_x509_store if store is uncachable

There's no need for get_cached_x509_store call if the return value won't
be used for caching anyway.

Closes #14306

15 months agocontrithanks.sh: use -F with -v to match lines as strings
Daniel Stenberg [Tue, 30 Jul 2024 10:14:48 +0000 (12:14 +0200)] 
contrithanks.sh: use -F with -v to match lines as strings

Makes names involving [brackets] work.

15 months agoGHA/non-native: bump FreeBSD/arm64 python modules
Viktor Szakats [Tue, 30 Jul 2024 09:52:27 +0000 (11:52 +0200)] 
GHA/non-native: bump FreeBSD/arm64 python modules

FreeBSD seems to upgrade their Python separately for arm64
and Intel. Today, arm64 caught up with the Intel packages.
Update our CI to reflect it.

Closes #14310

15 months agoGHA: bump github/codeql-action and msys2/setup-msys2
dependabot[bot] [Mon, 29 Jul 2024 14:45:58 +0000 (14:45 +0000)] 
GHA: bump github/codeql-action and msys2/setup-msys2

- bump github/codeql-action from 3.25.13 to 3.25.15

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.25.13 to 3.25.15.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/2d790406f505036ef40ecba973cc774a50395aac...afb54ba388a7dca6ecae48f608c4ff05ff4cc77a)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
Closes #14300

- bump msys2/setup-msys2 from 2.23.0 to 2.24.0

Bumps [msys2/setup-msys2](https://github.com/msys2/setup-msys2) from 2.23.0 to 2.24.0.
- [Release notes](https://github.com/msys2/setup-msys2/releases)
- [Changelog](https://github.com/msys2/setup-msys2/blob/main/CHANGELOG.md)
- [Commits](https://github.com/msys2/setup-msys2/compare/d0e80f58dffbc64f6a3a1f43527d469b4fc7b6c8...5df0ca6cbf14efcd08f8d5bd5e049a3cc8e07fd2)

---
updated-dependencies:
- dependency-name: msys2/setup-msys2
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
Closes #14301

15 months agox509asn1: clean up GTime2str
Daniel Stenberg [Tue, 30 Jul 2024 08:05:17 +0000 (10:05 +0200)] 
x509asn1: clean up GTime2str

Co-authored-by: Stefan Eissing
Reported-by: Dov Murik
Closes #14307

15 months agotool_operate: more defensive socket code
Daniel Stenberg [Mon, 29 Jul 2024 17:49:32 +0000 (19:49 +0200)] 
tool_operate: more defensive socket code

- use 'struct sockaddr' to getsockname() and its sa_family member

- use 'curl_socklen_t' instead of 'socklen_t'

- check for AF_INET6 to exist instead assuming it does

Should be generally more portable.

Reported-by: Harry Sintonen
Closes #14304

15 months agoconfigure: limit `__builtin_available` test to Darwin
Viktor Szakats [Sun, 14 Jul 2024 19:49:35 +0000 (21:49 +0200)] 
configure: limit `__builtin_available` test to Darwin

This feature test always fails on non-Apple systems. (For Apple targets
it's supported by llvm and Apple clang.)

Syncs behaviour with CMake.

Follow-up to cfd6f43d6ca7e57670b422bab7bbf10221a2cf3e #14127
Cherry-picked from #14097
Closes #14196

15 months agoRELEASE-NOTES: synced
Daniel Stenberg [Mon, 29 Jul 2024 19:24:32 +0000 (21:24 +0200)] 
RELEASE-NOTES: synced

15 months agoCURLOPT_SSL_CTX_FUNCTION.md: mention CA caching
Daniel Stenberg [Mon, 29 Jul 2024 16:58:40 +0000 (18:58 +0200)] 
CURLOPT_SSL_CTX_FUNCTION.md: mention CA caching

and add a few more see-also

Closes #14302

15 months agocmake: detect `libssh` via `pkg-config`
Viktor Szakats [Tue, 16 Jul 2024 10:48:13 +0000 (12:48 +0200)] 
cmake: detect `libssh` via `pkg-config`

Also:
- fix broken libssh `pkg-config` behaviour on old Linux.
  (when found, `LIBSSH_LINK_LIBRARIES` remains undefined.)

- delete manual libssh config from Old Linux CI job,
  it's no longer necessary.

Closes #14199

15 months agoGHA/non-native: improve, migrate x86_64 FreeBSD with tests from Cirrus CI
Viktor Szakats [Sat, 20 Jul 2024 11:31:45 +0000 (13:31 +0200)] 
GHA/non-native: improve, migrate x86_64 FreeBSD with tests from Cirrus CI

- run tests via `make test-ci` instead of `make check` with autotools.
- add `x86_64` job for FreeBSD, with tests.
  It matches the existing Cirrus CI job, with these differences:
  - finishes 3x faster (thanks to parallel tests enabled).
  - librtmp is not enabled because it's slated for removal by FreeBSD.
    (already past the removal deadline, thought the package still
    installs.)
  - DICT and TELNET servers fail to start. Couldn't figure out why.
    It means skipping test 1450 and 1452.
  - it runs more tests, e.g. websockets and ip6-localhost.
  - no `pkg update -f`.
  - it misses the `CRYPTOGRAPHY_DONT_BUILD_RUST=1`, `pkg delete curl`,
    `chmod 777`, `sudo -u nobody` and `sysctl net.inet.tcp.blackhole`
    tricks. The latter is the default in these runners, the others did
    not affect results.
- set `-j0` for tests in the NetBSD job. Flaky otherwise.

Closes #14244

15 months agocmake: detect `nettle` when building with GnuTLS
Viktor Szakats [Sat, 27 Jul 2024 09:47:56 +0000 (11:47 +0200)] 
cmake: detect `nettle` when building with GnuTLS

`nettle` is a direct dependency of curl, when building with GnuTLS.
Add a new `Find` module to detect it.

Also:
- GHA/macos: drop `nettle` hack no longer necessary.
- add `nettle` to `libcurl.pc`.
- also add `nettle` to `libcurl.pc` in autotools builds.

Follow-up to 781242ffa44a9f9b95b6da5ac5a1bf6372ec6257 #11967
Closes #14285

15 months agomacos: fix Apple SDK bug workaround for non-macOS targets
Viktor Szakats [Thu, 25 Jul 2024 22:10:21 +0000 (00:10 +0200)] 
macos: fix Apple SDK bug workaround for non-macOS targets

Turns out that MAC != OSX, despite what these names otherwise mean and
what's suggested by source code comments. "MAC" in fact means Darwin
(aka Apple), not macOS. "OSX" means macOS.

GitHub bumped the macos-14 runner default to Xcode 15.4, hitting the
llvm@15 incompatibility bug by default. Meaning the previous workaround
for the SDK bug is necessary.

This patch extend the workaround to not apply to mobile OS variants.

Follow-up to ff784af461175584c73e7e2b65af00b1a5a6f67f #14159
Reported-by: Sergey
Confirmed-by: Marcel Raad
Fixes #14269
Closes #14275

15 months agowolfssl: CA store share fix
Stefan Eissing [Fri, 26 Jul 2024 13:07:52 +0000 (15:07 +0200)] 
wolfssl: CA store share fix

When sharing the x509 store in wolfSSL, always use an explicitly
constructed one, as the SSLCTX might have "only" an internal one which
is not obeying reference count lifetimes.

Fixes #14278
Reported-by: Alex Snast
Closes #14279