Stefan Eissing [Sun, 4 Aug 2024 11:22:03 +0000 (13:22 +0200)]
wolfssl: improve shutdown handling
Improve handling of shutdown when sending gets blocked.
Add workaround for <https://github.com/wolfSSL/wolfssl/issues/7784>
where wolfSSL keeps on adding close notify messages to its outgoing
buffer on ever attempt.
Stefan Eissing [Sun, 4 Aug 2024 11:08:51 +0000 (13:08 +0200)]
openssl: improve shutdown handling
Make sure that `io_need` is cleared and set at the filter operations.
Add some more tracing for shutdown situations.
Improve shutdown handling for blocked sends. OpenSSL is a bit tricksy
here that it only reports WANT_WRITE on SSL_shutdown(), but never on
SSL_read() on blocked sends. So we need to use both.
At last, set SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER when available since we
are not always retrying sends from the very same address, as testing
showed.
Alex Snast [Tue, 25 Jun 2024 16:34:24 +0000 (09:34 -0700)]
wolfssl: add support for ssl cert blob / ssl key blob options
wolfSSL supports setting certificates/private keys from memory blobs
which allow us to implement both CURLOPT_SSLCERT_BLOB and
CURLOPT_SSLKEY_BLOB options.
Viktor Szakats [Sun, 4 Aug 2024 11:52:44 +0000 (13:52 +0200)]
cmake: add support for versioned symbols option
Implement the `--enable-versioned-symbols` feature available in
`./configure` for CMake.
Enable with `-DCURL_LIBCURL_VERSIONED_SYMBOLS=ON`. Customize the version
prefix with `-DCURL_LIBCURL_VERSIONED_SYMBOLS_PREFIX=MYPREFIX_`. By
default the prefix matches what autotools uses.
When enabled, the feature is detected and a warning shown if
unavailable. (E.g. on Apple and Windows, it isn't.)
Included `HIDDEN {};` to match autotools, though I don't know if it's
necessary, useful or making any difference.
Differences from the autotools implementation:
- soversion is dynamic instead of hard-coded.
- omits referencing non-curl symbols.
- allows prefix/flavour override.
- more universal feature detection.
- doesn't rely on the in-repo `lib/libcurl.vers.in` file.
Also:
- add mbedTLS and BearSSL versioned symbol prefix support to autotools.
- enable this option in an old-linux job.
Jay Satiro [Wed, 10 Jan 2024 08:53:23 +0000 (03:53 -0500)]
easy: fix curl_easy_upkeep for shared connection caches
- Determine which connection cache is or will be used by the easy handle
and perform connection upkeep on that cache.
This change allows curl_easy_upkeep to be effective on easy handles that
are using a shared connection cache, either from a user created shared
connection cache or a user created multi which has its own shared
connection cache.
Prior to this change curl_easy_upkeep would upkeep the connection cache
for the easy handle only if that cache was from the multi owned by the
easy handle (ie curl_easy_perform was previously called and there's a
connection cache exclusive to the easy handle in
data->multi_easy->conn_cache).
Jay Satiro [Tue, 23 Jul 2024 21:34:41 +0000 (17:34 -0400)]
vtls: stop offering alpn http/1.1 for http2-prior-knowledge
- For HTTPS if http2-prior-knowledge is set then only offer h2 (HTTP/2)
alpn to the server for protocol negotiation.
Prior to this change both HTTP/2 ("h2") and HTTP/1.1 ("http/1.1") were
offered for ALPN when http2-prior-knowledge was set.
CURL_HTTP_VERSION_2_PRIOR_KNOWLEDGE (tool: --http2-prior-knowledge) is
meant to send non-TLS HTTP requests HTTP/2 when it is known the server
supports them. However when HTTPS is used then it attempts to first
negotiate the connection with ALPN. In that case the user likely does
not want to offer http/1.1 to the server as an acceptable protocol.
Setting SSLHonorCipherOrder to on means it honors the server cipher
order. From the documentation: "When choosing a cipher during an SSLv3
or TLSv1 handshake, normally the client's preference is used. If this
directive is enabled, the server's preference will be used instead."
Also the commit inhibits test_17_07_ssl_ciphers. The test tries to
tests if all the ciphers specified, and only those, are properly set
in curl. For that to work we need have cases where some or all ciphers
do no intersect with the cipher-set of the server. We need to be able
to assert a failed connection based on a cipher set mismatch.
That is why a restricted set of ciphers is used on the server. This
set is so chosen that it contains the well known most secure ciphers.
Except with the slower aes256 variant intentionally left out, to be
able to test above described.
As test_17_07_ssl_ciphers is currently the only test that tests the
functioning of the --ciphers and --tls13-ciphers options, it is
important that its coverage is as good as possible.
Viktor Szakats [Fri, 2 Aug 2024 23:09:57 +0000 (01:09 +0200)]
tidy-up: OS names
Use these words and casing more consistently across text, comments and
one curl tool output:
AIX, ALPN, ANSI, BSD, Cygwin, Darwin, FreeBSD, GitHub, HP-UX, Linux,
macOS, MS-DOS, MSYS, MinGW, NTLM, POSIX, Solaris, UNIX, Unix, Unicode,
WINE, WebDAV, Win32, winbind, WinIDN, Windows, Windows CE, Winsock.
Daniel Stenberg [Sat, 3 Aug 2024 22:03:41 +0000 (00:03 +0200)]
maketgz: accept option to include latest commit hash
If the second argument to the script is "commit", then this will
generate a file named `docs/tarball-commit.txt` that contains the latest
commit hash (git rev-parse HEAD) at the time the script runs.
Doing this breaks the reproducibility so it will not be used for "real"
releases but is meant for automated daily snapshots and similar.
Reported-by: Dan Fandrich
Fixes #14363
Closes #14369
Daniel Stenberg [Sat, 3 Aug 2024 18:24:12 +0000 (20:24 +0200)]
curl: --help [option] displays documentation for given cmdline option
Since the documentation text blob might be gzipped, it needs to search
for what to output in a streaming manner. It then first searches for
"\nALL OPTIONS".
Then, it looks for the start to display at "\n -[option]" and stops
again at "\n -". Except for the last option in the man page, which
ends at "\nFILES" - the subtitle for the section following all options
in the manpage.
Viktor Szakats [Fri, 12 Jul 2024 19:10:57 +0000 (21:10 +0200)]
cmake: tidy-ups
- tidy-up comments.
- use lowercase, underscore prefixed names for internal variables.
- use `IN LISTS` and `IN ITEMS` in `foreach()` loops.
- rename variable name `OUTPUT` to a more distinctive one.
- tidy-up `STREQUAL` syntax.
- delete commented code.
- indent/whitespace.
Viktor Szakats [Sat, 3 Aug 2024 10:01:28 +0000 (12:01 +0200)]
GHA/windows: move Cygwin into its own workflow
Cygwin runs stable but slow, while native Windows in the same workflow
runs fast but unreliable (hangs). GHA requires all jobs to finish before
allowing to re-run failed ones. Before this patch this meant waiting for
the slow Cygwin jobs to re-run hung native jobs. After this patch Cygwin
jobs run in their own workflow, allowing to re-run broken Windows jobs
earlier.
Stefan Eissing [Thu, 25 Jul 2024 11:10:01 +0000 (13:10 +0200)]
cfilters: send flush
Since data can be held in connection filter buffers when sending gives
EAGAIN, add methods to query this and perform flushing of those buffers.
The transfer loop will continue sending until all upload data is
processed and the connection is flushed.
- add `CF_QUERY_SEND_PENDING` to query filters
- add `CF_CTRL_DATA_SEND_FLUSH` to flush filters
- change `Curl_req_want_send()` to query the connection
if it needs flushing
- use `Curl_req_want_send()` to determine the POLLOUT
in the PERFORMING multi state
- implement flush handling in the HTTP/2 connection filter
Stefan Eissing [Thu, 18 Jul 2024 09:29:37 +0000 (11:29 +0200)]
lib: add eos flag to send methods
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.
Stefan Eissing [Thu, 11 Jul 2024 10:44:15 +0000 (12:44 +0200)]
vtls: init ssl peer only once
- check that `struct ssl_peer` is only initialized once
- fix vtls peer init to run only once
- check in peer init that hostname is not empty, fail otherwise
Stefan Eissing [Fri, 12 Jul 2024 10:46:50 +0000 (12:46 +0200)]
url: dns_entry related improvements
Replace Curl_resolv_unlock() with Curl_resolv_unlink():
-replace inuse member with refcount in Curl_dns_entry
- pass Curl_dns_entry ** to unlink, so it gets always cleared
- solve potential (but unlikley) UAF in FTP's handling of looked up
Curl_dns_entry. Esp. do not use addr information after unlinking an entry.
In reality, the unlink will not free memory, as the dns entry is still
referenced by the hostcache. But this is not safe and relying on no other
code pruning the cache in the meantime.
- pass permanent flag when adding a dns entry instead of fixing timestamp
afterwards.
url.c: fold several static *resolve_* functions into one.
Stefan Eissing [Tue, 23 Jul 2024 10:21:51 +0000 (12:21 +0200)]
Curl_rand_bytes to control env override
- in DEBUGBUILD, all specifying if true random numbers
are desired or simulated ones via CURL_ENTROPY
- allows to use randoms in other DEBUG checks to not
interfere with the CURL_ENTROPY
- without this change, any Curl_rand() use will alter
results of some AUTHENTICATION methods like DIGEST
Dan Fandrich [Thu, 22 Jun 2023 17:03:16 +0000 (10:03 -0700)]
CI: enable parallel testing in CI builds
The test-ci target now uses 2 processes by default, but the amount of
parallelism is tuned for each CI service and build environment based on
results of a number of test runs. Some CI services use super-
oversubscribed build machines that can barely run the curl tests
already with no parallelism without frequently failing with
timing-induced failures. These continue to be run without parallelism.
Other services provide two fast, unloaded cores and these run with 14
processes, which is a good default for this kind of environment.
Here's a summary of the number of test processes by CI service:
- GHA/macos: disable libidn2 in combination builds, syncing with
autotools. Otherwise it's now auto-detected via pkg-config by default.
Follow-up to f43adc2c4978f7f82a359e89186e58a31d17b0ad #14137
Viktor Szakats [Sat, 29 Jun 2024 01:30:14 +0000 (03:30 +0200)]
curl: support embedding a CA bundle
Add the ability to embed a CA bundle into the curl binary. It is used
when no other runtime or build-time option set one.
This helps curl-for-win macOS and Linux builds to run standalone, and
also helps Windows builds to avoid picking up the CA bundle from an
arbitrary (possibly world-writable) location (though this behaviour is
not currently disablable).
Fixes:
```
curl\lib\vtls\cipher_suite.c(193,3): error C2220: the following warning is treated as an error
curl\lib\vtls\cipher_suite.c(193,3): warning C4310: cast truncates constant value
```
Daniel Stenberg [Fri, 2 Aug 2024 07:49:31 +0000 (09:49 +0200)]
Makefile: remove 'scripts' duplicate from DIST_SUBDIRS
Also fix the .dist replacing by avoiding all Makefiles because it
otherwise also went into the temporary release folder and got confused
about the Makefile.dist in there.
Stefan Eissing [Thu, 1 Aug 2024 07:53:50 +0000 (09:53 +0200)]
http2: improve rate limiting of downloads
Set the initial stream window size to 64KB and increase that to the 10MB
we used to start with on the first server reply, unless a rate limit is
in effect.
Continously monitory changes to the transfers rate limit and adjust the
stream window size accordingly. `max_recv_speed` is a transfer propert
that can be changed during processing by a callback.
Stefan Eissing [Thu, 1 Aug 2024 14:16:55 +0000 (16:16 +0200)]
tests/http: configure test httpd to honor client cipher order
Let the client, e.g. curl, influence the cipher selected in a TLS
handshake. TLS backends have different preferences and honor that
in httpd the same as Caddy does.
Also makes for a more fair compare of different TLS backends.
Daniel Stenberg [Thu, 1 Aug 2024 11:51:43 +0000 (13:51 +0200)]
dist: fix reproducible build from release tarball
Make it possible to rebuild an identical copy from a release tarball. It
was previously only possible from a checked out git repository.
- add release-tools.sh to dist
- keep Makefile.dist around to include it in dist
- regenerate tool_huge.c with the new version in dist
- fix the dist CI job to not do make clean like before
Daniel Stenberg [Thu, 1 Aug 2024 09:16:21 +0000 (11:16 +0200)]
GHA: scan git repository and detect unvetted binary files
The only binary-looking files that are accepted in the git repository
need to match the checksums in the sha256sum file
".github/scripts/binarycheck.sums".
This is done to make sure that no one has planted any hidden (encrypted)
potentially dangerous payload in the tree.
Tal Regev [Wed, 31 Jul 2024 18:49:51 +0000 (21:49 +0300)]
GHA/windows: remove vcpkg bin path in MSVC jobs
- the path is wrong, because we compile on debug, and we are using
the release bin path.
- the path is not needed, cmake curl copy the needed dlls to the
compilation cmake folder where the curl exe is found.
Viktor Szakats [Fri, 19 Jul 2024 20:42:42 +0000 (22:42 +0200)]
GHA/windows: timeout earlier with hung tests
Finishing tests takes on average 10 or less minutes depending on
platform. Reduce job step timeouts to reflect that. It helps
concluding hung/failed tests earlier, which allows to retry them
earlier.
This makes it more difficult to tell from a job if it hung or not,
because we lose the long runtime as a telltale sign. Let's see how it
works out and adjust as necessary.
Viktor Szakats [Wed, 31 Jul 2024 09:53:31 +0000 (11:53 +0200)]
cmake: distcheck for files in CMake subdir
- add CMake option to verify if the `CMake/*.cmake`, `CMake/*.in` files
are listed as distributable in autotools' `EXTRA_DIST`. The check can
be enabled with `-DENABLE_DIST_TEST=ON` CMake option.
Tal Regev [Tue, 30 Jul 2024 03:07:50 +0000 (06:07 +0300)]
vtls: avoid forward declaration in MultiSSL builds
The MSVC compiler cannot have forward declaration with const and static
variable, causing this error:
```
curl\lib\vtls\vtls.c(417,44): warning C4132: 'Curl_ssl_multi': const object should be initialized
```
Viktor Szakats [Tue, 30 Jul 2024 08:52:32 +0000 (10:52 +0200)]
cmake: drop `if(PKG_CONFIG_FOUND)` guard for `pkg_check_modules()`
The oldest cmake supported by curl is v3.7.0, which already has such
guard (using `PKG_CONFIG_EXECUTABLE`) inside `pkg_check_modules()`. The
advantage of leaving that guard to CMake is that it will define/reset
all output variables, while the manual guard doesn't do this and also
leaves for example `NETTLE_FOUND` undefined.
Delete the single use of this guard from the recently added `nettle`
detection, where I included it by accident. Then possibly re-introduce
it universally if we find it useful after more evaluation.
Viktor Szakats [Sat, 20 Jul 2024 11:31:45 +0000 (13:31 +0200)]
GHA/non-native: improve, migrate x86_64 FreeBSD with tests from Cirrus CI
- run tests via `make test-ci` instead of `make check` with autotools.
- add `x86_64` job for FreeBSD, with tests.
It matches the existing Cirrus CI job, with these differences:
- finishes 3x faster (thanks to parallel tests enabled).
- librtmp is not enabled because it's slated for removal by FreeBSD.
(already past the removal deadline, thought the package still
installs.)
- DICT and TELNET servers fail to start. Couldn't figure out why.
It means skipping test 1450 and 1452.
- it runs more tests, e.g. websockets and ip6-localhost.
- no `pkg update -f`.
- it misses the `CRYPTOGRAPHY_DONT_BUILD_RUST=1`, `pkg delete curl`,
`chmod 777`, `sudo -u nobody` and `sysctl net.inet.tcp.blackhole`
tricks. The latter is the default in these runners, the others did
not affect results.
- set `-j0` for tests in the NetBSD job. Flaky otherwise.
Viktor Szakats [Thu, 25 Jul 2024 22:10:21 +0000 (00:10 +0200)]
macos: fix Apple SDK bug workaround for non-macOS targets
Turns out that MAC != OSX, despite what these names otherwise mean and
what's suggested by source code comments. "MAC" in fact means Darwin
(aka Apple), not macOS. "OSX" means macOS.
GitHub bumped the macos-14 runner default to Xcode 15.4, hitting the
llvm@15 incompatibility bug by default. Meaning the previous workaround
for the SDK bug is necessary.
This patch extend the workaround to not apply to mobile OS variants.
Stefan Eissing [Fri, 26 Jul 2024 13:07:52 +0000 (15:07 +0200)]
wolfssl: CA store share fix
When sharing the x509 store in wolfSSL, always use an explicitly
constructed one, as the SSLCTX might have "only" an internal one which
is not obeying reference count lifetimes.
Fixes #14278 Reported-by: Alex Snast
Closes #14279