Stefan Eissing [Thu, 25 Jul 2024 11:10:01 +0000 (13:10 +0200)]
cfilters: send flush
Since data can be held in connection filter buffers when sending gives
EAGAIN, add methods to query this and perform flushing of those buffers.
The transfer loop will continue sending until all upload data is
processed and the connection is flushed.
- add `CF_QUERY_SEND_PENDING` to query filters
- add `CF_CTRL_DATA_SEND_FLUSH` to flush filters
- change `Curl_req_want_send()` to query the connection
if it needs flushing
- use `Curl_req_want_send()` to determine the POLLOUT
in the PERFORMING multi state
- implement flush handling in the HTTP/2 connection filter
Stefan Eissing [Thu, 18 Jul 2024 09:29:37 +0000 (11:29 +0200)]
lib: add eos flag to send methods
Adds a `bool eos` flag to send methods to indicate that the data
is the last chunk the invovled transfer wants to send to the server.
This will help protocol filters like HTTP/2 and 3 to forward the
stream's EOF flag and also allow to EAGAIN such calls when buffers
are not yet fully flushed.
Stefan Eissing [Thu, 11 Jul 2024 10:44:15 +0000 (12:44 +0200)]
vtls: init ssl peer only once
- check that `struct ssl_peer` is only initialized once
- fix vtls peer init to run only once
- check in peer init that hostname is not empty, fail otherwise
Stefan Eissing [Fri, 12 Jul 2024 10:46:50 +0000 (12:46 +0200)]
url: dns_entry related improvements
Replace Curl_resolv_unlock() with Curl_resolv_unlink():
-replace inuse member with refcount in Curl_dns_entry
- pass Curl_dns_entry ** to unlink, so it gets always cleared
- solve potential (but unlikley) UAF in FTP's handling of looked up
Curl_dns_entry. Esp. do not use addr information after unlinking an entry.
In reality, the unlink will not free memory, as the dns entry is still
referenced by the hostcache. But this is not safe and relying on no other
code pruning the cache in the meantime.
- pass permanent flag when adding a dns entry instead of fixing timestamp
afterwards.
url.c: fold several static *resolve_* functions into one.
Stefan Eissing [Tue, 23 Jul 2024 10:21:51 +0000 (12:21 +0200)]
Curl_rand_bytes to control env override
- in DEBUGBUILD, all specifying if true random numbers
are desired or simulated ones via CURL_ENTROPY
- allows to use randoms in other DEBUG checks to not
interfere with the CURL_ENTROPY
- without this change, any Curl_rand() use will alter
results of some AUTHENTICATION methods like DIGEST
Dan Fandrich [Thu, 22 Jun 2023 17:03:16 +0000 (10:03 -0700)]
CI: enable parallel testing in CI builds
The test-ci target now uses 2 processes by default, but the amount of
parallelism is tuned for each CI service and build environment based on
results of a number of test runs. Some CI services use super-
oversubscribed build machines that can barely run the curl tests
already with no parallelism without frequently failing with
timing-induced failures. These continue to be run without parallelism.
Other services provide two fast, unloaded cores and these run with 14
processes, which is a good default for this kind of environment.
Here's a summary of the number of test processes by CI service:
- GHA/macos: disable libidn2 in combination builds, syncing with
autotools. Otherwise it's now auto-detected via pkg-config by default.
Follow-up to f43adc2c4978f7f82a359e89186e58a31d17b0ad #14137
Viktor Szakats [Sat, 29 Jun 2024 01:30:14 +0000 (03:30 +0200)]
curl: support embedding a CA bundle
Add the ability to embed a CA bundle into the curl binary. It is used
when no other runtime or build-time option set one.
This helps curl-for-win macOS and Linux builds to run standalone, and
also helps Windows builds to avoid picking up the CA bundle from an
arbitrary (possibly world-writable) location (though this behaviour is
not currently disablable).
Fixes:
```
curl\lib\vtls\cipher_suite.c(193,3): error C2220: the following warning is treated as an error
curl\lib\vtls\cipher_suite.c(193,3): warning C4310: cast truncates constant value
```
Daniel Stenberg [Fri, 2 Aug 2024 07:49:31 +0000 (09:49 +0200)]
Makefile: remove 'scripts' duplicate from DIST_SUBDIRS
Also fix the .dist replacing by avoiding all Makefiles because it
otherwise also went into the temporary release folder and got confused
about the Makefile.dist in there.
Stefan Eissing [Thu, 1 Aug 2024 07:53:50 +0000 (09:53 +0200)]
http2: improve rate limiting of downloads
Set the initial stream window size to 64KB and increase that to the 10MB
we used to start with on the first server reply, unless a rate limit is
in effect.
Continously monitory changes to the transfers rate limit and adjust the
stream window size accordingly. `max_recv_speed` is a transfer propert
that can be changed during processing by a callback.
Stefan Eissing [Thu, 1 Aug 2024 14:16:55 +0000 (16:16 +0200)]
tests/http: configure test httpd to honor client cipher order
Let the client, e.g. curl, influence the cipher selected in a TLS
handshake. TLS backends have different preferences and honor that
in httpd the same as Caddy does.
Also makes for a more fair compare of different TLS backends.
Daniel Stenberg [Thu, 1 Aug 2024 11:51:43 +0000 (13:51 +0200)]
dist: fix reproducible build from release tarball
Make it possible to rebuild an identical copy from a release tarball. It
was previously only possible from a checked out git repository.
- add release-tools.sh to dist
- keep Makefile.dist around to include it in dist
- regenerate tool_huge.c with the new version in dist
- fix the dist CI job to not do make clean like before
Daniel Stenberg [Thu, 1 Aug 2024 09:16:21 +0000 (11:16 +0200)]
GHA: scan git repository and detect unvetted binary files
The only binary-looking files that are accepted in the git repository
need to match the checksums in the sha256sum file
".github/scripts/binarycheck.sums".
This is done to make sure that no one has planted any hidden (encrypted)
potentially dangerous payload in the tree.
Tal Regev [Wed, 31 Jul 2024 18:49:51 +0000 (21:49 +0300)]
GHA/windows: remove vcpkg bin path in MSVC jobs
- the path is wrong, because we compile on debug, and we are using
the release bin path.
- the path is not needed, cmake curl copy the needed dlls to the
compilation cmake folder where the curl exe is found.
Viktor Szakats [Fri, 19 Jul 2024 20:42:42 +0000 (22:42 +0200)]
GHA/windows: timeout earlier with hung tests
Finishing tests takes on average 10 or less minutes depending on
platform. Reduce job step timeouts to reflect that. It helps
concluding hung/failed tests earlier, which allows to retry them
earlier.
This makes it more difficult to tell from a job if it hung or not,
because we lose the long runtime as a telltale sign. Let's see how it
works out and adjust as necessary.
Viktor Szakats [Wed, 31 Jul 2024 09:53:31 +0000 (11:53 +0200)]
cmake: distcheck for files in CMake subdir
- add CMake option to verify if the `CMake/*.cmake`, `CMake/*.in` files
are listed as distributable in autotools' `EXTRA_DIST`. The check can
be enabled with `-DENABLE_DIST_TEST=ON` CMake option.
Tal Regev [Tue, 30 Jul 2024 03:07:50 +0000 (06:07 +0300)]
vtls: avoid forward declaration in MultiSSL builds
The MSVC compiler cannot have forward declaration with const and static
variable, causing this error:
```
curl\lib\vtls\vtls.c(417,44): warning C4132: 'Curl_ssl_multi': const object should be initialized
```
Viktor Szakats [Tue, 30 Jul 2024 08:52:32 +0000 (10:52 +0200)]
cmake: drop `if(PKG_CONFIG_FOUND)` guard for `pkg_check_modules()`
The oldest cmake supported by curl is v3.7.0, which already has such
guard (using `PKG_CONFIG_EXECUTABLE`) inside `pkg_check_modules()`. The
advantage of leaving that guard to CMake is that it will define/reset
all output variables, while the manual guard doesn't do this and also
leaves for example `NETTLE_FOUND` undefined.
Delete the single use of this guard from the recently added `nettle`
detection, where I included it by accident. Then possibly re-introduce
it universally if we find it useful after more evaluation.
Viktor Szakats [Sat, 20 Jul 2024 11:31:45 +0000 (13:31 +0200)]
GHA/non-native: improve, migrate x86_64 FreeBSD with tests from Cirrus CI
- run tests via `make test-ci` instead of `make check` with autotools.
- add `x86_64` job for FreeBSD, with tests.
It matches the existing Cirrus CI job, with these differences:
- finishes 3x faster (thanks to parallel tests enabled).
- librtmp is not enabled because it's slated for removal by FreeBSD.
(already past the removal deadline, thought the package still
installs.)
- DICT and TELNET servers fail to start. Couldn't figure out why.
It means skipping test 1450 and 1452.
- it runs more tests, e.g. websockets and ip6-localhost.
- no `pkg update -f`.
- it misses the `CRYPTOGRAPHY_DONT_BUILD_RUST=1`, `pkg delete curl`,
`chmod 777`, `sudo -u nobody` and `sysctl net.inet.tcp.blackhole`
tricks. The latter is the default in these runners, the others did
not affect results.
- set `-j0` for tests in the NetBSD job. Flaky otherwise.
Viktor Szakats [Thu, 25 Jul 2024 22:10:21 +0000 (00:10 +0200)]
macos: fix Apple SDK bug workaround for non-macOS targets
Turns out that MAC != OSX, despite what these names otherwise mean and
what's suggested by source code comments. "MAC" in fact means Darwin
(aka Apple), not macOS. "OSX" means macOS.
GitHub bumped the macos-14 runner default to Xcode 15.4, hitting the
llvm@15 incompatibility bug by default. Meaning the previous workaround
for the SDK bug is necessary.
This patch extend the workaround to not apply to mobile OS variants.
Stefan Eissing [Fri, 26 Jul 2024 13:07:52 +0000 (15:07 +0200)]
wolfssl: CA store share fix
When sharing the x509 store in wolfSSL, always use an explicitly
constructed one, as the SSLCTX might have "only" an internal one which
is not obeying reference count lifetimes.
Fixes #14278 Reported-by: Alex Snast
Closes #14279
curl: support __ss_family use on NonStop platforms
The definition of sockaddr_storage incorrectly specifies the ss_family
field as __ss_family. This fix conditionally allows builds to succeed on
all NonStop platforms.
Signed-off-by: Randall S. Becker <randall.becker@nexbridge.ca>
Closes #14273
Stefan Eissing [Mon, 29 Jul 2024 08:23:20 +0000 (10:23 +0200)]
connect: fix connection shutdown for event based processing
connections being shutdown would register sockets for events, but then
never remove these sockets again. Nor would the shutdown effectively
been performed.
- If a socket event involves a transfer, check if that is the
connection cache internal handle and run its multi_perform()
instead (the internal handle is used for all shutdowns).
- When a timer triggers for a transfer, check also if it is
about the connection cache internal handle.
- During processing shutdowns in the connection cache, assess
the shutdown timeouts. Register a Curl_expire() of the lowest
value for the cache's internal handle.
Reported-by: Gordon Parke
Fixes #14280
Closes #14296
Daniel Stenberg [Mon, 29 Jul 2024 08:17:08 +0000 (10:17 +0200)]
tests: provide FTP directory contents in the test file
Instead of providing a fixed single synthetic response in the test
server itself. To allow us to better use *different* directory listings
in different test cases. In this change, most listings remain the same
as before.
The wildcard match tests still use synthetic responses but we should fix
that as well.
Viktor Szakats [Sat, 20 Jul 2024 16:20:06 +0000 (18:20 +0200)]
runtests: fold timing details with GHA, sync `-r` tflags
- move timing details into a foldable group when run in GitHub Actions.
Spec:
https://docs.github.com/en/actions/using-workflows/workflow-commands-for-github-actions#grouping-log-lines
- enable `-r` (run time stats) option in autotools' `test-ci` target,
syncing it with cmake.
Stefan Eissing [Fri, 26 Jul 2024 08:38:45 +0000 (10:38 +0200)]
transfer: speed limiting fix for 32bit systems
When checking if a speed limit on receives applies, compare the receive
sizes using the large int type to prevent an overflow on systems where
size_t is 32bit.
- implement the socket hash user/reader/writer processing also
for connections that are being shut down by the connection cache.
- split out handling of current vs. last pollset socket event handling
into a function available in other code parts
- add `shutdown_poll` pollset to `connectdata` struct so that changes
in the pollset can be recorded during shutdown. (The internal handle
cannot keep it since it might be used for many connections)
Reported-by: calvin2021y on github
Fixes #14252
Closes #14257
Daniel Stenberg [Sat, 20 Jul 2024 21:21:16 +0000 (23:21 +0200)]
tool_cb_prg: output "flying saucers" with leading carriage return
Because that is how the progress-bar is output, so when the progress-bar
has been shown at least once and the information is reset, like for a
redirect, there might be a moment where the size goes from known to
unknown and then the flying saucerts are shown after a brief display of
the progress-bar.
It could previously cause accidental character leftovers on the right
side of the bar when using a narrow display.
Reported-by: Chris Webb
Fixes #14213
Closes #14246