Viktor Szakats [Thu, 12 Sep 2024 13:16:51 +0000 (15:16 +0200)]
cmake: fix broken dependency chain for cmdline-opts, tidy-ups
- make `curl.1` and `curl.txt` depend on `DPAGES`.
To trigger a rebuild when an individual manpage is updated.
- tell CMake that the cmdline-opts command also creates `curl.txt`.
- make `tool_hugehelp.c` depend on `curl.txt` (was: `curl.1`), to match
what it actually uses for input.
- stop using `generate-curl.1` as an indirect way to create `curl.txt`
in time for `tool_hugehelp.c`. After the fixes above there is a direct
depedency chain between them.
- move `ASCIIPAGE` and `MANPAGE` variables to top-level, re-use them in
`src` and prefix them with `CURL_` to avoid clashing with other
projects.
- drop double quotes from `generate-curl.1` as a hint that it is not
a filename, but a target name.
Viktor Szakats [Thu, 5 Sep 2024 00:11:35 +0000 (02:11 +0200)]
build: tidy up deprecation suppression, enable warnings for clang
Suppress deprecation warnings the closest to the deprecated code, using
`CURL_IGNORE_DEPRECATION()`. Then drop build-specific suppressions, and
file-wide ones. The latter is not compatible with Unity mode. Also
replace manual suppressions with a macro to apply to all compilers with
deprecation warning support. Also enable deprecation warnings for clang.
- curl/curl.h: enable deprecation warnings for clang.
- docs/examples: stop setting `CURL_DISABLE_DEPRECATION` with autotools.
Suppression moved to C-level earlier. Syncs with cmake.
Follow-up to 5fc61a37c1b177cefbc11dc9f5eef7b2a14538da #14123
- tests/http/clients: stop setting `CURL_DISABLE_DEPRECATION` in
autotools. If it becomes necessary in the future, it can be done in
C via the macro. Syncs with cmake.
Viktor Szakats [Tue, 20 Aug 2024 09:13:19 +0000 (11:13 +0200)]
cmake: expand `CURL_USE_PKGCONFIG` to non-cross `MINGW`
Enable `CURL_USE_PKGCONFIG` by default for more environments:
- for `MINGW` targets when not using cross-compilation.
- stop restricting vcpkg to MSVC. (this currently unlocks mingw,
also unlocked by the update above.)
Viktor Szakats [Thu, 19 Sep 2024 20:38:04 +0000 (22:38 +0200)]
GHA: speed up builds in torture jobs, tidy up
- use cmake with ninja.
- drop unnecessary packages.
Also:
- Linux: switch to Ubuntu 20.24 runner. Initially for newer packages,
then left it there for variation and as canary.
- Linux: make it easier to enable valgrind by dropping `-n` tflags.
- Linux: show `curl -V` after build.
- Linux: sync job names with macOS.
- Linux, macOS: review and adjust job timeouts.
- Linux, macOS: sync job configs more:
Linux: enable libssh2, macOS: enable brotli, zstd, libssh2.
- macOS: prefer `source` (over manual `PATH` editing) for `venv` setup.
- macOS: drop redundant `-DENABLE_CURLDEBUG=ON`.
- macOS: drop redundant `-n` tflags. There is no valgrind on macOS.
- macOS: allow overriding test parallelism per job.
- macOS: unlock all disabled tests for torture jobs.
Viktor Szakats [Fri, 6 Sep 2024 08:26:06 +0000 (10:26 +0200)]
autotools: add support for 'unity' builds, enable in CI
Implement the "unity" builds as known from CMake, but for autotools.
It's limited to `lib` and `src` (CMake also supports it in `tests`).
Enable with: `--enable-unity` (disabled by default)
Unity builds speed up builds significantly. Cygwin and Windows builds in
particular, but the effect is noticeable on most systems. It also allows
discovering unity issues with autotools, benefitting also CMake when
building the same combination. In CI it makes turnaround times quicker.
This closes build performance with CMake. autotools still lags behind
because it builds shared and static libcurl in two, separate passes.
CMake does it in one. Manpage compilation isn't batched, it is in CMake.
After unity and test bundle support the slowest parts of the build are
the configuration phase (which is effectively a tedious, non-parallel,
compilation and/or linking of 300+ tiny programs. The next bottleneck
is compiling individual examples and finally test servers (only slow
with autotools).
The autotools implementation is slightly less efficient than CMake,
because 3 sources are permanently excluded while in CMake this isn't
necessary and solved more efficiently while building libtests. There is
also no 'unity' support for tests, making them a less efficient also.
Enable it in CI for most `configure` jobs. Except in GHA/dist (though
it works fine there too), to use the default config there. Also skip for
the Linux AWC-LC job where it made builds time a few seconds longer
(reason undiscovered.)
Viktor Szakats [Sat, 14 Sep 2024 00:21:33 +0000 (02:21 +0200)]
cmake: separate target for examples, optimize CI, fix fallouts
- Move `docs/examples` builds under a separate target.
- Make `BUILD_EXAMPLES` default to `ON`. It means to generate the rules
for `docs/examples` by default, but not build them. To build them,
an explicit `make curl-examples` (or ninja, etc) command is necessary.
This syncs behaviour with autotools, and also how both cmake and
autotools are building tests.
- GHA: update cmake jobs to use the new way of building examples.
- GHA: move examples build step at the end of the job, after building
and running tests. This allows to have build and test run results
faster, and leave the seldom-changing examples build to the end.
Building examples is the slowest build step with no practical way to
make them fast.
- appveyor: enable building examples in two old-MSVC jobs.
- examples: fix examples to build cleanly with old MSVC versions.
- GHA/non-native: move example build log under a GHA foldable section.
- GHA/windows: move building examples into separate step for Linux cross
jobs.
Stefan Eissing [Tue, 10 Sep 2024 12:08:17 +0000 (14:08 +0200)]
request: on shutdown send, proceed normally on timeout
When ending an FTP upload, we shut down the connection gracefully, since
the server should be notified we had send all bytes. Mostly, this is a
NOP without TLS involved. With TLS, close-notify messages should be
exchanged.
As reported in #14843, not all servers seem to do that. Since it is the
server's responsiblity to check it has received everything, we just log
the timeout and proceed as if everything is fine.
In the receive direction, we still fail the transfer if the server does
not shut down its direction properly.
Stefan Eissing [Thu, 19 Sep 2024 10:23:55 +0000 (12:23 +0200)]
alt-svc: honor data->state.httpwant
When a transfer is set for a speficif HTTP version, only lookup that
protocol in the alt-svc mappings. When no speicific version is
requested, scan all entries as before.
Stefan Eissing [Thu, 19 Sep 2024 09:47:29 +0000 (11:47 +0200)]
url: connection reuse on h3 connections
- When searching for existing connections, interpret the
default CURL_HTTP_VERSION_2TLS as "anything goes". This
will allow us to reuse HTTP/3 connections better
- add 'http/1.1' as allowed protocol identifier in Alt-Svc
files
- add test_02_0[345] for testing protocol selection on
provided alt-svc files
Stefan Eissing [Fri, 20 Sep 2024 14:06:25 +0000 (16:06 +0200)]
libssh.c: handle EGAINS during proto-connect correctly
The implementation did not check the poll flags when a ssh_connect()
EAGAINed. Also the poll check did not allow for both directions
to be signalled by libssh, which it does.
This appeared during CI testing with higher paralellism.
Stefan Eissing [Fri, 20 Sep 2024 13:58:21 +0000 (15:58 +0200)]
multi.c: make stronger check for paused transfer before asserting
With higher parallelism in CI, the ASSERT triggered on pause tests.
Strengthen the check. We might want to think about removing
KEEP_RECV_PAUSE|KEEP_SEND_PAUSE altogether.
Viktor Szakats [Thu, 19 Sep 2024 22:51:35 +0000 (00:51 +0200)]
GHA: keep default pkgconf, do not replace with pkg-config on Linux
Ubuntu has the `pkgconf` package installed by default that implements
the `pkg-config` command. Switch CI `apt` commands over to `pkgconf`
to avoid replacing it in every job run.
Avoids:
```
The following packages will be REMOVED:
pkgconf r-base-dev
The following NEW packages will be installed:
[...] pkg-config [...]
```
https://github.com/curl/curl/actions/runs/10949915766/job/30404126342?pr=14972#step:2:20
Viktor Szakats [Wed, 4 Sep 2024 01:25:09 +0000 (03:25 +0200)]
unit1660: fix unreachable code warning in no-SSL builds
Or no-HTTP, no-HSTS builds, also MSH3 builds.
CMake, VS2022, Debug, x64, no SSL, Static, Build-only:
```
C:\projects\curl\tests\unit\unit1660.c(46,1): error C2220: the following warning is treated as an error
C:\projects\curl\tests\unit\unit1660.c(46,1): warning C4702: unreachable code
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/50531210/job/cjewvo9agavthr3o#L216
Required-by: #14922
Cherry-picked from #14772
Closes #14971
Viktor Szakats [Mon, 2 Sep 2024 22:26:26 +0000 (00:26 +0200)]
cmake: allow building tests in unity mode
Makes building tests noticeably faster.
Apply changes/fixes/workarounds to make Unity work:
- rename test variables to avoid collisions or shadowing each other when
combined into single units.
- add workaround to avoid applying `lib/memdebug.h` overrides to system
headers declaring/defining `getaddrinfo()`/`freeaddrinfo()` for
`tests/server/resolve.c`. This replaces a previous workaround that
worked for that specific source.
- rename test macro `CTRL` clashing with Cygwin `sys/ioctl.h`.
- add include guard to `test.h`.
Also:
- exclude `tests/http/clients` which are all single-source. (like
`docs/examples`.)
Viktor Szakats [Fri, 6 Sep 2024 12:09:40 +0000 (14:09 +0200)]
lib: fix unity builds with BearSSL, MSH3, Quiche, OmniOS
- fix MSH3 static symbol clash.
- fix Quiche static symbol clash.
- fix local macro clash with BearSSL header.
- fix local macro clash with OmniOS system header.
```
In file included from ../../lib/urldata.h:197,
from ../../lib/altsvc.c:32,
from libcurlall.c:2:
../../lib/cf-socket.h:55:25: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token
55 | #define sa_addr _sa_ex_u.addr
| ^
In file included from ../../lib/urldata.h:197,
from ../../lib/altsvc.c:32,
from libcurlall.c:2:
../../lib/cf-socket.h:55:25: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token
55 | #define sa_addr _sa_ex_u.addr
| ^
```
Ref: https://github.com/curl/curl/actions/runs/10738314933/job/29781644299?pr=14772#step:3:6115
Discovered while adding support for "unity" builds for autotools.
Required-by: #14922
Cherry-picked from #14815
Closes #14932
Dan Fandrich [Tue, 17 Sep 2024 22:14:31 +0000 (15:14 -0700)]
tests: Only log warnings or worse by default in smbserver
There shouldn't be anything displayed during a normal run, but only if
server debugging is enabled. Also, set log_file to a magic value to
disable it, otherwise impacket installs its own logger that messes with
what we want.
Viktor Szakats [Mon, 2 Sep 2024 21:34:13 +0000 (23:34 +0200)]
cmake, `Makefile.mk`: use `-isystem` for dep headers, silence BearSSL issues
Patch started out for working around compiler warnings in BearSSL latest
tarball release v0.6 (2018-08-14) and Apple clang 14 with CMake.
Then turned into patching CMake and `Makefile.mk` builds to use
`-isystem` instead `-I` when adding header directories for
dependencies. This avoids compiler warnings in dependency headers,
syncing behaviour with autotools.
Also:
- `Makefile.mk`: add support for BearSSL.
- delete warning suppression for mbedTLS headers. No longer necessary
after this patch.
Follow-up to 434db995a7566a76f6a16391897816ddd1011adc #12720
Silenced BearSSL warnings:
```
In file included from curl/lib/vtls/bearssl.c:28:
In file included from bearssl/inc/bearssl.h:127:
bearssl/inc/bearssl_hash.h:727:5: warning: 'BR_DOXYGEN_IGNORE' is not defined, evaluates to 0 [-Wundef]
^
bearssl/inc/bearssl_hash.h:745:5: warning: 'BR_DOXYGEN_IGNORE' is not defined, evaluates to 0 [-Wundef]
^
In file included from curl/lib/vtls/bearssl.c:28:
In file included from bearssl/inc/bearssl.h:136:
bearssl/inc/bearssl_ssl.h:1253:20: warning: implicit conversion loses integer precision: 'unsigned int' to 'uint16_t' (aka 'unsigned short') [-Wimplicit-int-conversion]
cc->version_min = version_min;
~ ^~~~~~~~~~~
bearssl/inc/bearssl_ssl.h:1254:20: warning: implicit conversion loses integer precision: 'unsigned int' to 'uint16_t' (aka 'unsigned short') [-Wimplicit-int-conversion]
cc->version_max = version_max;
~ ^~~~~~~~~~~
bearssl/inc/bearssl_ssl.h:1327:28: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'uint16_t' (aka 'unsigned short') [-Wimplicit-int-conversion]
ctx->protocol_names_num = num;
~ ^~~
5 warnings generated.
```
Also these two cases, which are caused by an unidentified component
(outside curl) cranking up MSVC warnings in external headers to `/W4`
when ZLIB is deselected:
https://github.com/curl/curl/pull/14859#issuecomment-2351809153
mbedTLS 3.6.1:
```
C:\vcpkg\installed\x64-windows\include\psa\crypto_struct.h(254,13): error C2220: the following warning is treated as an error [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
C:\vcpkg\installed\x64-windows\include\psa\crypto_struct.h(254,13): warning C4200: nonstandard extension used: zero-sized array in struct/union [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
```
Ref: https://github.com/curl/curl/actions/runs/10842694205/job/30107466989?pr=14859#step:10:29
nghttp3 1.5.0:
```
C:\vcpkg\installed\x64-windows\include\nghttp3\nghttp3.h(2678,1): error C2220: the following warning is treated as an error [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
C:\vcpkg\installed\x64-windows\include\nghttp3\nghttp3.h(2678,1): warning C4324: 'nghttp3_pri': structure was padded due to alignment specifier [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
```
Ref: https://github.com/curl/curl/actions/runs/10871875297/job/30166233862?pr=14859#step:10:28
Viktor Szakats [Mon, 26 Aug 2024 11:53:41 +0000 (13:53 +0200)]
cmake: delete unused `NEED_LBER_H`, `HAVE_LDAP_H`
Both are used by `./configure` internally, not by curl C code.
CMake adds `NEED_LBER_H` to `CURL_TEST_DEFINES`, which is used by
`curl_internal_test()`, but there is no LDAP-related test made with that
call.
Thus, stop detecting and publishing these from CMake.
Stefan Eissing [Thu, 19 Sep 2024 13:59:20 +0000 (15:59 +0200)]
tets: testrunner fairness
Collect all ready runners from select() and process in a loop. This
assures fairness in processing among all runners.
Formerly, only the first ready runner in the list of all was processed,
leading to later runners being delayed in processing and reporting
overly long test durations.
Also, reduce the backend idle timeout for the h2/h3 test servers so that
process shutdowns take less time.
Viktor Szakats [Sat, 7 Sep 2024 00:43:35 +0000 (02:43 +0200)]
build: `buildinfo.txt` improvements
- cmake: drop `configure.os`.
This also includes OS version, but thus far it's not important enough
to include it.
- autotools: drop redundant, autotools-only `{target|host}.vendor`.
(it's part of the triplet in `{target|host}`.)
- swap order to `*.cpu` -> `*.os` to match triplet-order.
- cmake: drop redundant `target`.
It's manually filled and only in a (so far) few CI jobs. Let's revisit
when this becomes useful.
- move `buildinfo.txt` to build root.
- dist: add `buildinfo.txt` to `DISTCLEANFILES`.
- autotools: detect human readable compiler version.
- autotools: replace `XXYY` `compiler.version` with "X.Y"-style.
(also to match cmake.)
- autotools: use distinct `compiler_id` for Apple clang: `APPLECLANG`.
To match cmake and also because the the "X.Y"-style version number
is the Apple version, while `XXYY` was a value roughly translated to
mainline llvm/clang version.
- show buildinfo at the end of the configure stage, when run in CI, or
when `CURL_BUILDINFO` or `CURL_CI` env is set.
- drop `tasklist` call before `taskkill`.
`taskkill` offers two ways to kill a `pid`:
1. `-pid <pid>`
If `<pid>` is missing it returns 128 and outputs:
```
ERROR: The process "<pid>" not found.
```
2. `-fi "PID eq <pid>"`
If `<pid>` is missing, it returns 0 and outputs:
```
INFO: No tasks running with the specified criteria.
```
The curl runner script doesn't check the result of the call and both
stdout and stderr are redirected to NUL.
Meaning the `tasklist` calls pre-verifying if the PID exists are not
necessary and we can drop them to put less strain on the runner
environment.
Stefan Eissing [Wed, 18 Sep 2024 14:32:07 +0000 (16:32 +0200)]
tests: testrunner reliability improvements
- perform torture tests with '-j2' for shorter runtime
- when waiting on test results overly long, log the tests
waited for and eventually log the test log directories
for easier analysis what is wrong in CI jobs.
- sockfilt.c: treat the windows errno 109 (ERROR_BROKEN_PIPE)
as a socket closed by the client and do not exit.
- when verifying https server, do not in addition check
the http server behind it also
- when tearing down the stunnel of a non-responsive https
server, tear down the http server with it
- NetBSD: enable impacket. Debatable if this hefty package is worth it
for the single SMB test (1451) run curl uses it for.
(on the upside, it seems to install instantly on NetBSD)
It's also a versioned package name that requires active maintenance.
The unversioned name doesn't work, though that's what the package
search returns: https://pkgsrc.se/net/py-impacket
- FreeBSD: stop installing impacket deps explicitly.
They are automatically installed via the impacket package.
- Linux: put time limit on test runs.
Started hanging yesterday:
linux: https://github.com/curl/curl/actions/runs/10875011573/job/30172946986#step:41:3731
awslc: https://github.com/curl/curl/actions/runs/10879782864/job/30185188509
wolfssl: https://github.com/curl/curl/actions/runs/10879782864/job/30185188509
- distcheck: reduce time limit to 15m (was 30m)
Linux tests started to hang, so reduce the timeout from 30 minutes
to 15. These jobs normally take 3 to 6 minutes to finish.
https://github.com/curl/curl/actions/runs/10879479914
Viktor Szakats [Wed, 11 Sep 2024 10:41:57 +0000 (12:41 +0200)]
CI: move Azure jobs to GHA, fix fallouts, sshserver, runtests tweaks
CI:
- GHA/windows: enable OpenSSH server, SysInternals `handle`, `impacket`.
Skip `impacket` on MSYS2 due to install failure.
Skip OpenSSH server for old/standalone mingw-w64 (building curl
without SSH there.)
- GHA/windows: make test tool installs a separate step.
- GHA/cygwin: enable OpenSSH server.
Skip `impacket`: it's compiling for 7 minutes then breaks.
Skip `stunnel` due to sluggish test run performance.
(This update is unrelated to Azure jobs.)
- GHA/linux: migrate Linux jobs from Azure CI.
- GHA/linux: migrate scanbuild job from Azure CI.
- GHA/linux: enable libssh2 in a job. Also enable valgrind.
- CI/windows: enable SSPI in two jobs.
- CI/windows: disable zlib in one more job.
- CI/windows: improve `if` condition checking GnuTLS.
- CI/windows: ignore SFTP/SCP tests as necessary.
- universally ignore SCP tests, they fail everywhere.
- ignore test 612.
- ignore test 613 616 618 with MSYS2 mingw-w64.
- ignore test 614 with libssh.
- ignore all SFTP with MSYS2 native.
- ignore all SFTP with vcpkg with `libssh2[core,zlib]`.
- ignore a couple of SFTP tests with MSYS2 mingw-w64.
(This matches settings on Azure CI.)
- GHA/windows: ignore failing 1451 'Basic SMB request' test for
old mingw-w64 7.3.0 (but not for 9.5.0!):
```
2024-09-11 21:45:59,738 ERROR smbComNegotiate: b'NT LM 0.12\x00' is not in list
[...]
curl: (7) Could not connect to server
[...]
FAIL 1451: 'Basic SMB request' SMB
```
Ref: https://github.com/curl/curl/actions/runs/10816280746/job/30007130770#step:13:3546
- CI/Azure: delete, now moved to GHA.
sshserver:
- sshserver: fix permissions for SSH host key on Windows, allowing sshd
to launch.
- sshserver: fix initializing config paths with Cygwin/MSYS2 sshd.
- sshserver: fix initializing config paths in parallel builds.
- sshserver: delete redundant `DenyUsers`. This also opens the way
to allow multiple usernames.
- sshserver: fix `AllowUsers` for GHA/windows, by allowing the
domainless username again.
Follow-up to 3ee7c676ec8ef32e3a9cb4d0fb2929a8335d58db #5721
- sshserver: fix `Deprecated option` warnings in `sshd_config`:
```
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 7: Deprecated option AuthorizedKeysFile2
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 25: Deprecated option KeyRegenerationInterval
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 35: Deprecated option RhostsRSAAuthentication
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 36: Deprecated option RSAAuthentication
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 37: Deprecated option ServerKeyBits
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 41: Deprecated option UseLogin
```
- sshserver: fix `Deprecated option` warnings in `ssh_config` with
Cygwin/MSYS2.
- sshserver: fix dumping config files due to the filenames missing their
full paths.
- sshserver: add workaround to make `logmsg` messages visible.
Before this patch they only went to a file and never shown.
runtests:
- runtests: log details when these Windows commands are called:
`handle`, `taskkill`, `tasklist`.
- runtests: add documentation links to Windows tools:
`handle`, `taskkill`, `tasklist`, `icacls`.
- runtests: add `-t` (kill whole tree) option to `taskkill` in
`servers.pm`, syncing it with the other `taskkill` call.
Follow-up to bc72a78a11764558639131d51fa8e7c81ee1e113 #14488
- runtests: show warning if Sysinternals `handle` tool is missing.
- runtests: drop Windows XP Home compatibility `tskill` call.
The call was made on all Windows versions. It's possibly overkill to
do this, because XP Home is probably rarely used for running curl
tests these days. In case it's needed, it'd be better to put it under
an explicit option.
- runtests: show Perl version and path.
Fix/silence fallouts:
- unit2603: fix building with disabled HTTP support.
- unit2604: silence `-Woverlength-strings` warnings in C89 mode.
- test437, test1614: fix to pass with no-IPv6 builds.
Viktor Szakats [Sat, 14 Sep 2024 20:29:13 +0000 (22:29 +0200)]
GHA/macos: replace make with ninja for cmake builds
Ninja performs better than GNU Make. Also the generation step is a bit
faster. The cost is installing ninja where missing from the image, and
there is no pre-existing package install step. Otherwise the
installation is fast because the tool is tiny.
(It would probably make sense to include ninja on the runner image; it's
a near zero-cost optimization opportunity for all CI jobs using cmake.)
Viktor Szakats [Sat, 14 Sep 2024 20:03:54 +0000 (22:03 +0200)]
GHA/macos: tidy-ups, install impacket for cmake jobs
- install impacket for cmake jobs.
- stop installing autotools for cmake jobs.
(also saving the extra seconds necessary for impacket install.)
- delete no longer necessary openssl pkg-config patch.
- `brew unlink openssl` unconditionally. This is safe, we no longer need
it at the default path, and the condition were never met anyway for
a while now.
- stop installing openssl, it's there on the base image.
(it was also missed for cmake, but that's okay)
Since ASCII transfers on FTP means sending CRLF line endings, we should
still keep converting them to LF-only on platforms where text files
typically do not use CRLF.
This also DOES NOT convert existing CRLF line endings on ASCII uploads
but only does stand-alone LF => CRLF.
Stefan Eissing [Fri, 13 Sep 2024 11:12:50 +0000 (13:12 +0200)]
connect: store connection info when really done
Output the 'Connected to...' info message when the connection has been
fully established and all information is available.
Due to our happy eyeballing, we should not emit info messages in
filters, because they may be part of an eyeballing attempt and may be
discarded later for another chain.
Stefan Eissing [Fri, 13 Sep 2024 16:36:53 +0000 (18:36 +0200)]
transfer: fix sendrecv() without interim poll
There was a "clever" optimization that skipped sendrecv() handling when
the transfer's pollset was empty. This happens for paused transfers, for
example.
Unforunately, if the libcurl application never calls curl_multi_poll(),
the pollset is and will aways remain empty, prevent the transfer from
progressing.
Remove this "optimization" and always try send/receive where applicable.
Fixes #14898 Reported-by: Victor Kislov
Closes #14901
Daniel McCarney [Thu, 12 Sep 2024 16:38:51 +0000 (12:38 -0400)]
vtls/rustls: support strong CSRNG data
Now that the curl rustls vtls backend is using rustls 0.14 we can
address the weak random situation by using
`rustls_default_crypto_provider_random()` to provide a `Curl_ssl`
`random` callback that fills the provided buffer with cryptographically
secure random data.
The mentions in `docs/` about weak RNG when using rustls are removed as
they are no longer applicable.
Daniel McCarney [Thu, 12 Sep 2024 16:31:59 +0000 (12:31 -0400)]
vtls/rustls: simplify ciphersuite skipping
Now that the rustls vtls backend is using rustls 0.14 we can take
advantage of `rustls_supported_ciphersuite_protocol_version()` to skip
TLS 1.3 and TLS 1.2 ciphersuites as required without needing to
interrogate the ciphersuite names as `rustls_str`s.
Daniel McCarney [Thu, 12 Sep 2024 16:18:26 +0000 (12:18 -0400)]
vtls/rustls: rustls-ffi 0.14.0 update
* Documentation is updated to describe new required version, and to link
to the upstream README about cryptography providers.
* GitHub workflow is updated to fetch 0.14.0.
* Breaking changes in`lib/vtls/rustls.c` are addressed:
* The `rustls_client_config_builder_build()` function now uses an out
parameter for the built config instead of returning it directly.
This allows the building process to fail if the default crypto
provider state isn't appropriate, or another error condition occurs.
* Default ciphersuites are collected using renamed functions named to
make it clear the ciphersuites are associated with the default
crypto provider.
* Customization of ciphersuites is now done via
a `rustls_crypto_provider_builder` used to instantiate a
`rustls_crypto_provider`. The customized provider can then can be
used with `rustls_client_config_builder_new_custom` in place of
providing ciphersuites directly.
* `rustls_connection_get_negotiated_ciphersuite()` now returns the
ciphersuite ID directly.
Daniel McCarney [Tue, 6 Aug 2024 17:49:03 +0000 (13:49 -0400)]
vtls/rustls: differentiate error messages
It's easier to diagnose a problem when there is one place where the
error message can be emitted. For that reason this commit updates two
errors that were shared between other fallible operations to use unique
messages.
Daniel McCarney [Fri, 19 Jul 2024 16:23:18 +0000 (12:23 -0400)]
vtls/rustls: simplify builder cleanup
Don't build `config_builder` just to free the resulting config, free the
builder directly.
When `cr_init_backend` encounters an error condition setting up the
Rustls client configuration it must do something with the
`config_builder` that was constructed earlier to avoid a memory leak.
The previous implementation preferred to use a pattern of building the
builder (thus consuming it) and then freeing the built config (to avoid
a memory leak). However, the purpose/intent is clearer when we just free
the builder directly instead of building it and freeing the result.
Stefan Eissing [Mon, 9 Sep 2024 11:52:44 +0000 (13:52 +0200)]
tests: tweak lock file handling and timers
- when server are killed by a test case, do not wait for the server lock
file to go away. These tests are mostly about client timeouts and the
server will hang until killed.
- when killing a server successfully, check for a remaining lock file,
log its existence and remove it.
- lower the delay timings on SLOWDOWN by half
- add SLOWDOWNDATA server command to only slow down the FTP data bytes,
not the control ones.
- lower some timeout values
Viktor Szakats [Thu, 12 Sep 2024 13:08:05 +0000 (15:08 +0200)]
autotools: fix `--with-ca-embed` build rule
Add missing rule dependency on the user-specified CA bundle. This fixes
including it when using the curl distro tarball, and other cases.
Also:
- fix the internal name of the CA bundle to avoid nested quotes.
It broke broke the rule dependency for the make tool.
- exclude the generated (empty) `tool_ca_embed.c` file from the distro
tarball. Patch-by: Daniel Stenberg
Follow-up to 8a3740bc8e558b9a9d4a652b74cf27a0961d7010 #14059 Reported-by: rampageX on github
Fixes #14879
Closes #14882
Stefan Eissing [Thu, 12 Sep 2024 08:03:33 +0000 (10:03 +0200)]
http2: when uploading data from stdin, fix eos forwarding
When uploading data from stdin ('-T -'), and the EOS was only detected
on a 0-length read, the EOS was not forwarded to the filters. This led
HTTP/2 to hang on not forwarding this to the server.
Added test_07_14 to reproduce and verify.
Fixes #14870 Reported-by: nekopsykose on github
Closes #14877
Viktor Szakats [Wed, 11 Sep 2024 21:42:43 +0000 (23:42 +0200)]
cmake: ensure `CURL_USE_OPENSSL`/`USE_OPENSSL_QUIC` are set in sync
OpenSSL must be selected (manually or by default) and detected for
`USE_OPENSSL_QUIC=ON` to work. This was documented before this patch,
but with CMake it was possible to use `USE_OPENSSL_QUIC=ON` without
an explicit or default `CURL_USE_OPENSSL=ON`, leading to a confused
internal state.
Fix by checking the QUIC condition early, showing a warning if enabled
without OpenSSL, and ignoring QUIC in such case.
Also: Fix casing of OpenSSL in option description.
Stefan Eissing [Wed, 11 Sep 2024 10:10:10 +0000 (12:10 +0200)]
test537: cap the rlimit max this test runs
Modern debian reports a soft limit of 134217724 which makes this test
run for minutes to allocate all file descriptors. Impose a cap of 256k,
so we do not run this tests on systems with larger rlimits.
Stefan Eissing [Wed, 11 Sep 2024 11:53:44 +0000 (13:53 +0200)]
QUIC: on connect, keep on trying on draining server
Do not give up connect on servers that are in draining state. This might
indicate the QUIC server restarting and the UDP packet routing still
hitting the instance shutting down.
Instead keep on connecting until the overall TIMEOUT fires.