(They are compatible with `--enable-unity` and `-DCMAKE_UNITY_BUILD=ON`
options, for further speed-up.)
Makes libtests and unit tests build _fast_, needing little disk space
even in static mode. Similar to CMake unity mode, but with a custom
script, also supporting autotools builds.
The price is having to deal with symbols/macros colliding between
`lib*.c` and `unit*.c` sources. Maybe with naming conventions or other
solutions this can be improved gradually and reduce the need for manual
intervention by `mk-bundle.mk`. I've included a script that does the bulk
of detecting name collisions.
Also:
- CI: enable test bundles.
- CI: build tests in more jobs.
- lib2305: fix FILE handle leak.
- unit1661: fix memleak found by torture test by releasing the `bufref`
structure in `unit_stop()` that was allocated in `unit_setup()`.
```
test 1661...[bufref unit tests]
Leak detected: memory still allocated: 13 bytes
allocated by /home/runner/work/curl/curl/tests/unit/unit1661.c:70
1661: torture FAILED: function number 1 in test.
```
Ref: https://github.com/curl/curl/actions/runs/10967279334/job/30456745290?pr=14772#step:8:41
Similar test suite builds with autotools default and cmake+bundle+unity:
- GHA/Linux: 33s vs 7s
https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
- GHA/macOS 34s vs 2s
https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
- GHA/FreeBSD: 15m25 vs 6m21 (full workflow time, ~qemu)
https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
- GHA/Cygwin: 9m52 vs 32s
https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 3m52 vs 14s
https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 5m45 vs 30s
https://github.com/curl/curl/actions/runs/10705668808/job/29681628787
Viktor Szakats [Fri, 6 Sep 2024 22:32:01 +0000 (00:32 +0200)]
build: tidy up and improve versioned-symbols options
- autotools: add support for custom prefix.
Usage: `--enable-versioned-symbols=MYPREFIX_`
Catching up with cmake.
- add default versioned prefix for Rustls builds.
- delete `HIDDEN` entry from version script `lib/libcurl.vers.in`.
It referred to symbol names that never existed in libcurl.
- cmake: use `lib/libcurl.vers.in` as a template instead of generating
it from scratch. Adapt `./configure` accordingly, and add comments
to keep hard-wired soname in sync with `lib/Makefile.soname`.
- autotools: delete Schannel and Secure Transport version prefixes.
Windows and macOS don't support the versioned symbols option.
Viktor Szakats [Thu, 29 Aug 2024 15:05:43 +0000 (17:05 +0200)]
configure: catch Apple in more target triplets
Before this patch, only these triplets were considered Apple:
`<cpu>-apple-darwin`
After this patch, these are also considered Apple:
`<cpu>-apple-(ios*|tvos*|visionos*|watchos*|<ETC>)`
`$host_os` (the last third of the triplet) still has a valid use
to differentiate between OS flavours, though for now this isn't
used, aligning with CMake.
Add CI checker to compare `libcurl.pc` and `curl-config` files
generated by autotools and cmake builds.
Fix differences and apply tiny cleanups:
- curl-config: use single-quotes for literals.
- curl-config: quote all variables.
- curl-config: replace double with single quotes in a substituted value
that's always literal (`@prefix@`).
- libcurl.pc: spelling in `Description:`.
- libcurl.pc: avoid substitution in a comment.
- cmake: fill `libdir` with `${exec_prefix}` instead of a literal.
To sync with './configure'.
- configure: fix `CURL_CA_BUNDLE` value to not generate nested quotes
in `curl-config`.
- configure: add missing `LDFLAGS` to `Libs.private` in `libcurl.pc`.
To sync with CMake.
- cmake: skip adding `CMAKE_C_IMPLICIT_LINK_LIBRARIES` for MINGW and
UNIX. They added these values as seen in CI:
MINGW: `-lmingw32 -lgcc -lmoldname -lmingwex -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc -lmoldname -lmingwex`
Linux: `-lgcc -lgcc_s -lc -lgcc -lgcc_s`
- cmake: delete FIXME about enabling libssh2 by default.
`./configure` has special defaults for these packages (called: "off"):
brotli, zstd, libpsl, libssh2, libssh, wolfssl, librtmp
It looks for them, but only at system locations, which makes them
never detected e.g. on macOS. CMake doesn't offer such default mode
for now.
- GHA/macos: drop now redundant `-DCURL_DISABLE_LDAPS=ON`.
- cmake: use `CMAKE_INSTALL_INCLUDEDIR` and `CMAKE_INSTALL_LIBDIR`
instead of hardcoded `include`/`lib` when generating `libcurl.pc`.
Updates to the GHA workflow:
- move autotools out-of-tree and rename cmake out-of-tree directory
to `bld_cm` to tell it's cmake.
- disable static libcurl for `./configure` to match cmake.
- enable `pkg-config` debug output with `./configure`.
- dump list of Homebrew packages on macOS.
- dump `./configure` detailed logs.
- disable zstd and brotli for Linux, to match cmake.
There remain differences, mostly due to detection order and method. Also
some values are inherently different when using CMake and autotools,
such as `--cc`, `--configure`. autotools also generates duplicates for
`-lssl` and `-lcrypto`. macOS LDAP wants to link `-lber` while autotools
doesn't. Some build defaults are also different in autotools and cmake.
These differences are smoothened out for now by the checker script, or
via build options. Notice that lib order (a dupes) _can_ be significant
in some cases. E.g. the binutils linker is infamous for that on Windows.
Viktor Szakats [Thu, 12 Sep 2024 13:16:51 +0000 (15:16 +0200)]
cmake: fix broken dependency chain for cmdline-opts, tidy-ups
- make `curl.1` and `curl.txt` depend on `DPAGES`.
To trigger a rebuild when an individual manpage is updated.
- tell CMake that the cmdline-opts command also creates `curl.txt`.
- make `tool_hugehelp.c` depend on `curl.txt` (was: `curl.1`), to match
what it actually uses for input.
- stop using `generate-curl.1` as an indirect way to create `curl.txt`
in time for `tool_hugehelp.c`. After the fixes above there is a direct
depedency chain between them.
- move `ASCIIPAGE` and `MANPAGE` variables to top-level, re-use them in
`src` and prefix them with `CURL_` to avoid clashing with other
projects.
- drop double quotes from `generate-curl.1` as a hint that it is not
a filename, but a target name.
Viktor Szakats [Thu, 5 Sep 2024 00:11:35 +0000 (02:11 +0200)]
build: tidy up deprecation suppression, enable warnings for clang
Suppress deprecation warnings the closest to the deprecated code, using
`CURL_IGNORE_DEPRECATION()`. Then drop build-specific suppressions, and
file-wide ones. The latter is not compatible with Unity mode. Also
replace manual suppressions with a macro to apply to all compilers with
deprecation warning support. Also enable deprecation warnings for clang.
- curl/curl.h: enable deprecation warnings for clang.
- docs/examples: stop setting `CURL_DISABLE_DEPRECATION` with autotools.
Suppression moved to C-level earlier. Syncs with cmake.
Follow-up to 5fc61a37c1b177cefbc11dc9f5eef7b2a14538da #14123
- tests/http/clients: stop setting `CURL_DISABLE_DEPRECATION` in
autotools. If it becomes necessary in the future, it can be done in
C via the macro. Syncs with cmake.
Viktor Szakats [Tue, 20 Aug 2024 09:13:19 +0000 (11:13 +0200)]
cmake: expand `CURL_USE_PKGCONFIG` to non-cross `MINGW`
Enable `CURL_USE_PKGCONFIG` by default for more environments:
- for `MINGW` targets when not using cross-compilation.
- stop restricting vcpkg to MSVC. (this currently unlocks mingw,
also unlocked by the update above.)
Viktor Szakats [Thu, 19 Sep 2024 20:38:04 +0000 (22:38 +0200)]
GHA: speed up builds in torture jobs, tidy up
- use cmake with ninja.
- drop unnecessary packages.
Also:
- Linux: switch to Ubuntu 20.24 runner. Initially for newer packages,
then left it there for variation and as canary.
- Linux: make it easier to enable valgrind by dropping `-n` tflags.
- Linux: show `curl -V` after build.
- Linux: sync job names with macOS.
- Linux, macOS: review and adjust job timeouts.
- Linux, macOS: sync job configs more:
Linux: enable libssh2, macOS: enable brotli, zstd, libssh2.
- macOS: prefer `source` (over manual `PATH` editing) for `venv` setup.
- macOS: drop redundant `-DENABLE_CURLDEBUG=ON`.
- macOS: drop redundant `-n` tflags. There is no valgrind on macOS.
- macOS: allow overriding test parallelism per job.
- macOS: unlock all disabled tests for torture jobs.
Viktor Szakats [Fri, 6 Sep 2024 08:26:06 +0000 (10:26 +0200)]
autotools: add support for 'unity' builds, enable in CI
Implement the "unity" builds as known from CMake, but for autotools.
It's limited to `lib` and `src` (CMake also supports it in `tests`).
Enable with: `--enable-unity` (disabled by default)
Unity builds speed up builds significantly. Cygwin and Windows builds in
particular, but the effect is noticeable on most systems. It also allows
discovering unity issues with autotools, benefitting also CMake when
building the same combination. In CI it makes turnaround times quicker.
This closes build performance with CMake. autotools still lags behind
because it builds shared and static libcurl in two, separate passes.
CMake does it in one. Manpage compilation isn't batched, it is in CMake.
After unity and test bundle support the slowest parts of the build are
the configuration phase (which is effectively a tedious, non-parallel,
compilation and/or linking of 300+ tiny programs. The next bottleneck
is compiling individual examples and finally test servers (only slow
with autotools).
The autotools implementation is slightly less efficient than CMake,
because 3 sources are permanently excluded while in CMake this isn't
necessary and solved more efficiently while building libtests. There is
also no 'unity' support for tests, making them a less efficient also.
Enable it in CI for most `configure` jobs. Except in GHA/dist (though
it works fine there too), to use the default config there. Also skip for
the Linux AWC-LC job where it made builds time a few seconds longer
(reason undiscovered.)
Viktor Szakats [Sat, 14 Sep 2024 00:21:33 +0000 (02:21 +0200)]
cmake: separate target for examples, optimize CI, fix fallouts
- Move `docs/examples` builds under a separate target.
- Make `BUILD_EXAMPLES` default to `ON`. It means to generate the rules
for `docs/examples` by default, but not build them. To build them,
an explicit `make curl-examples` (or ninja, etc) command is necessary.
This syncs behaviour with autotools, and also how both cmake and
autotools are building tests.
- GHA: update cmake jobs to use the new way of building examples.
- GHA: move examples build step at the end of the job, after building
and running tests. This allows to have build and test run results
faster, and leave the seldom-changing examples build to the end.
Building examples is the slowest build step with no practical way to
make them fast.
- appveyor: enable building examples in two old-MSVC jobs.
- examples: fix examples to build cleanly with old MSVC versions.
- GHA/non-native: move example build log under a GHA foldable section.
- GHA/windows: move building examples into separate step for Linux cross
jobs.
Stefan Eissing [Tue, 10 Sep 2024 12:08:17 +0000 (14:08 +0200)]
request: on shutdown send, proceed normally on timeout
When ending an FTP upload, we shut down the connection gracefully, since
the server should be notified we had send all bytes. Mostly, this is a
NOP without TLS involved. With TLS, close-notify messages should be
exchanged.
As reported in #14843, not all servers seem to do that. Since it is the
server's responsiblity to check it has received everything, we just log
the timeout and proceed as if everything is fine.
In the receive direction, we still fail the transfer if the server does
not shut down its direction properly.
Stefan Eissing [Thu, 19 Sep 2024 10:23:55 +0000 (12:23 +0200)]
alt-svc: honor data->state.httpwant
When a transfer is set for a speficif HTTP version, only lookup that
protocol in the alt-svc mappings. When no speicific version is
requested, scan all entries as before.
Stefan Eissing [Thu, 19 Sep 2024 09:47:29 +0000 (11:47 +0200)]
url: connection reuse on h3 connections
- When searching for existing connections, interpret the
default CURL_HTTP_VERSION_2TLS as "anything goes". This
will allow us to reuse HTTP/3 connections better
- add 'http/1.1' as allowed protocol identifier in Alt-Svc
files
- add test_02_0[345] for testing protocol selection on
provided alt-svc files
Stefan Eissing [Fri, 20 Sep 2024 14:06:25 +0000 (16:06 +0200)]
libssh.c: handle EGAINS during proto-connect correctly
The implementation did not check the poll flags when a ssh_connect()
EAGAINed. Also the poll check did not allow for both directions
to be signalled by libssh, which it does.
This appeared during CI testing with higher paralellism.
Stefan Eissing [Fri, 20 Sep 2024 13:58:21 +0000 (15:58 +0200)]
multi.c: make stronger check for paused transfer before asserting
With higher parallelism in CI, the ASSERT triggered on pause tests.
Strengthen the check. We might want to think about removing
KEEP_RECV_PAUSE|KEEP_SEND_PAUSE altogether.
Viktor Szakats [Thu, 19 Sep 2024 22:51:35 +0000 (00:51 +0200)]
GHA: keep default pkgconf, do not replace with pkg-config on Linux
Ubuntu has the `pkgconf` package installed by default that implements
the `pkg-config` command. Switch CI `apt` commands over to `pkgconf`
to avoid replacing it in every job run.
Avoids:
```
The following packages will be REMOVED:
pkgconf r-base-dev
The following NEW packages will be installed:
[...] pkg-config [...]
```
https://github.com/curl/curl/actions/runs/10949915766/job/30404126342?pr=14972#step:2:20
Viktor Szakats [Wed, 4 Sep 2024 01:25:09 +0000 (03:25 +0200)]
unit1660: fix unreachable code warning in no-SSL builds
Or no-HTTP, no-HSTS builds, also MSH3 builds.
CMake, VS2022, Debug, x64, no SSL, Static, Build-only:
```
C:\projects\curl\tests\unit\unit1660.c(46,1): error C2220: the following warning is treated as an error
C:\projects\curl\tests\unit\unit1660.c(46,1): warning C4702: unreachable code
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/50531210/job/cjewvo9agavthr3o#L216
Required-by: #14922
Cherry-picked from #14772
Closes #14971
Viktor Szakats [Mon, 2 Sep 2024 22:26:26 +0000 (00:26 +0200)]
cmake: allow building tests in unity mode
Makes building tests noticeably faster.
Apply changes/fixes/workarounds to make Unity work:
- rename test variables to avoid collisions or shadowing each other when
combined into single units.
- add workaround to avoid applying `lib/memdebug.h` overrides to system
headers declaring/defining `getaddrinfo()`/`freeaddrinfo()` for
`tests/server/resolve.c`. This replaces a previous workaround that
worked for that specific source.
- rename test macro `CTRL` clashing with Cygwin `sys/ioctl.h`.
- add include guard to `test.h`.
Also:
- exclude `tests/http/clients` which are all single-source. (like
`docs/examples`.)
Viktor Szakats [Fri, 6 Sep 2024 12:09:40 +0000 (14:09 +0200)]
lib: fix unity builds with BearSSL, MSH3, Quiche, OmniOS
- fix MSH3 static symbol clash.
- fix Quiche static symbol clash.
- fix local macro clash with BearSSL header.
- fix local macro clash with OmniOS system header.
```
In file included from ../../lib/urldata.h:197,
from ../../lib/altsvc.c:32,
from libcurlall.c:2:
../../lib/cf-socket.h:55:25: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token
55 | #define sa_addr _sa_ex_u.addr
| ^
In file included from ../../lib/urldata.h:197,
from ../../lib/altsvc.c:32,
from libcurlall.c:2:
../../lib/cf-socket.h:55:25: error: expected ':', ',', ';', '}' or '__attribute__' before '.' token
55 | #define sa_addr _sa_ex_u.addr
| ^
```
Ref: https://github.com/curl/curl/actions/runs/10738314933/job/29781644299?pr=14772#step:3:6115
Discovered while adding support for "unity" builds for autotools.
Required-by: #14922
Cherry-picked from #14815
Closes #14932
Dan Fandrich [Tue, 17 Sep 2024 22:14:31 +0000 (15:14 -0700)]
tests: Only log warnings or worse by default in smbserver
There shouldn't be anything displayed during a normal run, but only if
server debugging is enabled. Also, set log_file to a magic value to
disable it, otherwise impacket installs its own logger that messes with
what we want.
Viktor Szakats [Mon, 2 Sep 2024 21:34:13 +0000 (23:34 +0200)]
cmake, `Makefile.mk`: use `-isystem` for dep headers, silence BearSSL issues
Patch started out for working around compiler warnings in BearSSL latest
tarball release v0.6 (2018-08-14) and Apple clang 14 with CMake.
Then turned into patching CMake and `Makefile.mk` builds to use
`-isystem` instead `-I` when adding header directories for
dependencies. This avoids compiler warnings in dependency headers,
syncing behaviour with autotools.
Also:
- `Makefile.mk`: add support for BearSSL.
- delete warning suppression for mbedTLS headers. No longer necessary
after this patch.
Follow-up to 434db995a7566a76f6a16391897816ddd1011adc #12720
Silenced BearSSL warnings:
```
In file included from curl/lib/vtls/bearssl.c:28:
In file included from bearssl/inc/bearssl.h:127:
bearssl/inc/bearssl_hash.h:727:5: warning: 'BR_DOXYGEN_IGNORE' is not defined, evaluates to 0 [-Wundef]
^
bearssl/inc/bearssl_hash.h:745:5: warning: 'BR_DOXYGEN_IGNORE' is not defined, evaluates to 0 [-Wundef]
^
In file included from curl/lib/vtls/bearssl.c:28:
In file included from bearssl/inc/bearssl.h:136:
bearssl/inc/bearssl_ssl.h:1253:20: warning: implicit conversion loses integer precision: 'unsigned int' to 'uint16_t' (aka 'unsigned short') [-Wimplicit-int-conversion]
cc->version_min = version_min;
~ ^~~~~~~~~~~
bearssl/inc/bearssl_ssl.h:1254:20: warning: implicit conversion loses integer precision: 'unsigned int' to 'uint16_t' (aka 'unsigned short') [-Wimplicit-int-conversion]
cc->version_max = version_max;
~ ^~~~~~~~~~~
bearssl/inc/bearssl_ssl.h:1327:28: warning: implicit conversion loses integer precision: 'size_t' (aka 'unsigned long') to 'uint16_t' (aka 'unsigned short') [-Wimplicit-int-conversion]
ctx->protocol_names_num = num;
~ ^~~
5 warnings generated.
```
Also these two cases, which are caused by an unidentified component
(outside curl) cranking up MSVC warnings in external headers to `/W4`
when ZLIB is deselected:
https://github.com/curl/curl/pull/14859#issuecomment-2351809153
mbedTLS 3.6.1:
```
C:\vcpkg\installed\x64-windows\include\psa\crypto_struct.h(254,13): error C2220: the following warning is treated as an error [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
C:\vcpkg\installed\x64-windows\include\psa\crypto_struct.h(254,13): warning C4200: nonstandard extension used: zero-sized array in struct/union [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
```
Ref: https://github.com/curl/curl/actions/runs/10842694205/job/30107466989?pr=14859#step:10:29
nghttp3 1.5.0:
```
C:\vcpkg\installed\x64-windows\include\nghttp3\nghttp3.h(2678,1): error C2220: the following warning is treated as an error [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
C:\vcpkg\installed\x64-windows\include\nghttp3\nghttp3.h(2678,1): warning C4324: 'nghttp3_pri': structure was padded due to alignment specifier [D:\a\curl\curl\bld\lib\libcurl_object.vcxproj]
(compiling source file 'CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c')
```
Ref: https://github.com/curl/curl/actions/runs/10871875297/job/30166233862?pr=14859#step:10:28
Viktor Szakats [Mon, 26 Aug 2024 11:53:41 +0000 (13:53 +0200)]
cmake: delete unused `NEED_LBER_H`, `HAVE_LDAP_H`
Both are used by `./configure` internally, not by curl C code.
CMake adds `NEED_LBER_H` to `CURL_TEST_DEFINES`, which is used by
`curl_internal_test()`, but there is no LDAP-related test made with that
call.
Thus, stop detecting and publishing these from CMake.
Stefan Eissing [Thu, 19 Sep 2024 13:59:20 +0000 (15:59 +0200)]
tets: testrunner fairness
Collect all ready runners from select() and process in a loop. This
assures fairness in processing among all runners.
Formerly, only the first ready runner in the list of all was processed,
leading to later runners being delayed in processing and reporting
overly long test durations.
Also, reduce the backend idle timeout for the h2/h3 test servers so that
process shutdowns take less time.
Viktor Szakats [Sat, 7 Sep 2024 00:43:35 +0000 (02:43 +0200)]
build: `buildinfo.txt` improvements
- cmake: drop `configure.os`.
This also includes OS version, but thus far it's not important enough
to include it.
- autotools: drop redundant, autotools-only `{target|host}.vendor`.
(it's part of the triplet in `{target|host}`.)
- swap order to `*.cpu` -> `*.os` to match triplet-order.
- cmake: drop redundant `target`.
It's manually filled and only in a (so far) few CI jobs. Let's revisit
when this becomes useful.
- move `buildinfo.txt` to build root.
- dist: add `buildinfo.txt` to `DISTCLEANFILES`.
- autotools: detect human readable compiler version.
- autotools: replace `XXYY` `compiler.version` with "X.Y"-style.
(also to match cmake.)
- autotools: use distinct `compiler_id` for Apple clang: `APPLECLANG`.
To match cmake and also because the the "X.Y"-style version number
is the Apple version, while `XXYY` was a value roughly translated to
mainline llvm/clang version.
- show buildinfo at the end of the configure stage, when run in CI, or
when `CURL_BUILDINFO` or `CURL_CI` env is set.
- drop `tasklist` call before `taskkill`.
`taskkill` offers two ways to kill a `pid`:
1. `-pid <pid>`
If `<pid>` is missing it returns 128 and outputs:
```
ERROR: The process "<pid>" not found.
```
2. `-fi "PID eq <pid>"`
If `<pid>` is missing, it returns 0 and outputs:
```
INFO: No tasks running with the specified criteria.
```
The curl runner script doesn't check the result of the call and both
stdout and stderr are redirected to NUL.
Meaning the `tasklist` calls pre-verifying if the PID exists are not
necessary and we can drop them to put less strain on the runner
environment.
Stefan Eissing [Wed, 18 Sep 2024 14:32:07 +0000 (16:32 +0200)]
tests: testrunner reliability improvements
- perform torture tests with '-j2' for shorter runtime
- when waiting on test results overly long, log the tests
waited for and eventually log the test log directories
for easier analysis what is wrong in CI jobs.
- sockfilt.c: treat the windows errno 109 (ERROR_BROKEN_PIPE)
as a socket closed by the client and do not exit.
- when verifying https server, do not in addition check
the http server behind it also
- when tearing down the stunnel of a non-responsive https
server, tear down the http server with it
- NetBSD: enable impacket. Debatable if this hefty package is worth it
for the single SMB test (1451) run curl uses it for.
(on the upside, it seems to install instantly on NetBSD)
It's also a versioned package name that requires active maintenance.
The unversioned name doesn't work, though that's what the package
search returns: https://pkgsrc.se/net/py-impacket
- FreeBSD: stop installing impacket deps explicitly.
They are automatically installed via the impacket package.
- Linux: put time limit on test runs.
Started hanging yesterday:
linux: https://github.com/curl/curl/actions/runs/10875011573/job/30172946986#step:41:3731
awslc: https://github.com/curl/curl/actions/runs/10879782864/job/30185188509
wolfssl: https://github.com/curl/curl/actions/runs/10879782864/job/30185188509
- distcheck: reduce time limit to 15m (was 30m)
Linux tests started to hang, so reduce the timeout from 30 minutes
to 15. These jobs normally take 3 to 6 minutes to finish.
https://github.com/curl/curl/actions/runs/10879479914
Viktor Szakats [Wed, 11 Sep 2024 10:41:57 +0000 (12:41 +0200)]
CI: move Azure jobs to GHA, fix fallouts, sshserver, runtests tweaks
CI:
- GHA/windows: enable OpenSSH server, SysInternals `handle`, `impacket`.
Skip `impacket` on MSYS2 due to install failure.
Skip OpenSSH server for old/standalone mingw-w64 (building curl
without SSH there.)
- GHA/windows: make test tool installs a separate step.
- GHA/cygwin: enable OpenSSH server.
Skip `impacket`: it's compiling for 7 minutes then breaks.
Skip `stunnel` due to sluggish test run performance.
(This update is unrelated to Azure jobs.)
- GHA/linux: migrate Linux jobs from Azure CI.
- GHA/linux: migrate scanbuild job from Azure CI.
- GHA/linux: enable libssh2 in a job. Also enable valgrind.
- CI/windows: enable SSPI in two jobs.
- CI/windows: disable zlib in one more job.
- CI/windows: improve `if` condition checking GnuTLS.
- CI/windows: ignore SFTP/SCP tests as necessary.
- universally ignore SCP tests, they fail everywhere.
- ignore test 612.
- ignore test 613 616 618 with MSYS2 mingw-w64.
- ignore test 614 with libssh.
- ignore all SFTP with MSYS2 native.
- ignore all SFTP with vcpkg with `libssh2[core,zlib]`.
- ignore a couple of SFTP tests with MSYS2 mingw-w64.
(This matches settings on Azure CI.)
- GHA/windows: ignore failing 1451 'Basic SMB request' test for
old mingw-w64 7.3.0 (but not for 9.5.0!):
```
2024-09-11 21:45:59,738 ERROR smbComNegotiate: b'NT LM 0.12\x00' is not in list
[...]
curl: (7) Could not connect to server
[...]
FAIL 1451: 'Basic SMB request' SMB
```
Ref: https://github.com/curl/curl/actions/runs/10816280746/job/30007130770#step:13:3546
- CI/Azure: delete, now moved to GHA.
sshserver:
- sshserver: fix permissions for SSH host key on Windows, allowing sshd
to launch.
- sshserver: fix initializing config paths with Cygwin/MSYS2 sshd.
- sshserver: fix initializing config paths in parallel builds.
- sshserver: delete redundant `DenyUsers`. This also opens the way
to allow multiple usernames.
- sshserver: fix `AllowUsers` for GHA/windows, by allowing the
domainless username again.
Follow-up to 3ee7c676ec8ef32e3a9cb4d0fb2929a8335d58db #5721
- sshserver: fix `Deprecated option` warnings in `sshd_config`:
```
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 7: Deprecated option AuthorizedKeysFile2
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 25: Deprecated option KeyRegenerationInterval
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 35: Deprecated option RhostsRSAAuthentication
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 36: Deprecated option RSAAuthentication
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 37: Deprecated option ServerKeyBits
D:/a/curl/curl/bld/tests/log/2/server/curl_sshd_config line 41: Deprecated option UseLogin
```
- sshserver: fix `Deprecated option` warnings in `ssh_config` with
Cygwin/MSYS2.
- sshserver: fix dumping config files due to the filenames missing their
full paths.
- sshserver: add workaround to make `logmsg` messages visible.
Before this patch they only went to a file and never shown.
runtests:
- runtests: log details when these Windows commands are called:
`handle`, `taskkill`, `tasklist`.
- runtests: add documentation links to Windows tools:
`handle`, `taskkill`, `tasklist`, `icacls`.
- runtests: add `-t` (kill whole tree) option to `taskkill` in
`servers.pm`, syncing it with the other `taskkill` call.
Follow-up to bc72a78a11764558639131d51fa8e7c81ee1e113 #14488
- runtests: show warning if Sysinternals `handle` tool is missing.
- runtests: drop Windows XP Home compatibility `tskill` call.
The call was made on all Windows versions. It's possibly overkill to
do this, because XP Home is probably rarely used for running curl
tests these days. In case it's needed, it'd be better to put it under
an explicit option.
- runtests: show Perl version and path.
Fix/silence fallouts:
- unit2603: fix building with disabled HTTP support.
- unit2604: silence `-Woverlength-strings` warnings in C89 mode.
- test437, test1614: fix to pass with no-IPv6 builds.
Viktor Szakats [Sat, 14 Sep 2024 20:29:13 +0000 (22:29 +0200)]
GHA/macos: replace make with ninja for cmake builds
Ninja performs better than GNU Make. Also the generation step is a bit
faster. The cost is installing ninja where missing from the image, and
there is no pre-existing package install step. Otherwise the
installation is fast because the tool is tiny.
(It would probably make sense to include ninja on the runner image; it's
a near zero-cost optimization opportunity for all CI jobs using cmake.)
Viktor Szakats [Sat, 14 Sep 2024 20:03:54 +0000 (22:03 +0200)]
GHA/macos: tidy-ups, install impacket for cmake jobs
- install impacket for cmake jobs.
- stop installing autotools for cmake jobs.
(also saving the extra seconds necessary for impacket install.)
- delete no longer necessary openssl pkg-config patch.
- `brew unlink openssl` unconditionally. This is safe, we no longer need
it at the default path, and the condition were never met anyway for
a while now.
- stop installing openssl, it's there on the base image.
(it was also missed for cmake, but that's okay)
Since ASCII transfers on FTP means sending CRLF line endings, we should
still keep converting them to LF-only on platforms where text files
typically do not use CRLF.
This also DOES NOT convert existing CRLF line endings on ASCII uploads
but only does stand-alone LF => CRLF.
Stefan Eissing [Fri, 13 Sep 2024 11:12:50 +0000 (13:12 +0200)]
connect: store connection info when really done
Output the 'Connected to...' info message when the connection has been
fully established and all information is available.
Due to our happy eyeballing, we should not emit info messages in
filters, because they may be part of an eyeballing attempt and may be
discarded later for another chain.
Stefan Eissing [Fri, 13 Sep 2024 16:36:53 +0000 (18:36 +0200)]
transfer: fix sendrecv() without interim poll
There was a "clever" optimization that skipped sendrecv() handling when
the transfer's pollset was empty. This happens for paused transfers, for
example.
Unforunately, if the libcurl application never calls curl_multi_poll(),
the pollset is and will aways remain empty, prevent the transfer from
progressing.
Remove this "optimization" and always try send/receive where applicable.
Fixes #14898 Reported-by: Victor Kislov
Closes #14901
Daniel McCarney [Thu, 12 Sep 2024 16:38:51 +0000 (12:38 -0400)]
vtls/rustls: support strong CSRNG data
Now that the curl rustls vtls backend is using rustls 0.14 we can
address the weak random situation by using
`rustls_default_crypto_provider_random()` to provide a `Curl_ssl`
`random` callback that fills the provided buffer with cryptographically
secure random data.
The mentions in `docs/` about weak RNG when using rustls are removed as
they are no longer applicable.
Daniel McCarney [Thu, 12 Sep 2024 16:31:59 +0000 (12:31 -0400)]
vtls/rustls: simplify ciphersuite skipping
Now that the rustls vtls backend is using rustls 0.14 we can take
advantage of `rustls_supported_ciphersuite_protocol_version()` to skip
TLS 1.3 and TLS 1.2 ciphersuites as required without needing to
interrogate the ciphersuite names as `rustls_str`s.
Daniel McCarney [Thu, 12 Sep 2024 16:18:26 +0000 (12:18 -0400)]
vtls/rustls: rustls-ffi 0.14.0 update
* Documentation is updated to describe new required version, and to link
to the upstream README about cryptography providers.
* GitHub workflow is updated to fetch 0.14.0.
* Breaking changes in`lib/vtls/rustls.c` are addressed:
* The `rustls_client_config_builder_build()` function now uses an out
parameter for the built config instead of returning it directly.
This allows the building process to fail if the default crypto
provider state isn't appropriate, or another error condition occurs.
* Default ciphersuites are collected using renamed functions named to
make it clear the ciphersuites are associated with the default
crypto provider.
* Customization of ciphersuites is now done via
a `rustls_crypto_provider_builder` used to instantiate a
`rustls_crypto_provider`. The customized provider can then can be
used with `rustls_client_config_builder_new_custom` in place of
providing ciphersuites directly.
* `rustls_connection_get_negotiated_ciphersuite()` now returns the
ciphersuite ID directly.
Daniel McCarney [Tue, 6 Aug 2024 17:49:03 +0000 (13:49 -0400)]
vtls/rustls: differentiate error messages
It's easier to diagnose a problem when there is one place where the
error message can be emitted. For that reason this commit updates two
errors that were shared between other fallible operations to use unique
messages.
Daniel McCarney [Fri, 19 Jul 2024 16:23:18 +0000 (12:23 -0400)]
vtls/rustls: simplify builder cleanup
Don't build `config_builder` just to free the resulting config, free the
builder directly.
When `cr_init_backend` encounters an error condition setting up the
Rustls client configuration it must do something with the
`config_builder` that was constructed earlier to avoid a memory leak.
The previous implementation preferred to use a pattern of building the
builder (thus consuming it) and then freeing the built config (to avoid
a memory leak). However, the purpose/intent is clearer when we just free
the builder directly instead of building it and freeing the result.
projects / thirdparty / curl.git / log
