Viktor Szakats [Tue, 22 Oct 2024 11:22:39 +0000 (13:22 +0200)]
cmake: avoid setting `BUILD_TESTING`
`BUILD_TESTING` variable is used by other projects and CMake internally.
Replace `cmake_dependent_option()` with `option()` and introduce an
internal variable to track if want and can do testing.
Viktor Szakats [Wed, 23 Oct 2024 13:40:18 +0000 (15:40 +0200)]
libssh2: delete duplicate `break`
```
lib/vssh/libssh2.c:2495:7: warning: 'break' will never be executed [-Wunreachable-code-break]
break;
^~~~~
```
CI did not catch it due to llvm skipping this check for all #included
files. It's designed this way to avoid performance issues and false
positive when checking headers:
https://github.com/llvm/llvm-project/issues/71046
Viktor Szakats [Wed, 23 Oct 2024 09:06:27 +0000 (11:06 +0200)]
GHA/windows: work around Git for Windows perf regression
Fix the significant perf regression for vcpkg jobs by switching to the
MSYS2 shell environment from Git for Windows. This env is already used
for old-mingw-w64 job that remained unaffected by this issue.
The issue began with the windows-runner update 20241015.1.0. It bumped
Git for Windows from Git 2.46.2.windows.1 to Git 2.47.0.windows.1. GfW
bumped its MSYS2 components, including `msys-2.0.dll`. That's Cygwin
code, which may have contributed to this. Pipes were involved and
`runtests.pl` relies on pipes heavily in parallel mode. (The issue was
not seen with parallel tests disabled, in retrospect.)
This is useful as a permanent solution too. It drop GfW as a dependency
and makes Windows jobs use one less shell/env flavour.
Long term it might help to use native Windows Perl to avoid the MSYS
layer completely, if there is a way to make that work.
Viktor Szakats [Wed, 23 Oct 2024 08:54:14 +0000 (10:54 +0200)]
GHA/linux: drop patch from openssl3 thread sanitizer
The patch is now part of the 3.4.0 stable release.
(Turns out it was part of 3.3.2 already.)
Also:
- rename this local build to match the scheme used with wolfssl.
- drop '3' from local openssl build name.
- sync job name with others.
- quote step names where missing.
Viktor Szakats [Tue, 17 Sep 2024 16:43:17 +0000 (18:43 +0200)]
runtests: use deterministic sort for `TESTINFO` lines
Sort TESTINFO lines by description within the number of skipped test.
It makes the list of skipped test groups easier to diff/compare between
jobs and runs.
Max Dymond [Tue, 22 Oct 2024 12:31:22 +0000 (13:31 +0100)]
ci: fix renovate's matching for OpenSSL and quictls
Renovate only matches on the raw version numbers of a package, but
OpenSSL includes `openssl-` as a prefix in the version number. This
change means that the match string now expects the `openssl-` prefix
and will just update the version portion.
This also updates quictls so that renovate can detect and update the
version correctly.
Viktor Szakats [Tue, 22 Oct 2024 15:02:14 +0000 (17:02 +0200)]
cmake/FindCares: fix version detection for c-ares 1.34.1
Due to a regression in c-ares 1.34.1, the non-pkg-config version
detection method broke for this version. c-ares 1.34.2 fixes it,
but update our detection code anyway to also work with 1.34.1.
Viktor Szakats [Tue, 22 Oct 2024 15:32:28 +0000 (17:32 +0200)]
cmake: use the `BSD` variable
- use `BSD` in addition to backwards-compatible method.
- add `BSD` to the configuration log and `buildinfo.txt` if detected.
- add `BSD` tag to `buildinfo.txt` also via `./configure`.
The `BSD` variable is supported by CMake 3.25.0 and upper.
Viktor Szakats [Tue, 22 Oct 2024 14:25:15 +0000 (16:25 +0200)]
GHA/windows: increase timeout for vcpkg jobs due to slowness
The openssl job no longer fits into 10 minutes since the 20241015.1.0
GHA windows-latest image update. This caused all runs to fail.
The `run tests` step takes ~10 minutes now, up from ~4. This is
6 minutes more than before these updates. It's seen with other vcpkg
jobs too, tests run slower than half speed since.
Bump the timeout to make it, though the headroom is now less than it
was.
Stefan Eissing [Tue, 22 Oct 2024 08:39:44 +0000 (10:39 +0200)]
CI: run with standard mod_http2
We used to include a special mod_h2 in our CI that supports the
directive H2MaxDataFrameLen for test_02_20. Since then, ubuntu-lastest
includes a more recent apache httpd. Let's see if we can live without
the special.
Viktor Szakats [Tue, 22 Oct 2024 11:51:32 +0000 (13:51 +0200)]
GHA/windows: ignore results for test 987
987 is `SMTPS with redundant explicit SSL request`.
Root cause undiscovered.
Started failing after GHA bumping the windows image to `20241015.1.0`:
https://github.com/actions/runner-images/commit/fcc4cdb1d095af1317859c4809364538953b3497
https://github.com/actions/runner-images/pull/10803
vcpkg packages also got bumped as a result. They seem unrelated:
c-ares 1.33.1 -> 1.34.1
nghttp2 1.62.1 -> 1.63.0
nghttp3 1.5.0 -> 1.6.0
(there may be more)
Viktor Szakats [Mon, 21 Oct 2024 17:06:50 +0000 (19:06 +0200)]
GHA/linux: tidy up and performance
- replace openssl3 default local build with packaged one.
- drop valgrind from IntelC job.
- drop IntelC no-ssl job.
- bump local openssl to 3.3.2.
- disable tests in the cmake variant of a job.
- add comment to the remaining local openssl3 build.
We can drop the patch after upgrading to upcoming 3.4.0.
- drop gcc-11 from jobs. packaged gcc is now newer at 13.2.0.
(saves more than 1m install time for each of the 5 jobs.)
Follow-up to 9cc9a6472c5d2e13a117ca02f432443db2d1be57 #9454
zjyhjqs [Wed, 16 Oct 2024 16:17:38 +0000 (00:17 +0800)]
cmake: fix compile warnings for clang-cl
clang-cl is an alternative command-line interface to Clang, designed
for compatibility with the Visual C++ compiler, `cl.exe`:
https://clang.llvm.org/docs/UsersManual.html#clang-cl
The way to test clang-cl in CMake:
- `CMAKE_<LANGUAGE>_COMPILER_ID`: "Clang"
- `CMAKE_<LANGUAGE>_COMPILER_FRONTEND_VARIANT`: "MSVC"
Note: `CMAKE_<LANGUAGE>_COMPILER_FRONTEND_VARIANT` was introduced since
CMake 3.14, but the variable `MSVC` works fine here.
Daniel Stenberg [Thu, 17 Oct 2024 14:01:08 +0000 (16:01 +0200)]
tool_operate: reuse the schannel backend check
The transfer_per_config is called once per new transfer. It now saves
the result of the first TLS backend check done so that subsequent
invokes are more efficient and reuses the existing knowledge.
This change also splits the logic into several smaller functions.
Stefan Eissing [Thu, 17 Oct 2024 15:00:41 +0000 (17:00 +0200)]
http2: auto reset stream on server eos
When a server signals EOS from its side and the curl upload is
unfinished and the server has not given a positive HTTP status response,
auto RST the stream to signal that the upload is incomplete and that the
whole transfer can be stopped.
Fixes the case where the server responds with 413 on an upload but does
not RST the stream from its side, as httpd and others do.
Reported-by: jkamp-aws on github
Fixes #15316
Closes #15325
Daniel Stenberg [Thu, 17 Oct 2024 15:50:02 +0000 (17:50 +0200)]
libtests: generate the lib1521 atomically
By renaming from a temporary file name to the .c once completed. This
avoids the risk that the checksrc job tries to verify the file before it
is complete, in parallel build setups.
Reported-by: Dan Frandrich
Fixes #15258
Closes #15327
Stefan Eissing [Thu, 17 Oct 2024 11:53:06 +0000 (13:53 +0200)]
openssl: improve retries on shutdown
Once SSL_shutdown() has been called, OpenSSL does not really seem to
like it when it is called again and the other side has some finally data
to deliver.
Instead SSL_read() needs to be used solely, once the close notify has
been sent from curl's side.
Daniel Stenberg [Thu, 17 Oct 2024 06:33:04 +0000 (08:33 +0200)]
GHA: switch off proselint
Because we cannot disable the individual warnings we do not care about,
making this tool almost unusable for our purposes. See
https://github.com/amperser/proselint/issues/1367
Instead, make 'very' a banned word (as recently that has been what
proselint most commonly points out for us).
Stefan Eissing [Wed, 16 Oct 2024 14:21:03 +0000 (16:21 +0200)]
tests/http: fix ubuntu GnuTLS CI failures
Override the system default config in test_17_09, since we want to check
all TLS versions. Provide own, empty config file to gnutls, so that any
system wide file has no effect.
The latest ubunu image in GH CI disables TLS 1.0 and 1.1
system wide for GnuTLS. Good intentions.
Viktor Szakats [Tue, 15 Oct 2024 00:58:44 +0000 (02:58 +0200)]
GHA/macos: merge autotools and cmake jobs
To match other workflows and to avoid repetition in rules.
Also:
- fix build example step for cmake. update a job to use it.
- use `cmake` to invoke the builds (instead of ninja directly).
- extend test 2100 exclusion to more jobs.
It fails with all `!debug gcc-12` jobs with autotools.
With cmake this only happened for gcc-12 Secure Transport jobs
for some reason.
Dan Fandrich [Wed, 16 Oct 2024 16:52:36 +0000 (09:52 -0700)]
CI: explicitly specify the OS version when necessary
Commit 8ea120f6 added --break-system-packages which works in Ubuntu
24.04 but not 22.04, so explicitly specify that version in the runner
instead of relying on ubuntu-latest to provide it. Some runners have
regressed back to 22.04 for ubuntu-latest, resulting in build failures.
Dan Fandrich [Sat, 12 Oct 2024 17:38:40 +0000 (10:38 -0700)]
tests: capture stdin to get the vsftpd version number
vsftpd 3.0 at least writes its version number to stdin (!) instead of
stderr. This works due for backwards compatibility reasons in UNIX, so
we must check stdin for anything written there to reliably parse the
version string.
Viktor Szakats [Wed, 16 Oct 2024 10:14:52 +0000 (12:14 +0200)]
src: guard for double declaration of `curl_ca_embed` in unity builds
Seen with curl-for-win linux-musl-from-mac build with gcc 9.2.0.
```
n file included from /Users/runner/work/curl-for-win/curl-for-win/curl/_x64-linux-musl-bld/src/CMakeFiles/curl.dir/Unity/unity_0_c.c:136:
/Users/runner/work/curl-for-win/curl-for-win/curl/_x64-linux-musl-bld/src/tool_ca_embed.c:4:28: warning: redundant redeclaration of 'curl_ca_embed' [-Wredundant-decls]
4 | extern const unsigned char curl_ca_embed[];
| ^~~~~~~~~~~~~
In file included from /Users/runner/work/curl-for-win/curl-for-win/curl/_x64-linux-musl-bld/src/CMakeFiles/curl.dir/Unity/unity_0_c.c:88:
/Users/runner/work/curl-for-win/curl-for-win/curl/src/tool_operate.c:107:28: note: previous declaration of 'curl_ca_embed' was here
107 | extern const unsigned char curl_ca_embed[];
| ^~~~~~~~~~~~~
```
https://github.com/curl/curl-for-win/actions/runs/11192203640/job/31116070669#step:3:4894
Daniel Stenberg [Sun, 13 Oct 2024 21:50:11 +0000 (23:50 +0200)]
libssh2: put the readdir buffers into struct
... instead of separate malloc() calls:
- removes two mallocs (and associated error handling paths)
- makes cleanup easier
Also reduce maximum SFTP file path lengths to 1024 bytes universally
everywhere. Using the system's own MAX_PATH did not make sense since
this is mostly about getting a remote file name.
Viktor Szakats [Mon, 14 Oct 2024 09:27:54 +0000 (11:27 +0200)]
GHA: silence proselint warnings and an error
Fix new issues found by `proselint`.
Also:
- silence this technical warning:
```
:0: DeprecationWarning: /home/runner/.proselintrc was found instead of a JSON file. Rename to /home/runner/.proselintrc.json.
```
- fix an input filename.
`proselints` fails now if an input file is missing.
Reported-by: Jay Satiro
Bug: https://github.com/curl/curl/pull/15291#issuecomment-2410505100
Closes #15293
Daniel Stenberg [Mon, 14 Oct 2024 12:09:59 +0000 (14:09 +0200)]
curl.h: remove the struct pointer for CURL/CURLSH/CURLM typedefs
It makes the callbacks get different signnatures when used from within
libcurl vs outside of it by libcurl-using applications (such as the
libtests) and this triggers UndefinedBehaviorSanitizer errors.
Stefan Eissing [Thu, 5 Sep 2024 14:41:53 +0000 (16:41 +0200)]
ftp: move listen handling to socket filter
Move the listen/accept handling of the FTP active data connection
into the socket filter and monitor 'connected' status of that as
with passive connections - more or less.
The advantage is that the socket filter now reports being connected
only when the server has actually called and accept() has been done.
This enables to bootstrap the filter chain on the data connection
just like any other. A require SSL filter can then be added right
at the start and does not need to be patched in later.
Still, the active connection keeps on needing special handling in
ftp.c as the control connection needs to be monitored while waiting
as the server might send error responses this way. So, things did
not turn out quite as squeaky clean as hoped for, but still seems
better to do that way.
Stefan Eissing [Thu, 10 Oct 2024 09:44:39 +0000 (11:44 +0200)]
wolfSSL: fix handling of TLSv1.3 sessions
Register a callback to get notified of new SSL sessions by wolfSSL.
Remove the explicit session retrieval after handshake, since this does
not work for TLSv1.3.
Adjust test expectations now that TLSv1.3 session resumption works
in wolfSSL.
Viktor Szakats [Fri, 11 Oct 2024 08:53:32 +0000 (10:53 +0200)]
smb: do not redefine `getpid` on Windows
Replace with namespaced local macro `Curl_getpid()`.
Redefining symbols can backfire if that symbol is used in system
headers, especially with unity build. We haven't seen a fallout in CI
or supported envs, but do it anyway for good measure.
Viktor Szakats [Sat, 12 Oct 2024 08:45:31 +0000 (10:45 +0200)]
GHA: optimize test prereq steps
- Linux: move test and pytest prereqs right before test run.
- returns build phase results faster.
- allows skipping steps for jobs that don't need them.
- makes dependencies more transparent.
- sync prereq install step names.
- use `tests/requirements.txt` more.
Viktor Szakats [Fri, 11 Oct 2024 16:36:43 +0000 (18:36 +0200)]
cmake: tidy-ups and rebase fixups
- limit `SIZEOF_SA_FAMILY_T` detection to non-Windows.
- make sure `sys/socket.h` exists before detecting `SIZEOF_SA_FAMILY_T`.
- limit `mach_absolute_time()` detection to `APPLE`. Drop from Windows
pre-cache.
- skip `HAVE_LIBSOCKET` detection for Windows, drop pre-cached value.
- drop redundant pre-cached `HAVE_LIBZ` for Windows.
- `curl_required_libpaths()`: stop accepting multiple arguments.
To prepare for `CMAKE_REQUIRED_LINK_DIRECTORIES` support.
Follow-up to 7bab201abe3915a0167c002f9308950cb8a06e4b #15193
- GSS: fix recent rebase mistakes:
- fix variable name.
- do not add a header twice.
Follow-up to 91d451b48809f20415ba8627786f5d4f5aaf8bfe #15157
- GSS: quote a variable.
Dan Fandrich [Wed, 9 Oct 2024 23:47:20 +0000 (16:47 -0700)]
tool_xattr: create the user.creator xattr attribute
This indicates that the file was created by curl which can help a user
determine the origin of a file. Like the other attributes, this is only
enabled with the --xattr option.
- reduce `check_include_file_concat()` use to those headers that either
depend on a previously detected header, or another header or symbol
detection depend on it.
- replace `check_symbol_exists()` with `check_function_exists()` for
functions that are detected with `AC_CHECK_FUNCS()` in `./configure`.
This makes `setmode()` no longer be detected with MSYS, syncing
this with `./configure`. Instead `_setmode()` is used now also in
CMake MSYS builds. This is consistent with Cygwin builds also.
- add comment about which header/symbol detection depends on what
header. Based on `./configure` mainly.
- form `CURL_TEST_DEFINES` manually, and include only those macros which
are actually used in `CMake/CurlTests.c`.
- change `curl_internal_test()` to use `CMAKE_REQUIRED_DEFINITIONS`,
instead of `CMAKE_REQUIRED_FLAGS` to simplify the logic, and to allow
dropping the latter macro completely.
- drop `windows.h` from header and symbol checks.
- `./configure`: add comment about whether `netinet/in6.h`, `sys/un.h`
are indeed meant to be included for all detections. There is a chance
they were added there by accident.
Stefan Eissing [Wed, 9 Oct 2024 12:46:32 +0000 (14:46 +0200)]
TLS: TLSv1.3 earlydata support for curl
Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.
If a known TLS session announces early data support, and the feature is
enabled *and* it is not a "connect-only" transfer, delay the TLS
handshake until the first request is being sent.
- Add --tls-earldata as new boolean command line option for curl.
- Add CURLSSLOPT_EARLYDATA to libcurl to enable use of the feature.
- Add CURLINFO_EARLYDATA_SENT_T to libcurl, reporting the amount of
bytes sent and accepted/rejected by the server.
Implementation details:
- store the ALPN protocol selected at the SSL session.
- When reusing the session and enabling earlydata, use exactly
that ALPN protocol for negoptiation with the server. When the
sessions ALPN does not match the connections ALPN, earlydata
will not be enabled.
- Check that the server selected the correct ALPN protocol for
an earlydata connect. If the server does not confirm or reports
something different, the connect fails.
- HTTP/2: delay sending the initial SETTINGS frames during connect,
if not connect-only.
Verification:
- add test_02_32 to verify earlydata GET with nghttpx.
- add test_07_70 to verify earlydata PUT with nghttpx.
- add support in 'hx-download', 'hx-upload' clients for the feature
Viktor Szakats [Sat, 5 Oct 2024 00:12:13 +0000 (02:12 +0200)]
cmake: replace `check_include_file_concat()` for LDAP and GSS detection
Replace `check_include_file_concat()` with `check_include_file()` in
GSS/LDAP detection to avoid these headers spilling into subsequent
feature checks.
- For LDAP, reverse detection order to match with `./configure`.
Though, in current LDAP packages `ldap.h` does include `lber.h`.
- For GSS, align header detection logic with `./configure`, where
`gssapi/gssapi_generic.h` might require `gssapi/gssapi.h`, and
`gssapi/gssapi_krb5.h` might require both.