Daniel Stenberg [Sat, 26 Oct 2024 20:12:41 +0000 (22:12 +0200)]
multi: split multi_runsingle into sub functions
Introduce five functions named after the state they serve:
- state_connect for MSTATE_CONNECT
- state_do for MSTATE_DO
- state_performing for MSTATE_PERFORMING
- state_ratelimiting for MSTATE_RATELIMITING
- state_resolving for MSTATE_RESOLVING
Viktor Szakats [Fri, 25 Oct 2024 01:01:58 +0000 (03:01 +0200)]
cmake: clear package version after `pkg-config` detection
`pkg_check_modules()` seems to leave `<PACKAGE>_VERSION` defined with an
empty value, if the package is not found.
When the package is also not found in the fallback branch,
`find_package_handle_standard_args()` logs and error message. In this
message it includes the bogus empty value as: `(found version "")`:
```
Could NOT find Libssh2 (missing: LIBSSH2_INCLUDE_DIR LIBSSH2_LIBRARY) (found version "")
```
https://github.com/curl/curl/actions/runs/11509727553/job/32040378958?pr=15408#step:31:99
Clear the version number to avoid the confusion:
```
Could NOT find Libssh2 (missing: LIBSSH2_INCLUDE_DIR LIBSSH2_LIBRARY)
```
https://github.com/curl/curl/actions/runs/11510022503/job/32041149129?pr=15408#step:31:99
Viktor Szakats [Thu, 24 Oct 2024 00:04:22 +0000 (02:04 +0200)]
cmake: document `-D` and env build options
Extend `INSTALL-CMAKE` document with the list of available options,
a short description and default values.
The list may not be 100% complete.
There are no component boundaries in CMake, so the line is blurry
between curl options, CMake options, CMake Find modules options.
I included certain CMake options that seemed useful, and/or have
dedicated use withing curl's CMake source. But, all CMake built-in
options are usable, as documented upstream in CMake.
The naming of the options has a heritage and the inconsistencies with
it, including a lack of clear namespace. This may be subject to future
updates, also after figuring out which name has special meaning within
CMake and/or CMake projects out of unwritten convention or something
more tangible.
CMake allows to initialize any internal variable via `-D`. This may be
useful to pre-initialize/override feature check results. The list
doesn't contain these, and they remain officially undocumented.
Also:
- make adjustments to keep the spellchecker happy.
- retrofit description changes to the cmake sources.
- stop documenting deprecated `Find*` variables.
Reported-by: Daniel Stenberg
Fixes https://github.com/curl/curl/discussions/14885
Closes #15388
Viktor Szakats [Thu, 24 Oct 2024 10:15:44 +0000 (12:15 +0200)]
cmake: tidy up and shorten symbol hiding initialization
- drop redundant local variables.
- drop extra Intel C feature check that always returned success due
to not passing the tested compiler option. Nobody reported an issue
with this since 2016, suggesting the version check alone is enough.
Daniel Stenberg [Thu, 24 Oct 2024 10:48:07 +0000 (12:48 +0200)]
url.md: clarify
- the specified URL can also get data sent to it
- rephrase the scheme guessing part
- mention target options for each URL for saving data
- mention --remote-name-all
- remove "warning" and make it into normal text
Daniel Stenberg [Thu, 24 Oct 2024 10:27:21 +0000 (12:27 +0200)]
version: minor cleanups
- remove typecasts and parentheses in zstd_version()
- create and use oldap_version() for OpenLDAP
- create and use psl_version() for libpsl
- reduce the size of the 40 byte buffers to 30 bytes
- use the brotil/zstd like the others (add the lib name in the functions)
- create and use idn_version for IDN builds
- handle (unlikely) error from ldap_get_option
- Create a new macro SCH_DEV() to manage verbose debug messages that are
only useful for debugging Schannel recv decryption.
schannel_recv contains a lot of useful debug messages to help debug the
function, however in practice they are not otherwise useful and showing
them in debug builds adds a lot of noise.
To show these messages curl must now be built with
CURL_SCHANNEL_DEV_DEBUG defined.
Prior to this change many, but not all, extra-verbose messages were
wrapped in DEBUGF() so they were only shown in debug builds.
Stefan Eissing [Wed, 23 Oct 2024 09:12:47 +0000 (11:12 +0200)]
schannel: ignore error on recv beyond close notify
When receiving data, schannel does a recv from the lower filters, e.g.
the socket, *before* it decrypts and analyses the buffered data it
already has. When that buffer contains a close-notify, e.g. the end of
the TLS stream, any error on the previous receive from the socket are
not applicable to its return codes.
Example from #153345: a server sends a close notify and closes its
connection. The encrypted data, including the close notify is received.
Another receive on the sockets gets a CONNABORTED which curl reports as
CURLE_RECV_ERROR. Schannel analyses its bufferi, sees the close notify
and early returns to the caller. On this return, the error on the
attempted receive does not apply.
renovate[bot] [Sat, 19 Oct 2024 23:17:04 +0000 (23:17 +0000)]
GHA: update five dependencies
- rojopolis/spellcheck-github-actions digest to ab8ac45
- nghttp2/nghttp2 to v1.64.0
- actions/cache digest to 6849a64
- github/codeql-action digest to 6624720
- Update actions/checkout digest to 11bd719
Viktor Szakats [Tue, 22 Oct 2024 11:22:39 +0000 (13:22 +0200)]
cmake: avoid setting `BUILD_TESTING`
`BUILD_TESTING` variable is used by other projects and CMake internally.
Replace `cmake_dependent_option()` with `option()` and introduce an
internal variable to track if want and can do testing.
Viktor Szakats [Wed, 23 Oct 2024 13:40:18 +0000 (15:40 +0200)]
libssh2: delete duplicate `break`
```
lib/vssh/libssh2.c:2495:7: warning: 'break' will never be executed [-Wunreachable-code-break]
break;
^~~~~
```
CI did not catch it due to llvm skipping this check for all #included
files. It's designed this way to avoid performance issues and false
positive when checking headers:
https://github.com/llvm/llvm-project/issues/71046
Viktor Szakats [Wed, 23 Oct 2024 09:06:27 +0000 (11:06 +0200)]
GHA/windows: work around Git for Windows perf regression
Fix the significant perf regression for vcpkg jobs by switching to the
MSYS2 shell environment from Git for Windows. This env is already used
for old-mingw-w64 job that remained unaffected by this issue.
The issue began with the windows-runner update 20241015.1.0. It bumped
Git for Windows from Git 2.46.2.windows.1 to Git 2.47.0.windows.1. GfW
bumped its MSYS2 components, including `msys-2.0.dll`. That's Cygwin
code, which may have contributed to this. Pipes were involved and
`runtests.pl` relies on pipes heavily in parallel mode. (The issue was
not seen with parallel tests disabled, in retrospect.)
This is useful as a permanent solution too. It drop GfW as a dependency
and makes Windows jobs use one less shell/env flavour.
Long term it might help to use native Windows Perl to avoid the MSYS
layer completely, if there is a way to make that work.
Viktor Szakats [Wed, 23 Oct 2024 08:54:14 +0000 (10:54 +0200)]
GHA/linux: drop patch from openssl3 thread sanitizer
The patch is now part of the 3.4.0 stable release.
(Turns out it was part of 3.3.2 already.)
Also:
- rename this local build to match the scheme used with wolfssl.
- drop '3' from local openssl build name.
- sync job name with others.
- quote step names where missing.
Viktor Szakats [Tue, 17 Sep 2024 16:43:17 +0000 (18:43 +0200)]
runtests: use deterministic sort for `TESTINFO` lines
Sort TESTINFO lines by description within the number of skipped test.
It makes the list of skipped test groups easier to diff/compare between
jobs and runs.
Max Dymond [Tue, 22 Oct 2024 12:31:22 +0000 (13:31 +0100)]
ci: fix renovate's matching for OpenSSL and quictls
Renovate only matches on the raw version numbers of a package, but
OpenSSL includes `openssl-` as a prefix in the version number. This
change means that the match string now expects the `openssl-` prefix
and will just update the version portion.
This also updates quictls so that renovate can detect and update the
version correctly.
Viktor Szakats [Tue, 22 Oct 2024 15:02:14 +0000 (17:02 +0200)]
cmake/FindCares: fix version detection for c-ares 1.34.1
Due to a regression in c-ares 1.34.1, the non-pkg-config version
detection method broke for this version. c-ares 1.34.2 fixes it,
but update our detection code anyway to also work with 1.34.1.
Viktor Szakats [Tue, 22 Oct 2024 15:32:28 +0000 (17:32 +0200)]
cmake: use the `BSD` variable
- use `BSD` in addition to backwards-compatible method.
- add `BSD` to the configuration log and `buildinfo.txt` if detected.
- add `BSD` tag to `buildinfo.txt` also via `./configure`.
The `BSD` variable is supported by CMake 3.25.0 and upper.
Viktor Szakats [Tue, 22 Oct 2024 14:25:15 +0000 (16:25 +0200)]
GHA/windows: increase timeout for vcpkg jobs due to slowness
The openssl job no longer fits into 10 minutes since the 20241015.1.0
GHA windows-latest image update. This caused all runs to fail.
The `run tests` step takes ~10 minutes now, up from ~4. This is
6 minutes more than before these updates. It's seen with other vcpkg
jobs too, tests run slower than half speed since.
Bump the timeout to make it, though the headroom is now less than it
was.
Stefan Eissing [Tue, 22 Oct 2024 08:39:44 +0000 (10:39 +0200)]
CI: run with standard mod_http2
We used to include a special mod_h2 in our CI that supports the
directive H2MaxDataFrameLen for test_02_20. Since then, ubuntu-lastest
includes a more recent apache httpd. Let's see if we can live without
the special.
Viktor Szakats [Tue, 22 Oct 2024 11:51:32 +0000 (13:51 +0200)]
GHA/windows: ignore results for test 987
987 is `SMTPS with redundant explicit SSL request`.
Root cause undiscovered.
Started failing after GHA bumping the windows image to `20241015.1.0`:
https://github.com/actions/runner-images/commit/fcc4cdb1d095af1317859c4809364538953b3497
https://github.com/actions/runner-images/pull/10803
vcpkg packages also got bumped as a result. They seem unrelated:
c-ares 1.33.1 -> 1.34.1
nghttp2 1.62.1 -> 1.63.0
nghttp3 1.5.0 -> 1.6.0
(there may be more)
Viktor Szakats [Mon, 21 Oct 2024 17:06:50 +0000 (19:06 +0200)]
GHA/linux: tidy up and performance
- replace openssl3 default local build with packaged one.
- drop valgrind from IntelC job.
- drop IntelC no-ssl job.
- bump local openssl to 3.3.2.
- disable tests in the cmake variant of a job.
- add comment to the remaining local openssl3 build.
We can drop the patch after upgrading to upcoming 3.4.0.
- drop gcc-11 from jobs. packaged gcc is now newer at 13.2.0.
(saves more than 1m install time for each of the 5 jobs.)
Follow-up to 9cc9a6472c5d2e13a117ca02f432443db2d1be57 #9454
zjyhjqs [Wed, 16 Oct 2024 16:17:38 +0000 (00:17 +0800)]
cmake: fix compile warnings for clang-cl
clang-cl is an alternative command-line interface to Clang, designed
for compatibility with the Visual C++ compiler, `cl.exe`:
https://clang.llvm.org/docs/UsersManual.html#clang-cl
The way to test clang-cl in CMake:
- `CMAKE_<LANGUAGE>_COMPILER_ID`: "Clang"
- `CMAKE_<LANGUAGE>_COMPILER_FRONTEND_VARIANT`: "MSVC"
Note: `CMAKE_<LANGUAGE>_COMPILER_FRONTEND_VARIANT` was introduced since
CMake 3.14, but the variable `MSVC` works fine here.
Daniel Stenberg [Thu, 17 Oct 2024 14:01:08 +0000 (16:01 +0200)]
tool_operate: reuse the schannel backend check
The transfer_per_config is called once per new transfer. It now saves
the result of the first TLS backend check done so that subsequent
invokes are more efficient and reuses the existing knowledge.
This change also splits the logic into several smaller functions.
Stefan Eissing [Thu, 17 Oct 2024 15:00:41 +0000 (17:00 +0200)]
http2: auto reset stream on server eos
When a server signals EOS from its side and the curl upload is
unfinished and the server has not given a positive HTTP status response,
auto RST the stream to signal that the upload is incomplete and that the
whole transfer can be stopped.
Fixes the case where the server responds with 413 on an upload but does
not RST the stream from its side, as httpd and others do.
Reported-by: jkamp-aws on github
Fixes #15316
Closes #15325
Daniel Stenberg [Thu, 17 Oct 2024 15:50:02 +0000 (17:50 +0200)]
libtests: generate the lib1521 atomically
By renaming from a temporary file name to the .c once completed. This
avoids the risk that the checksrc job tries to verify the file before it
is complete, in parallel build setups.
Reported-by: Dan Frandrich
Fixes #15258
Closes #15327
Stefan Eissing [Thu, 17 Oct 2024 11:53:06 +0000 (13:53 +0200)]
openssl: improve retries on shutdown
Once SSL_shutdown() has been called, OpenSSL does not really seem to
like it when it is called again and the other side has some finally data
to deliver.
Instead SSL_read() needs to be used solely, once the close notify has
been sent from curl's side.
Daniel Stenberg [Thu, 17 Oct 2024 06:33:04 +0000 (08:33 +0200)]
GHA: switch off proselint
Because we cannot disable the individual warnings we do not care about,
making this tool almost unusable for our purposes. See
https://github.com/amperser/proselint/issues/1367
Instead, make 'very' a banned word (as recently that has been what
proselint most commonly points out for us).
Stefan Eissing [Wed, 16 Oct 2024 14:21:03 +0000 (16:21 +0200)]
tests/http: fix ubuntu GnuTLS CI failures
Override the system default config in test_17_09, since we want to check
all TLS versions. Provide own, empty config file to gnutls, so that any
system wide file has no effect.
The latest ubunu image in GH CI disables TLS 1.0 and 1.1
system wide for GnuTLS. Good intentions.
Viktor Szakats [Tue, 15 Oct 2024 00:58:44 +0000 (02:58 +0200)]
GHA/macos: merge autotools and cmake jobs
To match other workflows and to avoid repetition in rules.
Also:
- fix build example step for cmake. update a job to use it.
- use `cmake` to invoke the builds (instead of ninja directly).
- extend test 2100 exclusion to more jobs.
It fails with all `!debug gcc-12` jobs with autotools.
With cmake this only happened for gcc-12 Secure Transport jobs
for some reason.
Dan Fandrich [Wed, 16 Oct 2024 16:52:36 +0000 (09:52 -0700)]
CI: explicitly specify the OS version when necessary
Commit 8ea120f6 added --break-system-packages which works in Ubuntu
24.04 but not 22.04, so explicitly specify that version in the runner
instead of relying on ubuntu-latest to provide it. Some runners have
regressed back to 22.04 for ubuntu-latest, resulting in build failures.
Dan Fandrich [Sat, 12 Oct 2024 17:38:40 +0000 (10:38 -0700)]
tests: capture stdin to get the vsftpd version number
vsftpd 3.0 at least writes its version number to stdin (!) instead of
stderr. This works due for backwards compatibility reasons in UNIX, so
we must check stdin for anything written there to reliably parse the
version string.
Viktor Szakats [Wed, 16 Oct 2024 10:14:52 +0000 (12:14 +0200)]
src: guard for double declaration of `curl_ca_embed` in unity builds
Seen with curl-for-win linux-musl-from-mac build with gcc 9.2.0.
```
n file included from /Users/runner/work/curl-for-win/curl-for-win/curl/_x64-linux-musl-bld/src/CMakeFiles/curl.dir/Unity/unity_0_c.c:136:
/Users/runner/work/curl-for-win/curl-for-win/curl/_x64-linux-musl-bld/src/tool_ca_embed.c:4:28: warning: redundant redeclaration of 'curl_ca_embed' [-Wredundant-decls]
4 | extern const unsigned char curl_ca_embed[];
| ^~~~~~~~~~~~~
In file included from /Users/runner/work/curl-for-win/curl-for-win/curl/_x64-linux-musl-bld/src/CMakeFiles/curl.dir/Unity/unity_0_c.c:88:
/Users/runner/work/curl-for-win/curl-for-win/curl/src/tool_operate.c:107:28: note: previous declaration of 'curl_ca_embed' was here
107 | extern const unsigned char curl_ca_embed[];
| ^~~~~~~~~~~~~
```
https://github.com/curl/curl-for-win/actions/runs/11192203640/job/31116070669#step:3:4894
Daniel Stenberg [Sun, 13 Oct 2024 21:50:11 +0000 (23:50 +0200)]
libssh2: put the readdir buffers into struct
... instead of separate malloc() calls:
- removes two mallocs (and associated error handling paths)
- makes cleanup easier
Also reduce maximum SFTP file path lengths to 1024 bytes universally
everywhere. Using the system's own MAX_PATH did not make sense since
this is mostly about getting a remote file name.
Viktor Szakats [Mon, 14 Oct 2024 09:27:54 +0000 (11:27 +0200)]
GHA: silence proselint warnings and an error
Fix new issues found by `proselint`.
Also:
- silence this technical warning:
```
:0: DeprecationWarning: /home/runner/.proselintrc was found instead of a JSON file. Rename to /home/runner/.proselintrc.json.
```
- fix an input filename.
`proselints` fails now if an input file is missing.
Reported-by: Jay Satiro
Bug: https://github.com/curl/curl/pull/15291#issuecomment-2410505100
Closes #15293
Daniel Stenberg [Mon, 14 Oct 2024 12:09:59 +0000 (14:09 +0200)]
curl.h: remove the struct pointer for CURL/CURLSH/CURLM typedefs
It makes the callbacks get different signnatures when used from within
libcurl vs outside of it by libcurl-using applications (such as the
libtests) and this triggers UndefinedBehaviorSanitizer errors.