Viktor Szakats [Thu, 21 Nov 2024 18:22:43 +0000 (19:22 +0100)]
GHA: disable building tests, apps, docs in dependencies
Also:
- for LibreSSL download the official source tarball instead of
using the tagged Git repo and running the build script which
merged the OpenBSD libressl repo into it. The latter method
was also broken at the time of this commit.
LibreSSL build options are unchanged, but by using the tarball now
instead of two repos and a generator script, it also should be faster,
and more stable.
Viktor Szakats [Thu, 21 Nov 2024 19:07:21 +0000 (20:07 +0100)]
cmake: include `wolfssl/options.h` first
It was missing while detecting `wolfSSL_DES_ecb_encrypt`,
`wolfSSL_BIO_new` and `wolfSSL_BIO_set_shutdown`.
We have not seen it causing issues in stable wolfSSL releases as of
v5.7.4, until a recent commit in wolfSSL master, which broke detections:
```
curl/CMakeFiles/CMakeScratch//CheckSymbolExists.c:8:19: error: ‘wolfSSL_BIO_new’ undeclared (first use in this function); did you mean ‘wolfSSL_CTX_new’?
curl/CMakeFiles/CMakeScratch//CheckSymbolExists.c:8:19: error: ‘wolfSSL_BIO_set_shutdown’ undeclared (first use in this function); did you mean ‘wolfSSL_set_shutdown’?
```
This in turn disabled `HTTPS-proxy` and failed related pytests:
https://github.com/curl/curl/actions/runs/11953800545/job/33324250039?pr=15620
The wolfSSL build says:
```
Note: Make sure your application includes "wolfssl/options.h" before any other wolfSSL headers.
You can define "WOLFSSL_USE_OPTIONS_H" in your application to include this automatically.
```
This patch makes sure to follow this rule across the curl codebase.
Also:
- include `wolfssl/options.h` first in `lib/vtls/wolfssl.c`.
It was preceded by `wolfssl/version.h`, which did not cause issues.
Background for the pre-existing include order:
Ref: deb9462ff2de8e955c67ed441f5f48619a31198d #3903
Ref: https://curl.se/mail/lib-2015-04/0069.html
Wyatt O'Day [Thu, 21 Nov 2024 14:08:31 +0000 (09:08 -0500)]
schannel: remove TLS 1.3 ciphersuite-list support
Drop TLS 1.3 ciphersuite-list support from SChannel because of the
number of bugs in SChannel itself (a closed-source TLS library). TLS 1.3
with SChannel still works, however the ciphersuite negotiation is left
entirely to SChannel.
Bug: https://hackerone.com/reports/2792484 Reported-by: newfunction on hackerone
Fixes https://github.com/curl/curl/issues/15482
Closes https://github.com/curl/curl/pull/15621
Viktor Szakats [Thu, 14 Nov 2024 21:25:15 +0000 (22:25 +0100)]
cmake: do not echo most inherited `LDFLAGS` to config files
Sync with autotools and filter out most linker flags inherited via
`CMAKE_SHARED_LINKER_FLAGS` (that includes `LDFLAGS` env) before
echoing them in `libcurl.pc` `Libs.private` and `curl-config`
`--static-libs`.
Jay Satiro [Wed, 13 Nov 2024 21:17:37 +0000 (16:17 -0500)]
curl_multi_socket_all.md: soften the deprecation warning
- Instead of saying "do not use", explain that the function was
deprecated for performance reasons.
Some users may have a legitimate use of this function even though we
deprecated it. Since there are no plans to remove it from the API get
rid of the "do not use" warning.
Bug: https://curl.se/mail/lib-2024-11/0029.html Reported-by: Jacob Champion
Closes https://github.com/curl/curl/pull/15576
The callstack used to end with `Curl_write_plain()` accepting a socket
till 7.87.0. This call got swapped for `Curl_conn_send()`, expecting
a sockindex. `socket_write()` was updated accordingly. Its callers
missed it and continued operating on sockets: `do_sec_send()`,
`sec_write()`, passing it down the stack and `Curl_conn_send()`
resolving it as if it were a sockindex.
It affected FTP Kerberos authentication.
Discovered through MSVC warnings:
```
curl\lib\krb5.c(652,28): warning C4244: 'function': conversion from 'curl_socket_t' to 'int', possible loss of data
curl\lib\krb5.c(654,28): warning C4244: 'function': conversion from 'curl_socket_t' to 'int', possible loss of data
curl\lib\krb5.c(656,26): warning C4244: 'function': conversion from 'curl_socket_t' to 'int', possible loss of data
curl\lib\krb5.c(657,26): warning C4244: 'function': conversion from 'curl_socket_t' to 'int', possible loss of data
curl\lib\krb5.c(665,24): warning C4244: 'function': conversion from 'curl_socket_t' to 'int', possible loss of data
curl\lib\krb5.c(666,24): warning C4244: 'function': conversion from 'curl_socket_t' to 'int', possible loss of data
```
Ref: https://github.com/curl/curl/actions/runs/11846599621/job/33014592805#step:9:32
Viktor Szakats [Thu, 14 Nov 2024 22:28:42 +0000 (23:28 +0100)]
configure: replace `$#` shell syntax
With a more portable alternative.
Fixes (seen on macOS):
```
../configure: line 47131: 1: command not found
```
Ref: https://github.com/curl/curl/actions/runs/11846071276/job/33012894013#step:7:635
Viktor Szakats [Wed, 13 Nov 2024 01:16:31 +0000 (02:16 +0100)]
cmake: restore cmake args list in `buildinfo.txt`
This feature was recently dropped because of a bad side-effect of
silencing unused cmake command-line option warnings.
Fix this issue by retrieving variable values using `get_property()`,
instead of accessing the variables directly. It allows restoring
this feature without the bad side-effect.
Viktor Szakats [Wed, 13 Nov 2024 13:44:05 +0000 (14:44 +0100)]
build: omit certain deps from `libcurl.pc` unless found via `pkg-config`
The idea of linking dependencies found to `libcurl.pc` turns out not
to work in practice in some cases.
Specifically: gss, ldap, mbedtls, libmsh3, rustls
A `.pc` may not work or be missing for a couple of reasons:
- not all build methods generate it: mbedTLS, Rustls
- generated file is broken: msh3
Ref: https://github.com/nibanks/msh3/pull/225
- installed package flavour isn't shipping with one:
FreeBSD GSS, OmniOS LDAP, macOS LDAP
The effect of such issues shall be subtle in theory, because
`libcurl.pc` normally lists these dependencies in the `Requires.private`
section meant for static linking. But, e.g. `pkg-config --exists`
requires these to be present, and builds sometimes use this check
regardless of build type. This bug is not present in `pkgconf`; it only
checks for them when `--static` is also passed.
Fix these by adding affected `.pc` references to `libcurl.pc` only when
we detected the dependency via `pkg-config`.
There are a few side-effects of this solution:
- references are never added for dependencies where curl doesn't
implement `pkg-config` detection. These are:
- autotools: ldap, mbedtls, msh3
- cmake: ldap (pending #15273)
- generated `libcurl.pc` depends on the build-time environment.
- generated `libcurl.pc` depends on curl build tool (cmake, autotools).
- generated `libcurl.pc` depends on curl build implementation details.
Make an exception for GNU GSS, where I blindly guess that `gss.pc` is
always available, as no issues were reported.
Other, not mentioned, dependencies continue to be added regardless
of the detection method.
Reported-by: Harmen Stoppels, Thomas, Daniel Engberg, Andy Fiddaman
Fixes #15469
Fixes #15507
Fixes #15535
Fixes https://github.com/curl/curl/pull/15163#issuecomment-2473358444
Closes #15573
Viktor Szakats [Mon, 11 Nov 2024 10:03:08 +0000 (11:03 +0100)]
cmake: sync GSS config code with other deps
- stop passing explicit libpaths via `CMAKE_SHARED_LINKER_FLAGS` and
`CMAKE_EXE_LINKER_FLAGS`. `link_directories()` is doing that already.
- use `curl_required_libpaths()` to pass libpaths to the feature test. Reported-by: Daniel Engberg
Fixes #15536
Also fixes GSS feature detection with non-gcc/clang compilers,
such as MSVC.
- add libpaths to `CURL_LIBPATHS`.
- move `GSS_CFLAGS`, `GSS_LDFLAGS` stringifications to FindGSS.
To match the `CFLAGS` format returned by the rest of Find modules.
- reorder calls to match other dependencies.
- don't extend system `LDFLAGS` when FindGSS did not return any.
- ignore `LDFLAGS` when detecting GSS via `pkg-config`. `LDFLAGS` holds
a copy of libpaths and libs in this case. Ignore those to avoid these
duplicates making into `libcurl.pc` and `curl-config`. Also syncing
behavior with other Find modules which also ignore raw `LDFLAGS`.
- ignore raw `LDFLAGS` coming from `krb5-config --libs`. FindGSS
no longer returns dependency-specific `LDFLAGS` after this. Syncing
behavior with other Find modules.
- reduce scope of checker state push/pop/set.
Viktor Szakats [Tue, 12 Nov 2024 12:37:33 +0000 (13:37 +0100)]
strtok: use namespaced `strtok_r` macro instead of redefining it
krb5 defines `strtok_r` for Windows unconditionally in its public
header:
https://github.com/krb5/krb5/blob/dc5554394e5a4363b3e109623edbeb9ad6c18a62/src/include/win-mac.h#L214-L215
resulting in this warning:
```
lib\strtok.h(31,9): warning C4005: 'strtok_r': macro redefinition
C:\vcpkg\installed\x64-windows\include\win-mac.h(215,9):
see previous definition of 'strtok_r'
```
The krb5 macro collides with curl's internal definition, in case
the `strtok_r` function is undetected and falling back to a local
replacement.
Reported-by: Tal Regev
Bug: https://github.com/curl/curl/pull/15549#issuecomment-2468251761
Closes #15564
Viktor Szakats [Mon, 11 Nov 2024 12:21:58 +0000 (13:21 +0100)]
configure: do not echo most inherited `LDFLAGS` to config files
`libcurl.pc` `Libs.private` (since 8.11.0, and in `Libs` before 7.20.0)
and `curl-config` `--static-libs` (since 7.17.1, and in `Libs` between
7.7.2-7.25.0). This included all flags inherited from the environment,
in addition to those coming from dependency detections.
To avoid spilling all linker flags inherited from the environment to
the libcurl config files, this patch omits them all, except `-L`, `-F`,
`--library-path=` and `-framework` options, which are still passed.
The rationale for the exceptions is that `LIBS` is passed as-is, and
`LDFLAGS`, `LIBS` are the canonical way to pass custom libs options
to a build. `LIBS` may not work without a matching custom libpath.
This brings autotools behaviour closer to cmake, and `curl-config`
closer to `libcurl.pc`.
Viktor Szakats [Wed, 13 Nov 2024 22:43:49 +0000 (23:43 +0100)]
GHA/linux: fix `pip3 install impacket` breakage
An upstream update `impacket` pip package started requiring `blinker`.
An older version is shipping with Ubuntu, causing this on install:
```
Attempting uninstall: blinker
Found existing installation: blinker 1.7.0
ERROR: Cannot uninstall blinker 1.7.0, RECORD file not found. Hint: The package was installed by debian.
```
Fix it by switching to venv and install everything separate from the
system.
The overhead is the same as using `pip --ignore-installed`, which also
installs everything from scratch.
The 3rd option is to uninstall the system `python3-blinker` package, but
it was the slowest.
IBM's iconv_open expects parameters to be a pointer to a 32 byte
character array with the unused fields set to 0.
Prior to this change, since 8c62479a (precedes 8.11.0), it was
incorrectly passed pointers to smaller length const strings and curl
would fail with error "blank argument where content is expected".
Reported-by: Andrew Kirillov
Ref: https://www.ibm.com/docs/en/i/7.5?topic=ssw_ibm_i_75/apis/iconvopn.html
Viktor Szakats [Wed, 13 Nov 2024 17:00:34 +0000 (18:00 +0100)]
GHA/macos: follow Homebrew and switch to `pkgconf`
Homebrew switched to `pkgconf`, and now pkg-config installs an extra
package. Update package list to avoid that.
The side-effect of `pkgconf` is that this former log message:
```
-- Package 'libcrypto', required by 'libssh2', not found
```
https://github.com/curl/curl/actions/runs/11779568834/job/32808325442#step:7:84
is replaced by this, and repeated 10 times:
```
Package libcrypto was not found in the pkg-config search path.
Perhaps you should add the directory containing `libcrypto.pc'
to the PKG_CONFIG_PATH environment variable
Package 'libcrypto', required by 'libssh2', not found
```
https://github.com/curl/curl/actions/runs/11792711391/job/32846858320#step:7:85
Viktor Szakats [Tue, 12 Nov 2024 14:22:50 +0000 (15:22 +0100)]
cmake: work around `ios.toolchain.cmake` breaking feature-detections
Fix builds with CMake configured to falsely return successful detection
when using `check_function_exists()` (and `check_library_exists()`, and
anything based on `try_compile()` that's relying on the linker). After
such mis-detection the build fails when trying to use the feature that
doesn't in fact exist.
The mis-detection is caused by this CMake setting:
```
set(CMAKE_TRY_COMPILE_TARGET_TYPE STATIC_LIBRARY)
```
It is set by default (or on conditions) when using 3rd-party toolchain:
https://github.com/leetal/ios-cmake/blob/master/ios.toolchain.cmake
After this patch the curl build overrides this setting for the duration
of feature tests, and logs a message about it.
Also preset and skip feature tests for known mis-detections:
- `connect()` in `libsocket`
- `getpass_r()`
- `eventfd()` (did not cause an issue due to a separate bug)
- `sendmmsg()` (did not cause an issue because it's Linux-only)
If mis-detections are still seen, the workaround is to force-set the
specific feature by passing `-DHAVE_*=OFF` to cmake.
Also consider passing `-DENABLE_STRICT_TRY_COMPILE=ON` for
`ios.toolchain.cmake` to fix the root cause.
Also:
- OtherTests.cmake: restore original value of
`CMAKE_TRY_COMPILE_TARGET_TYPE`. Before this patch it reset it
to empty.
- OtherTests.cmake: unset a local variable after use, quote a string.
Follow-up to 8e345057761a8f796403923a96f2c8fd3edca647 #15164
Follow-up to 8b76a8aeb21c8ae2261147af1bddd0d4637c252c #15525
Ref: https://github.com/leetal/ios-cmake/issues/47
Ref: https://gitlab.kitware.com/cmake/cmake/-/issues/18121
Ref: https://cmake.org/cmake/help/latest/variable/CMAKE_TRY_COMPILE_TARGET_TYPE.html Reported-by: Dan Rosser
Fixes #15557
Closes #15559
Daniel Engberg [Sun, 10 Nov 2024 21:23:26 +0000 (22:23 +0100)]
GHA/non-native: streamline installed packages on FreeBSD
Instead of installing the cmake package which is a meta-port (includes
documentation and manpages etc) install cmake-core which is cmake itself
to save a few cpu cycles.
Also drop nghttp2 in favour of the slimmer libnghttp2.
The script's previous treatment of this meta-data was a
misunderstanding. (Added in 1ebc53df25181908) The mistrust is not for
the root cert at this date (it would simply be removed from the bundle
then instead) but for created server certificates:
If a builtin certificate has a CKA_NSS_SERVER_DISTRUST_AFTER
timestamp before the SCT or NotBefore date of a certificate that
builtin issued, then clients can elect not to trust it.
That is however information that cannot be provided in the generated PEM
output.
Fixes #15547 Reported-by: Andrew Ayer
Closes #15552
Viktor Szakats [Mon, 11 Nov 2024 12:59:08 +0000 (13:59 +0100)]
curl-rustls.m4: keep existing `CPPFLAGS`/`LDFLAGS` when detected
Different variable names were used in flag save and restore operations,
which could cause existing `CPPFLAGS` and `LDFLAGS` be accidentally lost
when detecting Rustls.
Viktor Szakats [Fri, 8 Nov 2024 17:51:45 +0000 (18:51 +0100)]
build: use `_fseeki64()` on Windows, drop detections
A recent update caused CMake builds to mis-detect this symbol on iOS.
Auto-detection also seems redundant given that it's a Windows-only
function and most Windows builds were already opted-in.
Drop detections and use it in all Windows builds with large file support
enabled.
Ethan Everett [Fri, 8 Nov 2024 19:49:32 +0000 (11:49 -0800)]
nghttp2: use custom memory functions
Provide libcurl's memory functions as callbacks to replace nghttp2's own memory
functions. This allows custom memory callbacks provided by users of libcurl to
be used by nghttp2 as well.
Viktor Szakats [Thu, 7 Nov 2024 07:09:30 +0000 (08:09 +0100)]
macos: disable gcc `availability` workaround as needed
Homebrew gcc 14.2.0_1 fixed the issue, and the workaround is no longer
needed. Not only not needed, but the workaround is breaking builds with
the fixed gcc.
Auto-detect the upstream fix and stop applying the local workaround if
detected.
Viktor Szakats [Fri, 8 Nov 2024 00:29:50 +0000 (01:29 +0100)]
GHA/macos: let gcc dictate the configured Apple SDK
As discovered earlier, Homebrew gcc is built against a specific Apple
SDK version and doesn't work when matched up with a different version,
e.g. the one advertised as default by the macos runner image.
Before this patch this was resolved with brute force by zapping the
hack-layer gcc component to avoid the bad interference. This worked
for us, but it's fragile, accidental and doesn't translate to
real-world build environments. Thus, impractical.
Avoid this by explicitly selecting the SDK version gcc was built for and
meant to be used with, as shown by `gcc --print-sysroot`.
It assumes that the gcc binaries preinstalled on the runner images
always ship with the SDK version they reference. It also assumes
this works with and without `brew update`.
Also:
- add 4 quick build-only jobs to test all gcc/macos combos.
- list SDKs offered via CommandLineTools.
Suggested-by: Bo Anderson
Ref: https://github.com/Homebrew/homebrew-core/issues/194778#issuecomment-2462764619
Viktor Szakats [Fri, 8 Nov 2024 02:17:44 +0000 (03:17 +0100)]
GHA: add `apt update` where missing
To mitigate this kind of (repeat) errors:
```
Err:14 http://azure.archive.ubuntu.com/ubuntu noble-updates/main amd64 python3-werkzeug all 3.0.1-3ubuntu0.1
404 Not Found [IP: 40.81.13.82 80]
E: Failed to fetch http://azure.archive.ubuntu.com/ubuntu/pool/main/p/python-werkzeug/python3-werkzeug_3.0.1-3ubuntu0.1_all.deb
Fetched 10.4 MB in 4s (2593 kB/s)
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
```
https://github.com/curl/curl/actions/runs/11732257460/job/32684111508#step:3:74
Daniel Stenberg [Tue, 5 Nov 2024 10:52:38 +0000 (11:52 +0100)]
mbedtls: remove failf() use from mbedtls_random
Since data can be NULL in here, calling failf() can be bad. This should
also be a terribly rare error so the lack of error message for this
should be manageable.
Reported-by: wxiaoguang on github
Fixes #15485
Closes #15486
Daniel Stenberg [Fri, 1 Nov 2024 16:05:15 +0000 (17:05 +0100)]
openssl: extend the OpenSSL error messages
Previously there were three error situations that only added the (unhelpful)
OpenSSL error strings, now we prefix those with a short explanation for the
error situation.
curl_addrinfo: support operating systems with only getaddrinfo(3)
The gethostbyname(3) family was removed in POSIX-1.2008 in favor of
getaddrinfo(3) introduced in POSIX-1.2001. Modern POSIX systems such as
Sortix does not have gethostbyname nor the related definitions and
structures.
curl already only uses getaddrinfo(3) if available and thread safe,
although there is mild breakage if the related gethostbyname definitions
are missing.
This change attempts to fix that breakage:
Remove an unnecessary configure error if gethostbyname is missing since
getaddrinfo is enough as a fallback.
Rewrite Curl_ip2addr to not use struct hostent as it no longer is
standardized and create the struct Curl_addrinfo directly.
Only define the Curl_he2ai function on non-getaddrinfo systems where it
is going to be used with struct hoestent.
Revoke the fallback logic for when it's unknown whether getaddrinfo is
thread safe. It doesn't appear to make any sense since h_errno is
unrelated to getaddrinfo. The logic prevents new POSIX.1-2024 systems
from passing the thread safety test since h_errno does not exist anymore
and POSIX already requires getaddrinfo to be thread safe. There's
already a denylist in place for operating systems with known buggy
implementations.
Stefan Eissing [Thu, 31 Oct 2024 11:47:05 +0000 (12:47 +0100)]
ngtcp2: do not loop on recv
The vquic_recv_packets() function already loops when not all requested
packets can be received (until EAGAIN) and there is not need to do that
again in ngtcp2.
Viktor Szakats [Tue, 29 Oct 2024 15:13:45 +0000 (16:13 +0100)]
build: fix clang-cl builds, add CI job
- appveyor: add build-only job for clang-cl.
- cmake: `-pedantic-errors` enables `-Werror,-Wlanguage-extension-token`
automatically, which makes `__int64` detection fail.
Explictly disable this compiler warning for clang-cl to make the
feature detection work and to accept `__int64` in the source code.
- cmake: disable `-Wlanguage-extension-token` warning for clang-cl
to fix these when encountering `__int64`:
```
lib/formdata.c(797,29): error : extension used [-Werror,-Wlanguage-extension-token]
lib/warnless.c(117,33): error : extension used [-Werror,-Wlanguage-extension-token]
lib/warnless.c(60,28): message : expanded from macro 'CURL_MASK_SCOFFT'
lib/warnless.c(59,38): message : expanded from macro 'CURL_MASK_UCOFFT'
include\curl/system.h(352,40): message : expanded from macro 'CURL_TYPEOF_CURL_OFF_T'
```
- make `__GNUC__` warning suppressions apply to `__clang__` too.
Necessary for clang-cl, which defines the latter, but not the former.
(Regular clang defines both.)
- examples: fix clang-cl compiler warning in `http2-upload.c`.
```
docs\examples\http2-upload.c(56,5): error : no previous prototype for function 'my_gettimeofday' [-Werror,-Wmissing-prototypes]
docs\examples\http2-upload.c(56,1): message : declare 'static' if the function is not intended to be used outside of this translation unit
```
Marwan Yassini [Tue, 29 Oct 2024 17:22:03 +0000 (17:22 +0000)]
mqtt: fix mqtt.md wording and add clearer explanation
Some of the wording in the mqtt.md confused me as to how the commands
were used and what they did, so I cleared up some of the wording to
better explain what each command does.
- Remove reference to 'Developer Command Prompt for Visual Studio'
shortcut since it opens in x86 mode.
That prompt may confuse users since it is not easily switched to x64.
Our instruction says vcvarsall can be used to change the platform but it
is not in the path in any version that I checked (VS 2010, 2013, 2022).
Instead users will now only see the remaining instruction to use a
platform specific command prompt to build curl, like "x64 Native Tools".
There's several links as well to Microsoft documentation for users that
have more complicated requirements, such as using vcvarsall.
Daniel Stenberg [Tue, 29 Oct 2024 08:21:37 +0000 (09:21 +0100)]
tool_operate: url_proto improvements
- renamed to url_proto_and_rewrite to better reveal what it does
- clarify the functionality in the top comment
- make it return CURLE_OUT_OF_MEMORY appropriately
- remove check for URL being set, use assert instead