Daniel Stenberg [Tue, 21 Jan 2025 10:42:20 +0000 (11:42 +0100)]
asyn-thread: use c-ares to resolve HTTPS RR
Allow building with c-ares and yet use threaded resolver for the main
host A/AAAA resolving:
`--with-ares` provides the c-ares install path and defaults to use
c-ares for name resolving
`--with-threaded-resolver` still uses c-ares in the build (for HTTPS)
but uses the threaded resolver for "normal" resolves.
It works similarly for cmake: ENABLE_ARES enables ares, and if
ENABLE_THREADED_RESOLVER also is set, c-ares is used for HTTPS RR and
the threaded resolver for "normal" resolves.
HTTPSRR and c-ares-rr are new features return by curl_version_info() and
thus shown by curl -V.
The c-ares-rr feature bit is there to make it possible to distinguish
between builds using c-ares for all name resolves and builds that use
the threaded resolves for the regular name resolves and c-ares for
HTTPSRR only. "c-ares-rr" means it does not use c-ares for "plain" name
resolves.
Viktor Szakats [Sat, 25 Jan 2025 13:54:47 +0000 (14:54 +0100)]
cmake: drop `CURL_USE_PKGCONFIG` from `curl-config.cmake.in`
This variable was meant to be used by curl Find modules, but it turns
out it makes no sense to use those from `curl-config.cmake.in`. It means
this variable was not used before and will not be used in the future,
and therefore safe to delete.
Also add missing macros passed to `curl-config.cmake` to comment.
Daniel Stenberg [Fri, 24 Jan 2025 12:19:30 +0000 (13:19 +0100)]
content_encoding: put the decomp buffers into the writer structs
- no more malloc/free per chunk
- removes the extra malloc entirely
- make the buffer (much) smaller (10MB => 16KB!)
- rename 'decomp' to 'buffer' to clarify purpose
Stefan Eissing [Thu, 23 Jan 2025 10:48:06 +0000 (11:48 +0100)]
lib: redirect handling by protocol handler
Adds a `follow()` callback to protocol handlers, so they may decide how
to act on a `newurl` after a request has been done. This is optional.
This moves the HTTP code for handling redirects from multi.c to http.c
where it should be. If we ever add a protocol with its own logic, it
would install its own follow function.
Stefan Eissing [Wed, 22 Jan 2025 13:45:30 +0000 (14:45 +0100)]
lib: clarify 'conn->httpversion'
The variable `conn->httpversion` was used for several purposes and it
was unclear at which time the value represents what.
- rename `conn->httpversion` to `conn->httpversion_seen`
This makes clear that the variable only records the last
HTTP version seen on the connection - if any. And that it
no longer is an indication of what version to use.
- Change Alt-Svc handling to no longer modify `conn->httpversion`
but set `data->state.httpwant` for influencing the HTTP version
to use on a transfer.
- Add `data->req.httpversion_sent` to have a record of what
HTTP version was sent in a request
- Add connection filter type CF_TYPE_HTTP
- Add filter query `CF_QUERY_HTTP_VERSION` to ask what HTTP
filter version is in place
- Lookup filters HTTP version instead of using `conn->httpversion`
Test test_12_05 now switches to HTTP/1.1 correctly and the
expectations have been fixed.
Removed the connection fitler "is_httpN()" checks and using
the version query instead.
Viktor Szakats [Tue, 21 Jan 2025 16:11:45 +0000 (17:11 +0100)]
src: omit hugehelp and ca-embed from libcurltool
CMake builds using the Xcode generator broke with an error saying it
doesn't support multiple targets depending on the same custom commands.
These custom commands are generating `tool_hugehelp.c` and
`tool_c_embed.c` for the curl tool and libcurltool.
`unit1394` and `unit1604` tests use libcurltool to test tool-specific
functions. They don't need hugehelp and ca-embed. It's thus safe to
disable and exclude them when compiling the sources for libcurltool.
Use the `UNITTESTS` macro to detect a libcurltool build within C.
After this patch these sources are solely used for building the curl
tool. Making the build compatible with the CMake Xcode generator.
Apply the change to autotools too to keep build systems synchronized.
Make transfer attach/detach to/from connections chepaer.
- the "attach" event was no longer implemented by any filter
- the "detach" did the same as the "done" event for the filters
who still implemented it. It should be superfluous as the "done"
must always happen.
Jay Satiro [Sun, 19 Jan 2025 06:18:23 +0000 (01:18 -0500)]
easy_lock: use Sleep(1) for thread yield on old Windows
- Prefer Sleep(1) over sched_yield() for pre-Vista thread yield.
On Windows sched_yield is often implemented as Sleep(0) which only
yields to threads of highest priority to current priority. However,
during libcurl initialization if there is thread contention then it's
possible that there is a wait for a different library or OS thread of
a lesser priority and then the yield is not effective during that time.
On the other hand Sleep(1) will wait the minimum time slice which is
usually like 15ms or more.
Prior to this change 2c4bfef removed sched_yield detection on Windows,
which effectively removed the yield in the spin lock, and therefore this
change restores the yield but in a different way.
For Windows Vista and later we use SRW locks and do not have this issue.
Viktor Szakats [Sat, 18 Jan 2025 01:11:37 +0000 (02:11 +0100)]
GHA: add iOS jobs with LibreSSL, enable dependencies for Android via vcpkg
iOS:
- add jobs with autotools, CMake, CMake Xcode generator.
The Xcode generator is >10x slower than Unix Makefiles. Keep it
because it's the one recommended by CMake and for having its own
quirks we may want to know about.
- build, cache and use LibreSSL for these jobs.
With workaround for an iOS build issue fixed in master.
- make Xcode generator work by explicitly disabling code signing.
- make tests and examples build with the Xcode generator by setting
`-DMACOSX_BUNDLE_GUI_IDENTIFIER=se.curl`, to avoid
"Bundle identifier is missing" errors.
- cmake: disable `CURL_USE_PKGCONFIG` by default for Apple device.
- cmake: add `stdc++` library for BoringSSL and AWS-LC, with
`OPENSSL_USE_STATIC_LIBS=ON` set.
- cmake: add workaround for Xcode generator issue, where it cannot
handle two targets depending on one custom command. A better fix may
be dropping `tool_hugehelp.c` and `tool_ca_embed.c` from curltool
library. For a future PR.
Android:
- add vcpkg to Android jobs, enable dependencies. Assisted-by: Tal Regev via #16045
- make vcpkg work with autotools.
- pass `--with-brotli` to autotools to detect the vcpkg-supplied brotli.
- enable BoringSSL for Android and add a job with it.
- silence 457 CMake configure warnings about the Android NDK CMake
scripts targeting freshly deprecated CMake versions.
These were much more involved than imagined. Basically nothing works out
of the box, and when combined, everything becomes a unique edge case.
autotools builds were a much easier to make work than CMake ones.
Also:
- GHA/non-native: re-sync names to be shorter and more aligned with
other workflows.
- GHA: add `persist-credentials: false` where missing.
Unresolved issues:
- `OPENSSL_ROOT_DIR` ignored/mis-used when pointing it to LibreSSL.
CMake seems to prepend the sysroot to the passed absolute directory.
Found no workaround.
- CMake when combined with Android, both the Google-recommended method
and the built-in CMake method fail to provide a way to avoid
`pkg-config` packages at system directories. Failed to find a knob
that can remove `/usr/include` from the search path. The workaround is
to disable zstd. (I enabled it by default in this release, maybe
premature?: f2adb3b6d73cad0c28ec8a32f5fa969d0f6378a0 #15431)
Disabling `pkg-config` doesn't work because vcpkg dependencies do not
link without it.
- CMake's Xcode generator is slow because each `try_compile()` feature
check springs a new CMake + Xcode project taking a long time to run,
just to compile single-liner C files. A known issue, with no solution.
`-DCMAKE_MACOSX_BUNDLE=OFF` did not help, limiting build types to
a single one (e.g. `Debug`) also had no effect.
make | Xcode | GHA run
:---- | :---- | :--------------------------------------------------------------------
16s | 2m57s | https://github.com/curl/curl/actions/runs/12866334102/job/35868712426
23s | 4m13s | https://github.com/curl/curl/actions/runs/12868128013/job/35874212461
16s | 3m39s | https://github.com/curl/curl/actions/runs/12859073531/job/35849041880
14s | 2m23s | https://github.com/curl/curl/actions/runs/12858298423/job/35847201313
15s | 2m36s | https://github.com/curl/curl/actions/runs/12858058492/job/35846669761
19s | 3m19s | https://github.com/curl/curl/actions/runs/12868919430/job/35876601168
Viktor Szakats [Fri, 17 Jan 2025 20:33:12 +0000 (21:33 +0100)]
windows: merge `config-win32ce.h` into `config-win32.h`
They were more or less the same, but each missed some things the other
had. Windows CE is a subset of Win32, make the headers reflect that and
avoid duplications.
Daniel Stenberg [Sun, 19 Jan 2025 11:35:39 +0000 (12:35 +0100)]
libcurl/opts: do not save files in dirs where attackers have access
libcurl cannot fully protect against attacks where an attacker has write
access to the same directory where it is directed to save files. This is
particularly sensitive if you save files using elevated privileges.
Previously only mentioned in VULN-DISCLOSURE-POLICY.md.
Viktor Szakats [Sat, 18 Jan 2025 01:30:08 +0000 (02:30 +0100)]
GHA: tidy up quotes, checkout order, silence Android/CMake warnings
- move checkout step right before use.
- quotes in yaml and bash.
- unfold single-line run commands.
- set `CMAKE_WARN_DEPRECATED=OFF` for CMake with Android NDK to avoid
a wall of useless deprecation warnings hiding useful output.
These warnings happen in 3rd-party scripts, and unfixable in curl.
Jay Satiro [Fri, 17 Jan 2025 08:17:15 +0000 (03:17 -0500)]
checksrc.bat: remove explicit SNPRINTF bypass
- Remove the command line argument passed to checksrc.pl that accepts
SNPRINTF violations in docs/examples.
This is a follow-up to c445b742 which introduced a different management
of banned functions and removed the SNPRINTF rule in favor of banning
snprintf in lib. There's no longer a SNPRINTF warning to suppress.
Viktor Szakats [Thu, 16 Jan 2025 19:39:13 +0000 (20:39 +0100)]
build: stop detecting `sched_yield()` on Windows
On Windows a successful `sched_yield()` detection requires mingw-w64
built with POSIX threads (not Win32 threads) and GCC (not llvm/clang).
(linking to `winpthread` via custom options may also work.)
In CMake builds, it was pre-cached as unavailable before this patch.
When detected (via autotools), it got only used for Windows XP or older
targets combined with a non-GCC, non-clang compiler that doesn't support
`__builtin_ia32_pause()`, or with the Intel C compiler. According to
`lib/easy_lock.h`.
mingw-w64 only supports GCC and clang, leaving a very narrow chance when
`shed_yield()` gets called on Windows. Even then, `sched_yield()` is
implemented in `winpthread` as `Sleep(0)`, which may or not be a useful.
It's also trivial to implement locally if it is, and such rare build
combination is also deemed useful.
Thus, this patch marks `sched_yields()` permanently unavailable on the
Windows platform also with autotools, and instead of pre-caching, skip
this feature check with CMake.
This syncs `HAVE_SCHED_YIELDS` between builds methods on Windows.
Viktor Szakats [Thu, 16 Jan 2025 14:21:23 +0000 (15:21 +0100)]
cmake: pre-fill `HAVE_STDATOMIC_H`, `HAVE_ATOMIC` for mingw-w64
`stdatomic.h` and `_Atomic` were first available in gcc 4.9.0 and
llvm/clang 3.6. Set detection values accordingly and save these two
detections on configure runs.
Stefan Eissing [Fri, 17 Jan 2025 10:57:00 +0000 (11:57 +0100)]
TLS: check connection for SSL use, not handler
Protocol handler option PROTOPT_SSL is used to setup a connection
filters. Once that is done, used `Curl_conn_is_ssl()` to check if
a connection uses SSL.
There may be other reasons to add SSL to a connection, e.g. starttls.
Viktor Szakats [Wed, 15 Jan 2025 14:46:32 +0000 (15:46 +0100)]
android: add CI jobs, buildinfo, cmake docs, disable `CURL_USE_PKGCONFIG` by default
- GHA/non-native: add Android builds, both cmake and autotools,
both NDK 21 (oldest available) and 35 (newest available)
https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md
It comes with a maintenance burden to bump the oldest/latest values
with CI runner updates.
- cmake: disable `CURL_USE_PKGCONFIG` by default for Android.
To avoid picking up system package by default.
- build: add `ANDROID-<NDK-LEVEL>` flag to `buildinfo.txt`.
Also detect NDK level with the CMake built-in build method:
https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html#cross-compiling-for-android
- INSTALL.md: add CMake build instructions for Android.
- INSTALL.md: make NDK levels consistent in `./configure` example.
Viktor Szakats [Thu, 16 Jan 2025 18:12:02 +0000 (19:12 +0100)]
GHA: start using ARM Linux runners
- GHA/linux: allow per-job runner image override.
- GHA/linux: add an arm version of an existing job.
Add workaround for broken `PATH` in the arm runner image.
- GHA/non-native: add CPU arch to job name where missing.
- GHA/checkdocs: switch a linter job to arm.
Performance looks a little bit better than Intel
(presumably with lower power consumption).
Viktor Szakats [Thu, 16 Jan 2025 21:19:51 +0000 (22:19 +0100)]
GHA/linux: enable wolfSSH in a wolfSSL job
With tests. (Most SFTP tests fail with wolfSSH.)
Also:
- disable pseudo-terminal in wolfSSH builds (also in CircleCI).
Not needed for curl.
- test642: add "compressed" to the description.
To avoid being the same as test600.
Follow-up to b7b4dc0d49543175ab0d9bb1cdc257a2d7f7cf0a #1735
Stefan Eissing [Thu, 2 Jan 2025 15:34:52 +0000 (16:34 +0100)]
websocket: fix message send corruption
- Fix a bug in EAGAIN handling when sending frames that led to a
corrupted last byte of the frame sent.
- Restore sanity to curl_ws_send() behaviour:
- Partial writes are reported as OK with the actual number of
payload bytes sent.
- CURLE_AGAIN is only returned when none of the payload bytes
(or for 0-length frames, not all of the frame header bytes)
could be sent.
- curl_ws_send() now behaves like a common send() call.
- Change 'ws-data' test client to allow concurrent send/recv
operations and vary frame sizes and repeat count.
- Add DEBUG env var CURL_WS_CHUNK_EAGAIN to simulate blocking
after a chunk of an encoded websocket frame has been sent.
- Add tests.
Prior to this change data corruption may occur when sending websocket
messages due to two bugs:
1) 3e64569a (precedes 8.10.0) caused a data corruption bug in the last
byte of frame of large messages.
2) curl_ws_send had non-traditional send behavior and could return
CURLE_AGAIN with bytes sent and expect the caller to adjust buffer
and buflen in a subsequent call. That behavior was not documented.
Viktor Szakats [Thu, 16 Jan 2025 15:47:15 +0000 (16:47 +0100)]
autotools: add support for mingw UWP builds
To match cmake builds.
- GHA/windows: allow autotools UWP builds.
- detect UWP and add to `buildinfo.txt`.
Consider it enabled if `CPPFLAGS` contains `-DWINSTORECOMPAT`.
- disable telnet with UWP.
- enable Unicode with UWP.
- do not use `wldap32` with UWP.
- do not enable `USE_WIN32_CRYPTO` with UWP.
- make sure to link to `ws2_32` in UWP builds.
To fix `undefined reference to `in6addr_any'` when linking
`tests/server` programs. More in the comment.
Stefan Eissing [Wed, 15 Jan 2025 15:45:25 +0000 (16:45 +0100)]
cf-https-connect: look into httpsrr alpns when available
Improved the filter implementation to be flexible in which order h3 and
h2/h1 are attempted. When HTTPSRR is enabled, look at the ALPNs it found
and use the order given for connecting in default setups.
- GHA/windows: drop the word "old" from standalone mingw-w64 jobs to not
conflate it with "old mingw" we no longer support (while also keeping
it short).
Jay Satiro [Mon, 13 Jan 2025 08:57:45 +0000 (03:57 -0500)]
transfer: fix CURLOPT_CURLU override logic
- Change setopt and pretransfer to always reset URL related variables
for a CURLU handle set CURLOPT_CURLU.
This change is to ensure we are in compliance with the doc which says
CURLU handles must be able to override a URL set via CURLOPT_URL and
that if the contents of the CURLU handle changes between transfers then
the updated contents must be used.
Prior to this change, although subsequent transfers appear to be
performed correctly in those cases, the work URL `data->state.url` was
not updated. CURLINFO_EFFECTIVE_URL returns data->state.url to the user
so it would return the URL from the initial transfer which was the wrong
URL. It's likely there are other cases as well.
Ref: https://curl.se/libcurl/c/CURLOPT_CURLU.html
Reported-by: Nicolás San Martín
Fixes https://github.com/curl/curl/issues/15984
Closes https://github.com/curl/curl/pull/15985
Jay Satiro [Tue, 14 Jan 2025 09:21:38 +0000 (04:21 -0500)]
mprintf: terminate snprintf output on windows
- Null terminate the end of the snprintf output buffer on Windows.
Old versions of the Windows CRT (which are often found on later versions
of Windows) do not terminate the snprintf output buffer if the output
reaches the max size.
This is a follow-up to parent 7e32f656 which made the same change but
limited it to mingw, however it is a CRT version issue irrespective of
compiler.
Daniel Stenberg [Mon, 13 Jan 2025 12:24:31 +0000 (13:24 +0100)]
mprintf: fix integer handling in float precision
In the double output function when an extremely large width and
precision is set that reaches the libcurl maximum (325), the handling of
the precision part would do wrong which could lead to bad output.
Also: work-around for single-byte buffer snprintf overflow with mingw.
Daniel Stenberg [Mon, 13 Jan 2025 11:16:19 +0000 (12:16 +0100)]
telnet: handle single-byte input option
Coverity CID 1638753 correctly identies this code misbehaved if the
passed in suboption is exactly one byte long by substracting two from
the unsigned size_t variable.
Daniel Stenberg [Sun, 12 Jan 2025 16:35:39 +0000 (17:35 +0100)]
multihandle: add an ssl_scache here
The TLS session cache is now held by the multi handle unless it is
shared, so that all easy handles within a multi handle get the benefit
of sharing the same, larger, cache.
The multi handle session cache size is set to 25, unless it is the
internal one used for the easy interface - which still uses only 3.
Viktor Szakats [Sun, 12 Jan 2025 10:57:24 +0000 (11:57 +0100)]
config: drop unused code and variables
- cmake, config-*: drop unused `PACKAGE*`, `VERSION` variables.
- config-win32: indentation
- config-win32ce: drop mingw-specific code.
This header is not used with MinGW.
- config-win32ce: `_WIN64` is never true for Windows CE, drop.
Viktor Szakats [Sun, 12 Jan 2025 02:25:36 +0000 (03:25 +0100)]
cmake: drop VS2010 "Dialog Hell" workaround added in 2013
Delete the workaround added via a94a68a3c1d04ccb53e46baa69753bbf6354ee14
(2013-02-04). The commit message has no details. The comment mentions
"Dialog Hell", and seems to fix CMake missing to regenerate `CURL.sln`
with VS2010. It also added a FIXME saying the workaround can be deleted
with future versions of CMake.
At the time CMake's latest version was v2.8.10.
curl now requires v3.7 (2018) minimum, and v3.24 (2022) was the
latest CMake natively supporting VS2010. Assume this has since been
fixed.
Also: format an MSVC version reference in comment.
Viktor Szakats [Sat, 11 Jan 2025 23:55:32 +0000 (00:55 +0100)]
configure: streamline Windows large file feature check
Before this patch the `CURL_CHECK_WIN32_LARGEFILE` feature check was
running an `AC_COMPILE` snippet that always succeeded. (except for
Windows CE, which isn't supported in other parts of `./configure` yet.)
The only Windows toolchain autotools supports is mingw. Of them, curl
only supports mingw-w64. All mingw-w64 versions support large files.
This allows to drop the check and assume it supported on Windows. To not
lose Windows CE support, rework that too, without using `AC_COMPILE`.
Drop the feature check altogether for non-Windows targets.
Viktor Szakats [Fri, 10 Jan 2025 15:58:48 +0000 (16:58 +0100)]
curl_setup: fix missing `ADDRESS_FAMILY` type in rare build cases
Build failed when both `ADDRESS_FAMILY` and `sockaddr_un` stuct were
missing from the Windows SDK, with UnixSockets enabled.
Seen with GNU 4.4.0 in CeGCC 0.59.1:
```
lib/curl_setup.h:983: error: expected specifier-qualifier-list before 'ADDRESS_FAMILY'
lib/curl_setup.h:985: warning: struct has no members
```
Also reported with VS2003:
https://datagirl.xyz/posts/wolfssl_curl_w2k.html
Viktor Szakats [Sat, 11 Jan 2025 02:22:10 +0000 (03:22 +0100)]
windows: drop redundant `USE_WIN32_SMALL_FILES` macro
In effect it meant `_WIN32 && !USE_WIN32_LARGE_FILES`.
Replace it with these macros.
Also:
- configure: delete tautological check for small file support.
- configure: delete stray `_MSC_VER` reference. autotools does not
support MSVC.
- drop tautological checks for WinCE in `config-win32*.h` when setting
`USE_WIN32_LARGE_FILES`.
- merge related PP logic.
- prefer `#ifdef`, fix whitespace.
- drop compatibility error for `CURL_WANTS_CA_BUNDLE_ENV`.
This macro was once set by `Makefile.mk` and Watcom makefiles.
They are no longer supported, making the compatibility message moot.
Viktor Szakats [Thu, 9 Jan 2025 17:19:35 +0000 (18:19 +0100)]
msvc: tidy up `_CRT_*_NO_DEPRECATE` definitions
Dedupe and migrate MSVC-specific warning suppressions to `curl_setup.h`.
Make cmake set `_CRT_SECURE_NO_DEPRECATE` for examples and standalone
tests, and stop setting `_CRT_NONSTDC_NO_DEPRECATE` for them.
Details:
- drop version guards. On ancient MSVC version these macro are a no-op.
- move to `curl_setup.h` from `config-win32*.h`.
- sync macro values with CMake.
- cmake: stop setting them globally in favour of `curl_setup.h`.
- cmake: re-add these macros to `docs/examples` and `tests/http/clients`,
which do not use `curl_setup.h`.
- cmake: drop `_CRT_NONSTDC_NO_DEPRECATE` for examples and tests.
They build fine without.
- update comments.
projects / thirdparty / curl.git / log
