Viktor Szakats [Thu, 6 Feb 2025 13:09:58 +0000 (14:09 +0100)]
GHA/windows: always pass `-A <arch>` to cmake in vcpkg jobs
Instead of relying on the default `-A x64` on `windows-latest` runners,
tell cmake the arch explicitly, to be in sync with `matrix.arch`. Also
add support for arm64 and x86.
`-DVCPKG_TARGET_TRIPLET=` isn't enough to select the platform, ref:
https://github.com/curl/curl/actions/runs/13179082565/job/36785363766?pr=16210
Daniel Stenberg [Thu, 6 Feb 2025 08:28:42 +0000 (09:28 +0100)]
content_encoding: #error on too old zlib
The previous runtime check using strcmp() risks failing when zlib
reaches 1.10. While this instead changes the logic to a cruder
build-time instead of runtime, it avoids the 1.10 risk.
I verified that ZLIB_VERNUM has been provided since at least the 1.2.0.3
release.
Jay Satiro [Sat, 1 Feb 2025 23:12:18 +0000 (18:12 -0500)]
docs: better explain multi-part byte range behavior
- Better explain that if the requested range (--range or CURLOPT_RANGE)
contains multiple ranges then the response contains meta information
in addition to the requested bytes.
Prior to this change it was noted that a multiple part response was
returned as-is but not what that meant. In particular, meta information
is returned in addition to the requested bytes and that may have been
unexpected.
Reported-by: Ralf A. Timmermann
Fixes https://github.com/curl/curl/issues/16139
Closes https://github.com/curl/curl/pull/16150
Viktor Szakats [Wed, 5 Feb 2025 18:19:04 +0000 (19:19 +0100)]
libssh: silence `-Wconversion` with a cast (Windows 32-bit)
Seen with GCC 13 with Windows x86:
```
lib/vssh/libssh.c: In function 'myssh_statemach_act':
lib/vssh/libssh.c:1851:41: error: conversion from 'curl_off_t' {aka 'long long int'} to 'size_t' {aka 'unsigned int'} may change value [-Werror=conversion]
1851 | data->state.infilesize,
| ~~~~~~~~~~~^~~~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/13161422041/job/36737994642?pr=16182#step:3:5111
Viktor Szakats [Wed, 5 Feb 2025 12:52:06 +0000 (13:52 +0100)]
smb: silence `-Warray-bounds` with gcc 13+
The code look correct. The compiler gets confused by the `byte[1]`
struct member mapped into a memory buffer with a variable-sized
payload starting at this member. Perhaps there is a cleaner way
to silence this by changing the code.
First seen with gcc 13.2.0 in curl-for-win builds. Then with 13.2.1 and
the latest 14.2.0.
```
curl/lib/smb.c: In function 'smb_connection_state':
curl/lib/smb.c:895:5: warning: 'memcpy' offset [74, 80] from the object at 'buf' is out of the bounds of referenced subobject 'bytes' with type 'char[1]' at offset 73 [-Warray-bounds=]
895 | memcpy(smbc->challenge, nrsp->bytes, sizeof(smbc->challenge));
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
curl/lib/smb.c:130:8: note: subobject 'bytes' declared here
130 | char bytes[1];
| ^~~~~
```
Daniel Stenberg [Tue, 4 Feb 2025 16:51:57 +0000 (17:51 +0100)]
asyn-thread: fix HTTPS RR crash
By removing 'data' from the thread struct and passing it in as an
argument we avoid the case it could be dereferenced before stored when
shutting down HTTPS RR.
Also reordered the struct fields a little to remove holes.
Viktor Szakats [Tue, 4 Feb 2025 17:29:24 +0000 (18:29 +0100)]
cfilters: silence compiler warning
seen with gcc 4.4.0:
```
../../lib/cfilters.c: In function 'Curl_conn_http_version':
../../lib/cfilters.c:523: error: conversion to 'unsigned char' from 'int' may alter its value
```
Ref: https://github.com/curl/curl/actions/runs/13124120573/job/36616761121?pr=15975#step:9:20
Viktor Szakats [Tue, 4 Feb 2025 17:15:22 +0000 (18:15 +0100)]
transfer: fix returning init failures from `xfer_recv_shutdown_started()`
Before this patch it returned `CURLE_FAILED_INIT` on init failures, with
the value of 2. Fix it to return `false`.
Seen with clang 18.1.8:
```
../lib/transfer.c(181,12): warning: integer constant not in range of enumerated type 'bool' [-Wassign-enum]
181 | return CURLE_FAILED_INIT;
| ^
../lib/transfer.c(181,12): warning: implicit conversion from enumeration type 'CURLcode' to different enumeration type 'bool' [-Wenum-conversion]
181 | return CURLE_FAILED_INIT;
| ~~~~~~ ^~~~~~~~~~~~~~~~~
../lib/transfer.c(183,12): warning: integer constant not in range of enumerated type 'bool' [-Wassign-enum]
183 | return CURLE_FAILED_INIT;
| ^
../lib/transfer.c(183,12): warning: implicit conversion from enumeration type 'CURLcode' to different enumeration type 'bool' [-Wenum-conversion]
183 | return CURLE_FAILED_INIT;
| ~~~~~~ ^~~~~~~~~~~~~~~~~
```
Stefan Eissing [Tue, 4 Feb 2025 20:53:55 +0000 (21:53 +0100)]
pop3: revert connection ssl check
As reported in #16166, the STLS hangs with the check for SSL connection
filters, but is working with the old protocol handler way. Revert the
change, although it is unclear why it was no good here.
Fixes #16166 Reported-by: ralfjunker on github
Closes #16172
Stefan Eissing [Fri, 31 Jan 2025 12:13:34 +0000 (13:13 +0100)]
x509asn1: add parse recursion limit
For ASN.1 tags with indefinite length, curl's own parser for TLS
backends that do not support certificate inspection calls itself
recursively. A malicious server certificate can then lead to high
recursion level exhausting the stack space.
This PR limits the recursion level to 16 which should be safe on all
architectures.
mauke [Thu, 30 Jan 2025 05:28:50 +0000 (06:28 +0100)]
runtests.pl: fix precedence issue
The condition `!$cmdtype eq "perl"` (introduced in a4765b0551) is always
false. It checks whether a logical negation (giving true/false) is equal
to the string `"perl"`. This is impossible, so the logging never worked.
The intent was probably to negate the result of the string
comparison:`!($cmdtype eq "perl")` or simply `$cmdtype ne "perl"`.
Fixes #16128 Reported-by: Igor Todorovski
Closes #16129
- dedupe `CARES_STATICLIB` initalizations into `curl_setup.h`, to
ensure it's defined before the first (and every) `ares.h` include and
avoid a potential confusion.
- move `CARES_NO_DEPRECATED` from build level to `curl_setup.h`.
To work regardless of build system.
It is necessary because curl calls `ares_getsock()` from two places,
of which one feeds a chain of wrappers: `Curl_ares_getsock()`,
`Curl_resolver_getsock()`, `Curl_resolv_getsock()`.
Viktor Szakats [Wed, 29 Jan 2025 18:46:25 +0000 (19:46 +0100)]
GHA: tidy up `apt` commands
- drop `--quiet 2` option where used, to have uniform output.
- replace `apt` with `apt-get` in one job. sync options with rest.
- replace deprecated `apt-key` command with the alternative recommended
by `apt-key(8)`.
- drop stray `cd /tmp`, no longer needed after migrating to GHA.
- shorten `--option Dpkg::Use-Pty=0` to `-o Dpkg::Use-Pty=0`.
- add `-o Dpkg::Use-Pty=0` to hide `apt-get` progress bars taking
vertical log space, where missing.
- drop `-y --no-install-suggests --no-install-recommends` `apt-get`
options. They are the default in the ubuntu-24.04 image.
- GHA/distcheck: move `name:` to top in steps where not there.
- scripts/cijobs.pl: catch `apt-get` lines with the `-o` option.
Viktor Szakats [Mon, 27 Jan 2025 22:04:02 +0000 (23:04 +0100)]
openssl: define `HAVE_KEYLOG_CALLBACK` before use
Before this patch this macro was used in `vtls/openssl.h` without
setting it first, causing the `keylog_done` member be present in
struct `ossl_ctx` while the code did not use it.
Andrew Kaster [Tue, 21 Jan 2025 16:57:46 +0000 (09:57 -0700)]
ws: Reject frames with unknown reserved bits set
RFC 6455 Section 5.2 notes that for bits RSV1, RSV2, and RSV3 of the
framing header, a non-zero value that is not defined by a negotiated
extension MUST Fail the WebSocket connection.
Jay Satiro [Tue, 28 Jan 2025 04:48:18 +0000 (23:48 -0500)]
vtls: fix default SSL backend as a fallback
- Use build-time CURL_DEFAULT_SSL_BACKEND as a fallback when environment
variable CURL_SSL_BACKEND contains a backend that is unavailable.
Prior to this change if CURL_SSL_BACKEND was set then
CURL_DEFAULT_SSL_BACKEND was ignored even if the backend of the former
was unavailable. In that case libcurl would instead select the first
available backend in the list of backends.
Jay Satiro [Wed, 15 Jan 2025 08:56:11 +0000 (03:56 -0500)]
easy: allow connect-only handle reuse with easy_perform
- Detach and disconnect an attached connection before performing.
Prior to this change it was not possible to safely reuse an easy handle
with an attached connection in a second call to curl_easy_perform. The
only known case of this is a connect-only type handle where the
connection was detached when curl_easy_perform returned, only to be
reattached by either curl_easy_send/recv.
This commit effectively reverts 2f8ecd5d and be82a360, the latter of
which treated the reuse as an error. Prior to that change undefined
behavior may occur in such a case.
Bug: https://curl.se/mail/lib-2025-01/0044.html Reported-by: Aleksander Mazur
Closes https://github.com/curl/curl/pull/16008
Viktor Szakats [Mon, 27 Jan 2025 18:32:45 +0000 (19:32 +0100)]
checksrc: exclude generated bundle files to avoid race condition
Necessary to catch rare cases when `checksrc` hits these files when they
are not populated yet:
```
./curltool_unity.c:1:1: error: Missing copyright statement (COPYRIGHT)
^
```
https://github.com/curl/curl/actions/runs/12995546740/job/36242556713?pr=16094#step:37:123
Viktor Szakats [Sun, 26 Jan 2025 15:13:16 +0000 (16:13 +0100)]
Makefile.dist: delete
It had shorthand aliases to launch `./configure` and
`./configure --with-openssl`. The former hasn't worked for a long while
because of missing TLS.
Its `ca-bundle` and `ca-firefox` targets have been broken for 2.5 years
till recently. These targets also exist in `./configure` and have been
working all along.
Also:
- cmake: add support `curl-ca-bundle` and `curl-ca-firefox` targets.
- tests/testcurl.pl: drop obsolete build logic.
Daniel Stenberg [Mon, 27 Jan 2025 08:34:57 +0000 (09:34 +0100)]
urldata: tweak the UserDefined struct
By better sticking to listing the struct members sorted by size, this
struct is now 48 bytes smaller on my fairly maximized build, without
removing anything.
Turned 'connect_only' into two bits instead of an unsigned char with two
magic values.
Also put the 'gssapi_delegation' field within ifdef HAVE_GSSAPI.
Stefan Eissing [Mon, 27 Jan 2025 10:36:22 +0000 (11:36 +0100)]
http2: fix data_pending check
The h2 filter mistakenly also checked `sendbuf` when asked
about pending data. The call is only meant to account for
buffered data that still needs to be received.
Also, remove obsolete recvbuf in stream as we write received
headers and data directly.
Fixes #16084
Closes #16098 Reported-by: Deniz Sökmen
Viktor Szakats [Fri, 24 Jan 2025 13:22:23 +0000 (14:22 +0100)]
build: drop `tool_hugehelp.c.cvs`, tidy up macros, drop `buildconf.bat`
Rework the way `tool_hugehelp.c` is included in builds.
After this patch, with `./configure` and CMake `tool_hugehelp.c` is only
compiled when building with manuals enabled. With manuals disabled this
source file is not used anymore. The method is similar to how 8a3740bc8e558b9a9d4a652b74cf27a0961d7010 implemented `tool_ca_embed.c`.
`./configure` always generates it as before, otherwise the build fails.
- winbuild: rework to not need `buildconf.bat`, but automatically use
`tool_hugehelp.c` if present (e.g. when building from an official
source tarball) and enable `USE_MANUAL` accordingly.
- `buildconf.bat`: after dropping `tool_hugehelp.c` generation, the only
logic left was `cp Makefile.dist Makefile`. This allowed to launch
winbuild builds via GNU Make in a Git repo. Drop this option together
with the batch file.
- build `libcurltool` without `USE_MANUAL` macro to exclude the manual
and the dependence on the generator commands. Drop relying on
`UNITTESTS` for this purpose.
Follow-up to 96843f4ef74e02452972fd97fe15d8ff656f46ec #16068
- `src/mkhelp.pl`: include `tool_hugehelp.h` before using `USE_MANUAL`
to have it set in `config-*.h` builds with source tarballs created
with manual but without zlib.
Jay Satiro [Sat, 25 Jan 2025 08:17:10 +0000 (03:17 -0500)]
tests: change the behavior of swsbounce
- Change the swsbounce keyword to override the part number on a
subsequent request to the previous part number + 1.
Note the previous part number in this case is the part number that
was returned as a response to the previous request and contained
the swsbounce keyword.
Prior to this change swsbounce incremented the part number of the
subsequent request instead of overriding it, and did so in a more
limited fashion that prevented chaining swsbounce in multiple responses.
For example, if the test makes a request that causes the sws server to
return `<data>` as a response and that response contains `swsbounce`
then for the next response the sws server returns `<data1>`. If
`<data1>` also contains `swsbounce` then for the next response the sws
server now returns `<data2>` instead of the requested part.
Daniel Stenberg [Tue, 21 Jan 2025 10:42:20 +0000 (11:42 +0100)]
asyn-thread: use c-ares to resolve HTTPS RR
Allow building with c-ares and yet use threaded resolver for the main
host A/AAAA resolving:
`--with-ares` provides the c-ares install path and defaults to use
c-ares for name resolving
`--with-threaded-resolver` still uses c-ares in the build (for HTTPS)
but uses the threaded resolver for "normal" resolves.
It works similarly for cmake: ENABLE_ARES enables ares, and if
ENABLE_THREADED_RESOLVER also is set, c-ares is used for HTTPS RR and
the threaded resolver for "normal" resolves.
HTTPSRR and c-ares-rr are new features return by curl_version_info() and
thus shown by curl -V.
The c-ares-rr feature bit is there to make it possible to distinguish
between builds using c-ares for all name resolves and builds that use
the threaded resolves for the regular name resolves and c-ares for
HTTPSRR only. "c-ares-rr" means it does not use c-ares for "plain" name
resolves.
Viktor Szakats [Sat, 25 Jan 2025 13:54:47 +0000 (14:54 +0100)]
cmake: drop `CURL_USE_PKGCONFIG` from `curl-config.cmake.in`
This variable was meant to be used by curl Find modules, but it turns
out it makes no sense to use those from `curl-config.cmake.in`. It means
this variable was not used before and will not be used in the future,
and therefore safe to delete.
Also add missing macros passed to `curl-config.cmake` to comment.
Daniel Stenberg [Fri, 24 Jan 2025 12:19:30 +0000 (13:19 +0100)]
content_encoding: put the decomp buffers into the writer structs
- no more malloc/free per chunk
- removes the extra malloc entirely
- make the buffer (much) smaller (10MB => 16KB!)
- rename 'decomp' to 'buffer' to clarify purpose
Stefan Eissing [Thu, 23 Jan 2025 10:48:06 +0000 (11:48 +0100)]
lib: redirect handling by protocol handler
Adds a `follow()` callback to protocol handlers, so they may decide how
to act on a `newurl` after a request has been done. This is optional.
This moves the HTTP code for handling redirects from multi.c to http.c
where it should be. If we ever add a protocol with its own logic, it
would install its own follow function.
Stefan Eissing [Wed, 22 Jan 2025 13:45:30 +0000 (14:45 +0100)]
lib: clarify 'conn->httpversion'
The variable `conn->httpversion` was used for several purposes and it
was unclear at which time the value represents what.
- rename `conn->httpversion` to `conn->httpversion_seen`
This makes clear that the variable only records the last
HTTP version seen on the connection - if any. And that it
no longer is an indication of what version to use.
- Change Alt-Svc handling to no longer modify `conn->httpversion`
but set `data->state.httpwant` for influencing the HTTP version
to use on a transfer.
- Add `data->req.httpversion_sent` to have a record of what
HTTP version was sent in a request
- Add connection filter type CF_TYPE_HTTP
- Add filter query `CF_QUERY_HTTP_VERSION` to ask what HTTP
filter version is in place
- Lookup filters HTTP version instead of using `conn->httpversion`
Test test_12_05 now switches to HTTP/1.1 correctly and the
expectations have been fixed.
Removed the connection fitler "is_httpN()" checks and using
the version query instead.
Viktor Szakats [Tue, 21 Jan 2025 16:11:45 +0000 (17:11 +0100)]
src: omit hugehelp and ca-embed from libcurltool
CMake builds using the Xcode generator broke with an error saying it
doesn't support multiple targets depending on the same custom commands.
These custom commands are generating `tool_hugehelp.c` and
`tool_c_embed.c` for the curl tool and libcurltool.
`unit1394` and `unit1604` tests use libcurltool to test tool-specific
functions. They don't need hugehelp and ca-embed. It's thus safe to
disable and exclude them when compiling the sources for libcurltool.
Use the `UNITTESTS` macro to detect a libcurltool build within C.
After this patch these sources are solely used for building the curl
tool. Making the build compatible with the CMake Xcode generator.
Apply the change to autotools too to keep build systems synchronized.
Make transfer attach/detach to/from connections chepaer.
- the "attach" event was no longer implemented by any filter
- the "detach" did the same as the "done" event for the filters
who still implemented it. It should be superfluous as the "done"
must always happen.
Jay Satiro [Sun, 19 Jan 2025 06:18:23 +0000 (01:18 -0500)]
easy_lock: use Sleep(1) for thread yield on old Windows
- Prefer Sleep(1) over sched_yield() for pre-Vista thread yield.
On Windows sched_yield is often implemented as Sleep(0) which only
yields to threads of highest priority to current priority. However,
during libcurl initialization if there is thread contention then it's
possible that there is a wait for a different library or OS thread of
a lesser priority and then the yield is not effective during that time.
On the other hand Sleep(1) will wait the minimum time slice which is
usually like 15ms or more.
Prior to this change 2c4bfef removed sched_yield detection on Windows,
which effectively removed the yield in the spin lock, and therefore this
change restores the yield but in a different way.
For Windows Vista and later we use SRW locks and do not have this issue.
Viktor Szakats [Sat, 18 Jan 2025 01:11:37 +0000 (02:11 +0100)]
GHA: add iOS jobs with LibreSSL, enable dependencies for Android via vcpkg
iOS:
- add jobs with autotools, CMake, CMake Xcode generator.
The Xcode generator is >10x slower than Unix Makefiles. Keep it
because it's the one recommended by CMake and for having its own
quirks we may want to know about.
- build, cache and use LibreSSL for these jobs.
With workaround for an iOS build issue fixed in master.
- make Xcode generator work by explicitly disabling code signing.
- make tests and examples build with the Xcode generator by setting
`-DMACOSX_BUNDLE_GUI_IDENTIFIER=se.curl`, to avoid
"Bundle identifier is missing" errors.
- cmake: disable `CURL_USE_PKGCONFIG` by default for Apple device.
- cmake: add `stdc++` library for BoringSSL and AWS-LC, with
`OPENSSL_USE_STATIC_LIBS=ON` set.
- cmake: add workaround for Xcode generator issue, where it cannot
handle two targets depending on one custom command. A better fix may
be dropping `tool_hugehelp.c` and `tool_ca_embed.c` from curltool
library. For a future PR.
Android:
- add vcpkg to Android jobs, enable dependencies. Assisted-by: Tal Regev via #16045
- make vcpkg work with autotools.
- pass `--with-brotli` to autotools to detect the vcpkg-supplied brotli.
- enable BoringSSL for Android and add a job with it.
- silence 457 CMake configure warnings about the Android NDK CMake
scripts targeting freshly deprecated CMake versions.
These were much more involved than imagined. Basically nothing works out
of the box, and when combined, everything becomes a unique edge case.
autotools builds were a much easier to make work than CMake ones.
Also:
- GHA/non-native: re-sync names to be shorter and more aligned with
other workflows.
- GHA: add `persist-credentials: false` where missing.
Unresolved issues:
- `OPENSSL_ROOT_DIR` ignored/mis-used when pointing it to LibreSSL.
CMake seems to prepend the sysroot to the passed absolute directory.
Found no workaround.
- CMake when combined with Android, both the Google-recommended method
and the built-in CMake method fail to provide a way to avoid
`pkg-config` packages at system directories. Failed to find a knob
that can remove `/usr/include` from the search path. The workaround is
to disable zstd. (I enabled it by default in this release, maybe
premature?: f2adb3b6d73cad0c28ec8a32f5fa969d0f6378a0 #15431)
Disabling `pkg-config` doesn't work because vcpkg dependencies do not
link without it.
- CMake's Xcode generator is slow because each `try_compile()` feature
check springs a new CMake + Xcode project taking a long time to run,
just to compile single-liner C files. A known issue, with no solution.
`-DCMAKE_MACOSX_BUNDLE=OFF` did not help, limiting build types to
a single one (e.g. `Debug`) also had no effect.
make | Xcode | GHA run
:---- | :---- | :--------------------------------------------------------------------
16s | 2m57s | https://github.com/curl/curl/actions/runs/12866334102/job/35868712426
23s | 4m13s | https://github.com/curl/curl/actions/runs/12868128013/job/35874212461
16s | 3m39s | https://github.com/curl/curl/actions/runs/12859073531/job/35849041880
14s | 2m23s | https://github.com/curl/curl/actions/runs/12858298423/job/35847201313
15s | 2m36s | https://github.com/curl/curl/actions/runs/12858058492/job/35846669761
19s | 3m19s | https://github.com/curl/curl/actions/runs/12868919430/job/35876601168
Viktor Szakats [Fri, 17 Jan 2025 20:33:12 +0000 (21:33 +0100)]
windows: merge `config-win32ce.h` into `config-win32.h`
They were more or less the same, but each missed some things the other
had. Windows CE is a subset of Win32, make the headers reflect that and
avoid duplications.
Daniel Stenberg [Sun, 19 Jan 2025 11:35:39 +0000 (12:35 +0100)]
libcurl/opts: do not save files in dirs where attackers have access
libcurl cannot fully protect against attacks where an attacker has write
access to the same directory where it is directed to save files. This is
particularly sensitive if you save files using elevated privileges.
Previously only mentioned in VULN-DISCLOSURE-POLICY.md.