Stefan Eissing [Tue, 4 Feb 2025 20:53:55 +0000 (21:53 +0100)]
pop3: revert connection ssl check
As reported in #16166, the STLS hangs with the check for SSL connection
filters, but is working with the old protocol handler way. Revert the
change, although it is unclear why it was no good here.
Fixes #16166 Reported-by: ralfjunker on github
Closes #16172
Stefan Eissing [Fri, 31 Jan 2025 12:13:34 +0000 (13:13 +0100)]
x509asn1: add parse recursion limit
For ASN.1 tags with indefinite length, curl's own parser for TLS
backends that do not support certificate inspection calls itself
recursively. A malicious server certificate can then lead to high
recursion level exhausting the stack space.
This PR limits the recursion level to 16 which should be safe on all
architectures.
mauke [Thu, 30 Jan 2025 05:28:50 +0000 (06:28 +0100)]
runtests.pl: fix precedence issue
The condition `!$cmdtype eq "perl"` (introduced in a4765b0551) is always
false. It checks whether a logical negation (giving true/false) is equal
to the string `"perl"`. This is impossible, so the logging never worked.
The intent was probably to negate the result of the string
comparison:`!($cmdtype eq "perl")` or simply `$cmdtype ne "perl"`.
Fixes #16128 Reported-by: Igor Todorovski
Closes #16129
- dedupe `CARES_STATICLIB` initalizations into `curl_setup.h`, to
ensure it's defined before the first (and every) `ares.h` include and
avoid a potential confusion.
- move `CARES_NO_DEPRECATED` from build level to `curl_setup.h`.
To work regardless of build system.
It is necessary because curl calls `ares_getsock()` from two places,
of which one feeds a chain of wrappers: `Curl_ares_getsock()`,
`Curl_resolver_getsock()`, `Curl_resolv_getsock()`.
Viktor Szakats [Wed, 29 Jan 2025 18:46:25 +0000 (19:46 +0100)]
GHA: tidy up `apt` commands
- drop `--quiet 2` option where used, to have uniform output.
- replace `apt` with `apt-get` in one job. sync options with rest.
- replace deprecated `apt-key` command with the alternative recommended
by `apt-key(8)`.
- drop stray `cd /tmp`, no longer needed after migrating to GHA.
- shorten `--option Dpkg::Use-Pty=0` to `-o Dpkg::Use-Pty=0`.
- add `-o Dpkg::Use-Pty=0` to hide `apt-get` progress bars taking
vertical log space, where missing.
- drop `-y --no-install-suggests --no-install-recommends` `apt-get`
options. They are the default in the ubuntu-24.04 image.
- GHA/distcheck: move `name:` to top in steps where not there.
- scripts/cijobs.pl: catch `apt-get` lines with the `-o` option.
Viktor Szakats [Mon, 27 Jan 2025 22:04:02 +0000 (23:04 +0100)]
openssl: define `HAVE_KEYLOG_CALLBACK` before use
Before this patch this macro was used in `vtls/openssl.h` without
setting it first, causing the `keylog_done` member be present in
struct `ossl_ctx` while the code did not use it.
Andrew Kaster [Tue, 21 Jan 2025 16:57:46 +0000 (09:57 -0700)]
ws: Reject frames with unknown reserved bits set
RFC 6455 Section 5.2 notes that for bits RSV1, RSV2, and RSV3 of the
framing header, a non-zero value that is not defined by a negotiated
extension MUST Fail the WebSocket connection.
Jay Satiro [Tue, 28 Jan 2025 04:48:18 +0000 (23:48 -0500)]
vtls: fix default SSL backend as a fallback
- Use build-time CURL_DEFAULT_SSL_BACKEND as a fallback when environment
variable CURL_SSL_BACKEND contains a backend that is unavailable.
Prior to this change if CURL_SSL_BACKEND was set then
CURL_DEFAULT_SSL_BACKEND was ignored even if the backend of the former
was unavailable. In that case libcurl would instead select the first
available backend in the list of backends.
Jay Satiro [Wed, 15 Jan 2025 08:56:11 +0000 (03:56 -0500)]
easy: allow connect-only handle reuse with easy_perform
- Detach and disconnect an attached connection before performing.
Prior to this change it was not possible to safely reuse an easy handle
with an attached connection in a second call to curl_easy_perform. The
only known case of this is a connect-only type handle where the
connection was detached when curl_easy_perform returned, only to be
reattached by either curl_easy_send/recv.
This commit effectively reverts 2f8ecd5d and be82a360, the latter of
which treated the reuse as an error. Prior to that change undefined
behavior may occur in such a case.
Bug: https://curl.se/mail/lib-2025-01/0044.html Reported-by: Aleksander Mazur
Closes https://github.com/curl/curl/pull/16008
Viktor Szakats [Mon, 27 Jan 2025 18:32:45 +0000 (19:32 +0100)]
checksrc: exclude generated bundle files to avoid race condition
Necessary to catch rare cases when `checksrc` hits these files when they
are not populated yet:
```
./curltool_unity.c:1:1: error: Missing copyright statement (COPYRIGHT)
^
```
https://github.com/curl/curl/actions/runs/12995546740/job/36242556713?pr=16094#step:37:123
Viktor Szakats [Sun, 26 Jan 2025 15:13:16 +0000 (16:13 +0100)]
Makefile.dist: delete
It had shorthand aliases to launch `./configure` and
`./configure --with-openssl`. The former hasn't worked for a long while
because of missing TLS.
Its `ca-bundle` and `ca-firefox` targets have been broken for 2.5 years
till recently. These targets also exist in `./configure` and have been
working all along.
Also:
- cmake: add support `curl-ca-bundle` and `curl-ca-firefox` targets.
- tests/testcurl.pl: drop obsolete build logic.
Daniel Stenberg [Mon, 27 Jan 2025 08:34:57 +0000 (09:34 +0100)]
urldata: tweak the UserDefined struct
By better sticking to listing the struct members sorted by size, this
struct is now 48 bytes smaller on my fairly maximized build, without
removing anything.
Turned 'connect_only' into two bits instead of an unsigned char with two
magic values.
Also put the 'gssapi_delegation' field within ifdef HAVE_GSSAPI.
Stefan Eissing [Mon, 27 Jan 2025 10:36:22 +0000 (11:36 +0100)]
http2: fix data_pending check
The h2 filter mistakenly also checked `sendbuf` when asked
about pending data. The call is only meant to account for
buffered data that still needs to be received.
Also, remove obsolete recvbuf in stream as we write received
headers and data directly.
Fixes #16084
Closes #16098 Reported-by: Deniz Sökmen
Viktor Szakats [Fri, 24 Jan 2025 13:22:23 +0000 (14:22 +0100)]
build: drop `tool_hugehelp.c.cvs`, tidy up macros, drop `buildconf.bat`
Rework the way `tool_hugehelp.c` is included in builds.
After this patch, with `./configure` and CMake `tool_hugehelp.c` is only
compiled when building with manuals enabled. With manuals disabled this
source file is not used anymore. The method is similar to how 8a3740bc8e558b9a9d4a652b74cf27a0961d7010 implemented `tool_ca_embed.c`.
`./configure` always generates it as before, otherwise the build fails.
- winbuild: rework to not need `buildconf.bat`, but automatically use
`tool_hugehelp.c` if present (e.g. when building from an official
source tarball) and enable `USE_MANUAL` accordingly.
- `buildconf.bat`: after dropping `tool_hugehelp.c` generation, the only
logic left was `cp Makefile.dist Makefile`. This allowed to launch
winbuild builds via GNU Make in a Git repo. Drop this option together
with the batch file.
- build `libcurltool` without `USE_MANUAL` macro to exclude the manual
and the dependence on the generator commands. Drop relying on
`UNITTESTS` for this purpose.
Follow-up to 96843f4ef74e02452972fd97fe15d8ff656f46ec #16068
- `src/mkhelp.pl`: include `tool_hugehelp.h` before using `USE_MANUAL`
to have it set in `config-*.h` builds with source tarballs created
with manual but without zlib.
Jay Satiro [Sat, 25 Jan 2025 08:17:10 +0000 (03:17 -0500)]
tests: change the behavior of swsbounce
- Change the swsbounce keyword to override the part number on a
subsequent request to the previous part number + 1.
Note the previous part number in this case is the part number that
was returned as a response to the previous request and contained
the swsbounce keyword.
Prior to this change swsbounce incremented the part number of the
subsequent request instead of overriding it, and did so in a more
limited fashion that prevented chaining swsbounce in multiple responses.
For example, if the test makes a request that causes the sws server to
return `<data>` as a response and that response contains `swsbounce`
then for the next response the sws server returns `<data1>`. If
`<data1>` also contains `swsbounce` then for the next response the sws
server now returns `<data2>` instead of the requested part.
Daniel Stenberg [Tue, 21 Jan 2025 10:42:20 +0000 (11:42 +0100)]
asyn-thread: use c-ares to resolve HTTPS RR
Allow building with c-ares and yet use threaded resolver for the main
host A/AAAA resolving:
`--with-ares` provides the c-ares install path and defaults to use
c-ares for name resolving
`--with-threaded-resolver` still uses c-ares in the build (for HTTPS)
but uses the threaded resolver for "normal" resolves.
It works similarly for cmake: ENABLE_ARES enables ares, and if
ENABLE_THREADED_RESOLVER also is set, c-ares is used for HTTPS RR and
the threaded resolver for "normal" resolves.
HTTPSRR and c-ares-rr are new features return by curl_version_info() and
thus shown by curl -V.
The c-ares-rr feature bit is there to make it possible to distinguish
between builds using c-ares for all name resolves and builds that use
the threaded resolves for the regular name resolves and c-ares for
HTTPSRR only. "c-ares-rr" means it does not use c-ares for "plain" name
resolves.
Viktor Szakats [Sat, 25 Jan 2025 13:54:47 +0000 (14:54 +0100)]
cmake: drop `CURL_USE_PKGCONFIG` from `curl-config.cmake.in`
This variable was meant to be used by curl Find modules, but it turns
out it makes no sense to use those from `curl-config.cmake.in`. It means
this variable was not used before and will not be used in the future,
and therefore safe to delete.
Also add missing macros passed to `curl-config.cmake` to comment.
Daniel Stenberg [Fri, 24 Jan 2025 12:19:30 +0000 (13:19 +0100)]
content_encoding: put the decomp buffers into the writer structs
- no more malloc/free per chunk
- removes the extra malloc entirely
- make the buffer (much) smaller (10MB => 16KB!)
- rename 'decomp' to 'buffer' to clarify purpose
Stefan Eissing [Thu, 23 Jan 2025 10:48:06 +0000 (11:48 +0100)]
lib: redirect handling by protocol handler
Adds a `follow()` callback to protocol handlers, so they may decide how
to act on a `newurl` after a request has been done. This is optional.
This moves the HTTP code for handling redirects from multi.c to http.c
where it should be. If we ever add a protocol with its own logic, it
would install its own follow function.
Stefan Eissing [Wed, 22 Jan 2025 13:45:30 +0000 (14:45 +0100)]
lib: clarify 'conn->httpversion'
The variable `conn->httpversion` was used for several purposes and it
was unclear at which time the value represents what.
- rename `conn->httpversion` to `conn->httpversion_seen`
This makes clear that the variable only records the last
HTTP version seen on the connection - if any. And that it
no longer is an indication of what version to use.
- Change Alt-Svc handling to no longer modify `conn->httpversion`
but set `data->state.httpwant` for influencing the HTTP version
to use on a transfer.
- Add `data->req.httpversion_sent` to have a record of what
HTTP version was sent in a request
- Add connection filter type CF_TYPE_HTTP
- Add filter query `CF_QUERY_HTTP_VERSION` to ask what HTTP
filter version is in place
- Lookup filters HTTP version instead of using `conn->httpversion`
Test test_12_05 now switches to HTTP/1.1 correctly and the
expectations have been fixed.
Removed the connection fitler "is_httpN()" checks and using
the version query instead.
Viktor Szakats [Tue, 21 Jan 2025 16:11:45 +0000 (17:11 +0100)]
src: omit hugehelp and ca-embed from libcurltool
CMake builds using the Xcode generator broke with an error saying it
doesn't support multiple targets depending on the same custom commands.
These custom commands are generating `tool_hugehelp.c` and
`tool_c_embed.c` for the curl tool and libcurltool.
`unit1394` and `unit1604` tests use libcurltool to test tool-specific
functions. They don't need hugehelp and ca-embed. It's thus safe to
disable and exclude them when compiling the sources for libcurltool.
Use the `UNITTESTS` macro to detect a libcurltool build within C.
After this patch these sources are solely used for building the curl
tool. Making the build compatible with the CMake Xcode generator.
Apply the change to autotools too to keep build systems synchronized.
Make transfer attach/detach to/from connections chepaer.
- the "attach" event was no longer implemented by any filter
- the "detach" did the same as the "done" event for the filters
who still implemented it. It should be superfluous as the "done"
must always happen.
Jay Satiro [Sun, 19 Jan 2025 06:18:23 +0000 (01:18 -0500)]
easy_lock: use Sleep(1) for thread yield on old Windows
- Prefer Sleep(1) over sched_yield() for pre-Vista thread yield.
On Windows sched_yield is often implemented as Sleep(0) which only
yields to threads of highest priority to current priority. However,
during libcurl initialization if there is thread contention then it's
possible that there is a wait for a different library or OS thread of
a lesser priority and then the yield is not effective during that time.
On the other hand Sleep(1) will wait the minimum time slice which is
usually like 15ms or more.
Prior to this change 2c4bfef removed sched_yield detection on Windows,
which effectively removed the yield in the spin lock, and therefore this
change restores the yield but in a different way.
For Windows Vista and later we use SRW locks and do not have this issue.
Viktor Szakats [Sat, 18 Jan 2025 01:11:37 +0000 (02:11 +0100)]
GHA: add iOS jobs with LibreSSL, enable dependencies for Android via vcpkg
iOS:
- add jobs with autotools, CMake, CMake Xcode generator.
The Xcode generator is >10x slower than Unix Makefiles. Keep it
because it's the one recommended by CMake and for having its own
quirks we may want to know about.
- build, cache and use LibreSSL for these jobs.
With workaround for an iOS build issue fixed in master.
- make Xcode generator work by explicitly disabling code signing.
- make tests and examples build with the Xcode generator by setting
`-DMACOSX_BUNDLE_GUI_IDENTIFIER=se.curl`, to avoid
"Bundle identifier is missing" errors.
- cmake: disable `CURL_USE_PKGCONFIG` by default for Apple device.
- cmake: add `stdc++` library for BoringSSL and AWS-LC, with
`OPENSSL_USE_STATIC_LIBS=ON` set.
- cmake: add workaround for Xcode generator issue, where it cannot
handle two targets depending on one custom command. A better fix may
be dropping `tool_hugehelp.c` and `tool_ca_embed.c` from curltool
library. For a future PR.
Android:
- add vcpkg to Android jobs, enable dependencies. Assisted-by: Tal Regev via #16045
- make vcpkg work with autotools.
- pass `--with-brotli` to autotools to detect the vcpkg-supplied brotli.
- enable BoringSSL for Android and add a job with it.
- silence 457 CMake configure warnings about the Android NDK CMake
scripts targeting freshly deprecated CMake versions.
These were much more involved than imagined. Basically nothing works out
of the box, and when combined, everything becomes a unique edge case.
autotools builds were a much easier to make work than CMake ones.
Also:
- GHA/non-native: re-sync names to be shorter and more aligned with
other workflows.
- GHA: add `persist-credentials: false` where missing.
Unresolved issues:
- `OPENSSL_ROOT_DIR` ignored/mis-used when pointing it to LibreSSL.
CMake seems to prepend the sysroot to the passed absolute directory.
Found no workaround.
- CMake when combined with Android, both the Google-recommended method
and the built-in CMake method fail to provide a way to avoid
`pkg-config` packages at system directories. Failed to find a knob
that can remove `/usr/include` from the search path. The workaround is
to disable zstd. (I enabled it by default in this release, maybe
premature?: f2adb3b6d73cad0c28ec8a32f5fa969d0f6378a0 #15431)
Disabling `pkg-config` doesn't work because vcpkg dependencies do not
link without it.
- CMake's Xcode generator is slow because each `try_compile()` feature
check springs a new CMake + Xcode project taking a long time to run,
just to compile single-liner C files. A known issue, with no solution.
`-DCMAKE_MACOSX_BUNDLE=OFF` did not help, limiting build types to
a single one (e.g. `Debug`) also had no effect.
make | Xcode | GHA run
:---- | :---- | :--------------------------------------------------------------------
16s | 2m57s | https://github.com/curl/curl/actions/runs/12866334102/job/35868712426
23s | 4m13s | https://github.com/curl/curl/actions/runs/12868128013/job/35874212461
16s | 3m39s | https://github.com/curl/curl/actions/runs/12859073531/job/35849041880
14s | 2m23s | https://github.com/curl/curl/actions/runs/12858298423/job/35847201313
15s | 2m36s | https://github.com/curl/curl/actions/runs/12858058492/job/35846669761
19s | 3m19s | https://github.com/curl/curl/actions/runs/12868919430/job/35876601168
Viktor Szakats [Fri, 17 Jan 2025 20:33:12 +0000 (21:33 +0100)]
windows: merge `config-win32ce.h` into `config-win32.h`
They were more or less the same, but each missed some things the other
had. Windows CE is a subset of Win32, make the headers reflect that and
avoid duplications.
Daniel Stenberg [Sun, 19 Jan 2025 11:35:39 +0000 (12:35 +0100)]
libcurl/opts: do not save files in dirs where attackers have access
libcurl cannot fully protect against attacks where an attacker has write
access to the same directory where it is directed to save files. This is
particularly sensitive if you save files using elevated privileges.
Previously only mentioned in VULN-DISCLOSURE-POLICY.md.
Viktor Szakats [Sat, 18 Jan 2025 01:30:08 +0000 (02:30 +0100)]
GHA: tidy up quotes, checkout order, silence Android/CMake warnings
- move checkout step right before use.
- quotes in yaml and bash.
- unfold single-line run commands.
- set `CMAKE_WARN_DEPRECATED=OFF` for CMake with Android NDK to avoid
a wall of useless deprecation warnings hiding useful output.
These warnings happen in 3rd-party scripts, and unfixable in curl.
Jay Satiro [Fri, 17 Jan 2025 08:17:15 +0000 (03:17 -0500)]
checksrc.bat: remove explicit SNPRINTF bypass
- Remove the command line argument passed to checksrc.pl that accepts
SNPRINTF violations in docs/examples.
This is a follow-up to c445b742 which introduced a different management
of banned functions and removed the SNPRINTF rule in favor of banning
snprintf in lib. There's no longer a SNPRINTF warning to suppress.
Viktor Szakats [Thu, 16 Jan 2025 19:39:13 +0000 (20:39 +0100)]
build: stop detecting `sched_yield()` on Windows
On Windows a successful `sched_yield()` detection requires mingw-w64
built with POSIX threads (not Win32 threads) and GCC (not llvm/clang).
(linking to `winpthread` via custom options may also work.)
In CMake builds, it was pre-cached as unavailable before this patch.
When detected (via autotools), it got only used for Windows XP or older
targets combined with a non-GCC, non-clang compiler that doesn't support
`__builtin_ia32_pause()`, or with the Intel C compiler. According to
`lib/easy_lock.h`.
mingw-w64 only supports GCC and clang, leaving a very narrow chance when
`shed_yield()` gets called on Windows. Even then, `sched_yield()` is
implemented in `winpthread` as `Sleep(0)`, which may or not be a useful.
It's also trivial to implement locally if it is, and such rare build
combination is also deemed useful.
Thus, this patch marks `sched_yields()` permanently unavailable on the
Windows platform also with autotools, and instead of pre-caching, skip
this feature check with CMake.
This syncs `HAVE_SCHED_YIELDS` between builds methods on Windows.
Viktor Szakats [Thu, 16 Jan 2025 14:21:23 +0000 (15:21 +0100)]
cmake: pre-fill `HAVE_STDATOMIC_H`, `HAVE_ATOMIC` for mingw-w64
`stdatomic.h` and `_Atomic` were first available in gcc 4.9.0 and
llvm/clang 3.6. Set detection values accordingly and save these two
detections on configure runs.
Stefan Eissing [Fri, 17 Jan 2025 10:57:00 +0000 (11:57 +0100)]
TLS: check connection for SSL use, not handler
Protocol handler option PROTOPT_SSL is used to setup a connection
filters. Once that is done, used `Curl_conn_is_ssl()` to check if
a connection uses SSL.
There may be other reasons to add SSL to a connection, e.g. starttls.
Viktor Szakats [Wed, 15 Jan 2025 14:46:32 +0000 (15:46 +0100)]
android: add CI jobs, buildinfo, cmake docs, disable `CURL_USE_PKGCONFIG` by default
- GHA/non-native: add Android builds, both cmake and autotools,
both NDK 21 (oldest available) and 35 (newest available)
https://github.com/actions/runner-images/blob/main/images/ubuntu/Ubuntu2404-Readme.md
It comes with a maintenance burden to bump the oldest/latest values
with CI runner updates.
- cmake: disable `CURL_USE_PKGCONFIG` by default for Android.
To avoid picking up system package by default.
- build: add `ANDROID-<NDK-LEVEL>` flag to `buildinfo.txt`.
Also detect NDK level with the CMake built-in build method:
https://cmake.org/cmake/help/latest/manual/cmake-toolchains.7.html#cross-compiling-for-android
- INSTALL.md: add CMake build instructions for Android.
- INSTALL.md: make NDK levels consistent in `./configure` example.
Viktor Szakats [Thu, 16 Jan 2025 18:12:02 +0000 (19:12 +0100)]
GHA: start using ARM Linux runners
- GHA/linux: allow per-job runner image override.
- GHA/linux: add an arm version of an existing job.
Add workaround for broken `PATH` in the arm runner image.
- GHA/non-native: add CPU arch to job name where missing.
- GHA/checkdocs: switch a linter job to arm.
Performance looks a little bit better than Intel
(presumably with lower power consumption).
Viktor Szakats [Thu, 16 Jan 2025 21:19:51 +0000 (22:19 +0100)]
GHA/linux: enable wolfSSH in a wolfSSL job
With tests. (Most SFTP tests fail with wolfSSH.)
Also:
- disable pseudo-terminal in wolfSSH builds (also in CircleCI).
Not needed for curl.
- test642: add "compressed" to the description.
To avoid being the same as test600.
Follow-up to b7b4dc0d49543175ab0d9bb1cdc257a2d7f7cf0a #1735
Stefan Eissing [Thu, 2 Jan 2025 15:34:52 +0000 (16:34 +0100)]
websocket: fix message send corruption
- Fix a bug in EAGAIN handling when sending frames that led to a
corrupted last byte of the frame sent.
- Restore sanity to curl_ws_send() behaviour:
- Partial writes are reported as OK with the actual number of
payload bytes sent.
- CURLE_AGAIN is only returned when none of the payload bytes
(or for 0-length frames, not all of the frame header bytes)
could be sent.
- curl_ws_send() now behaves like a common send() call.
- Change 'ws-data' test client to allow concurrent send/recv
operations and vary frame sizes and repeat count.
- Add DEBUG env var CURL_WS_CHUNK_EAGAIN to simulate blocking
after a chunk of an encoded websocket frame has been sent.
- Add tests.
Prior to this change data corruption may occur when sending websocket
messages due to two bugs:
1) 3e64569a (precedes 8.10.0) caused a data corruption bug in the last
byte of frame of large messages.
2) curl_ws_send had non-traditional send behavior and could return
CURLE_AGAIN with bytes sent and expect the caller to adjust buffer
and buflen in a subsequent call. That behavior was not documented.
projects / thirdparty / curl.git / log
