Daniel Stenberg [Tue, 7 Jan 2025 17:06:24 +0000 (18:06 +0100)]
DEPRECATE: remove msh3 in six months
The msh3 backed for QUIC and HTTP/3 was introduced in April 2022 but has
never been made to work properly. It has seen no visible traction or
developer activity from the msh3 main author (or anyone else seemingly
interested) in two years. As a non-functional backend, it only adds
friction and "weight" to the development and maintenance.
Meanwhile, we have a fully working backend in the ngtcp2 one and we have
two fully working backends in OpenSSL-QUIC and quiche well on their way
of ending their experimental status in a future.
We remove msh3 support from the curl source tree in July 2025.
Neil Horman [Fri, 3 Jan 2025 15:17:56 +0000 (10:17 -0500)]
osslq: use SSL_poll to determine writeability of QUIC streams
This discussion:
https://github.com/openssl/openssl/discussions/23339#discussion-6094341
Specifically item number 2 (Send Blocking) was raised by the curl team,
noting that SSL_want_write returning false was not a good indicator of
when a stream is writeable. The suggestion in that discussion was to use
SSL_poll with an SSL_POLL_EVENT_W flag instead, as that is a proper
indication of when an SSL_object will allow writing without blocking.
While ssl_want_write updates its state based on the last error
encountered (implying a need to retry an operation to update the
last_error state again), SSL_poll checks stream buffer status during the
call, giving it more up to date information on request. This is the
method used by our guide demos (quic-hq-interop specifically), and it
works well.
This change has been run through the curl test suite, and shown to pass
all tests. However, given the initial problem description I'm not sure
if there is a test case that explicitly checks for blocking and
unblocking of streams. As such some additional testing may be warranted.
Stefan Eissing [Wed, 8 Jan 2025 15:34:38 +0000 (16:34 +0100)]
HTTP/2: strip TE request header
The TE request header field is invalid in HTTP/2. Since clients may not
know in advance if a connection negotiates HTTP/2, automatically strip
such a header when h2 is in play.
Add test_01_10 to verify.
Reported-by: Jiri Stary
Fixes #15941
Closes #15943
Stefan Eissing [Tue, 7 Jan 2025 11:41:26 +0000 (12:41 +0100)]
vtls: feature ssls-export for SSL session im-/export
Adds the experimental feature `ssls-export` to libcurl and curl for
importing and exporting SSL sessions from/to a file.
* add functions to libcurl API
* add command line option `--ssl-sessions <filename>` to curl
* add documenation
* add support in configure
* add support in cmake
+ add pytest case
Viktor Szakats [Tue, 7 Jan 2025 22:44:32 +0000 (23:44 +0100)]
appveyor: bump VS2008 jobs to VS2010
VS2008 has been partly broken for a while with its shared-debug builds
crashing on startup. Its compiler output (UTF-16 HTML) was also barely
readable even after conversion. It's also the only platform in CI
missing `stdint.h`.
This patch migrates a VS2008 job to VS2010 and drops another that
already had a VS2010 equivalent.
We recommend switching to VS2010 or newer when using MSVC to build curl.
Viktor Szakats [Tue, 7 Jan 2025 23:04:20 +0000 (00:04 +0100)]
appveyor: always use cmake `-A` option to select x64
The `Win64` generator suffix alternative was required by old CMake
versions (<3.1) only:
https://cmake.org/cmake/help/v3.22/generator/Visual%20Studio%2010%202010.html
As seen on Linux with 0.7.0:
```
/home/runner/msh3/include/msh3.h:377:18: error: width of ‘RESERVED’ exceeds its type
377 | bool RESERVED : 5;
| ^~~~~~~~
/home/runner/msh3/include/msh3.h:490:18: error: width of ‘RESERVED’ exceeds its type
490 | bool RESERVED : 7;
| ^~~~~~~~
```
https://github.com/curl/curl/actions/runs/12655717818/job/35266716846#step:35:195
Viktor Szakats [Tue, 7 Jan 2025 09:48:57 +0000 (10:48 +0100)]
msvc: fix building with `HAVE_INET_NTOP` and MSVC <=1900
MSVC 1900 and older is missing a `const` specifier in the `inet_ntop()`
declaration for the second argument. A workaround was in place for it
in cmake, but it didn't cover all necessary versions.
Replace the workaround with a different one, move it to `lib/inet_ntop.c`
and extend to all necessary MSVC versions.
Also add CI jobs for the older MSVC versions: 2013, 2015, 2017.
Stefan Eissing [Tue, 31 Dec 2024 15:24:46 +0000 (16:24 +0100)]
conncache: count shutdowns against host and max limits
Count connections to a host against a possibly configured destination
limit. Trigger multi `connchange` when a connection has been shutdown,
so pending transfers can try to get a connection once again.
Reported-by: baranyaib90 on github
Fixes #15857
Closes #15879
This isn't needed anymore after https://github.com/curl/curl/pull/15835,
since banned functions are just allowed in general in
`docs/examples/.checksrc`, and emits a warning when running make
checksrc:
`invalid warning specified in .checksrc: "SNPRINTF"`
Viktor Szakats [Sun, 29 Dec 2024 21:34:09 +0000 (22:34 +0100)]
cmake: publish/check supported protocols/features via `CURLConfig.cmake`
Via these variables, as lists:
- `CURL_SUPPORTED_PROTOCOLS`
- `CURL_SUPPORTED_FEATURES`
As individual flags:
- `CURL_SUPPORTS_<protocol/feature>` = `TRUE`
Also:
- set `CURL_VERSION_STRING` which was missing when using
`find_package(CURL CONFIG)` or
`find_package(CURL NO_MODULE)`.
- set `CURL_<prototol/feature>_FOUND` for compatibility.
- show full list of missing but required `COMPONENTS`.
Daniel Stenberg [Fri, 3 Jan 2025 17:15:44 +0000 (18:15 +0100)]
cleancmd.pl: strip out backticked words
To make sure they are not spellchecked. Also, leaving two backticks is
not good because they cause the spellchecker to misinterpret the
markdown file so they have to be removed as well.
Viktor Szakats [Wed, 1 Jan 2025 02:59:25 +0000 (03:59 +0100)]
GHA/http3-linux: fix cache rebuild conditions, switch to wolfSSL stable
ngtcp2 depends on crypto backends. nghttp2 depends on ngtcp2 and nghttp3
(for nghttpx server used in pytests).
Before this patch, ngtcp2, nghttp2 weren't rebuilt when their
dependencies changes. This worked fine until wolfSSL bumped its
soversion and caused CI to fail because ngtcp2 was not rebuilt and was
still referring to the old soname that was no longer offered by the
wolfSSL package.
Make sure to rebuild ngtcp2/nghttp2 when any of their dependencies bump.
To avoid rebuilding everything on every wolfSSL commit, switch to use
wolfSSL stable versions.
Daniel Stenberg [Thu, 2 Jan 2025 14:54:12 +0000 (15:54 +0100)]
docs/libcurl/opts: clarify the return values
Expand a little.
- mention the type name of the return code
- avoid stating which exact return codes that might be returned, as that
varies over time, builds and conditions
- avoid stating some always return OK
- refer to the manpage documenting all the return codes
Viktor Szakats [Thu, 2 Jan 2025 01:46:00 +0000 (02:46 +0100)]
cmake/FindLDAP: avoid framework locations for libs too (Apple)
We already avoid system framework paths while looking for LDAP headers
to avoid issues.
Do the same while looking for LDAP libraries. This makes sure to find
the regular ldap library (`libldap.tbd`) instead of picking up
`ldap.framework` and let that seep into `libcurl.pc` with a full path.
This makes LDAP detection work on Apple as before introducing FindLDAP.
Daniel Stenberg [Wed, 1 Jan 2025 19:37:47 +0000 (20:37 +0100)]
hash: add asserts in hash_element_dtor()
This just adds a precaution and shows a clear intention in the code.
Added because CodeSonar is reporting a false positive Use After Free on
this function.
Daniel Stenberg [Wed, 1 Jan 2025 00:00:33 +0000 (01:00 +0100)]
select: avoid a NULL deref in cwfds_add_sock
curl_multi_waitfds(m, NULL, ...);
=> Curl_waitfds_init(&cwfds, ufds, size);
=> Curl_waitfds_add_ps(&cwfds);
=> cwfds_add_sock(cwfds, ...);
Would then try to use the ->wfds array while set to NULL previously.
This should not happen, which this is now also protected with an assert
to trigger debug builds if it happens.
Jakub Jelen [Thu, 14 Nov 2024 16:57:48 +0000 (17:57 +0100)]
openssl: add support to use keys and certificates from PKCS#11 provider
In OpenSSL < 3.0, the modularity was provided by mechanism called
"engines". This is supported in curl, but the engines got deprecated
with OpenSSL 3.0 in favor of more versatile providers.
This adds a support for OpenSSL Providers, to use PKCS#11 keys, namely
through the pkcs11 provider. This is done using similar approach as the
engines and this is automatically built in when the OpenSSL 3 and newer
is used.
Signed-off-by: Jakub Jelen <jjelen@redhat.com>
Closes #15587
Viktor Szakats [Wed, 1 Jan 2025 13:10:53 +0000 (14:10 +0100)]
GHA/http3-linux: drop redundant `pkg-config` paths for ngtcp2/nghttp2
- ngtcp2: drop `$PWD/build` (= self)
- ngtcp2: drop nghttp3. It's only used for examples, which we do not use
here and are disabled by default.
- nghttp2: drop `$HOME/build` (does not exist)
Viktor Szakats [Tue, 31 Dec 2024 01:22:42 +0000 (02:22 +0100)]
build: replace configure check with PP condition (Android <21)
To make it build again with CMake + Android 20 and earlier.
8e345057761a8f796403923a96f2c8fd3edca647 synced `getpwuid_r()` detection
in cmake with autotools. It means cmake started detecting it with
Android <21 just like autotools, and thus cmake builds also need to
tackle the missing declaration with old Android SDK versions. Use a PP
solution, allowing to drop the autotools-specific on used before this
patch.
Viktor Szakats [Mon, 30 Dec 2024 19:57:41 +0000 (20:57 +0100)]
build: fix unsigned `time_t` detection for cmake, MS-DOS, AmigaOS
- cmake: add auto-detection. Sync this with autotools.
- enable for MS-DOS and AmigaOS builds.
(auto-detection doesn't work for cross-builds.)
- tidy up detection snippet.
- fix comment.
Viktor Szakats [Mon, 30 Dec 2024 19:02:19 +0000 (20:02 +0100)]
configure: drop unused detections and macros
- drop `HAVE_IOCTL` macro, drop exception.
- drop unused `setjmp.h` detection, drop exception.
It's a C89 header and result also not used in detections.
- use C89 `stdlib.h` without detection.
(It's still being detected by autotools anyway.)
Jay Satiro [Sat, 28 Dec 2024 23:47:45 +0000 (18:47 -0500)]
multi: fix return code for an already-removed easy handle
- Ensure that CURLM_OK is returned when curl_multi_remove_handle is
called with an already removed easy handle.
Prior to this change and since ba235ab2 which precedes 8.10.0, if
curl_multi_remove_handle was called with an already-removed easy handle
then the return code would be CURLM_OK or CURLM_BAD_EASY_HANDLE
depending respectively on whether the multi did or did not contain other
easy handles.
This change restores the old behavior of returning CURLM_OK in both
cases.
Reported-by: Ralph Sennhauser
Fixes https://github.com/curl/curl/issues/15844
Closes https://github.com/curl/curl/pull/15852
Jay Satiro [Thu, 26 Dec 2024 20:38:39 +0000 (15:38 -0500)]
http: ignore invalid Retry-After times
- Treat negative Retry-After date-based times as 0.
- Treat Retry-After times greater than 6 hours as 6 hours.
Prior to this change Retry-After did not have a limited range and the
server could have set a time greater than 6 hours or a date in the past
that would result in a negative time, either of which may be unexpected
by the user.
The 6 hour limit is purposely not documented so that it can be changed
in the future if necessary.
Daniel Stenberg [Mon, 30 Dec 2024 12:17:39 +0000 (13:17 +0100)]
curl/var: change byte offset syntax for assignments
Follow-up to 40c264db617d0 after discussions on IRC.
The new style is
name[0-99]=contents
and
name[0-99]@filename
A) This does not cause the same problems with old curl versions trying
the new syntax as this way will cause old curls just fail with syntax
error and not risk using the wrong file.
B) Adds the same byte range support for "normal" assigns, which the
previous syntax did not. Thus lets a user get a partial content of a
variable etc.
Added test 790 and 791 to verify non-file assigns with ranges.
Stefan Eissing [Sat, 28 Dec 2024 11:19:19 +0000 (12:19 +0100)]
mbedtls: fix handling of blocked sends
mbedtls is picky when a mbedtls_ssl_write) was previously blocked. It
requires to be called with the same amount of bytes again, or it will
lose bytes, e.g. reporting all was sent but they were not. Remember the
blocked length and use that when set.
Reported-by: Tamás Bálint Misius
Fixes #15801
Closes #15846
Daniel Stenberg [Fri, 27 Dec 2024 08:21:56 +0000 (09:21 +0100)]
checksrc: introduce 'banfunc' to ban specific functions
Use 'banfunc' and 'allowfunc' in .checksrc to specify which functions to
ban or allow to be used. This saves us from having to edit the script
going forward when we want to ban or allow specific functions.
This replaces a set of previous rules and all banned functions are now
checked with the BANNEDFUNC rule.
There is a set of default banned functions, shown by invoking
./checksrc.
Also, -a and -b options are added to specify allowed or banned functions
on the command line.
multi: fix curl_multi_waitfds reporting of fd_count
- Make curl_multi_waitfds consistent with the documentation.
Issue Addressed:
- The documentation of curl_multi_waitfds indicates that users should
be able to call curl_multi_waitfds with a NULL ufds. However, before
this change, the function would return CURLM_BAD_FUNCTION_ARGUMENT.
- Additionally, the documentation suggests that users can use this
function to determine the number of file descriptors (fds) needed.
However, the function would stop counting fds if the supplied fds
were exhausted.
Changes Made:
- NULL ufds Handling: curl_multi_waitfds can now accept a NULL ufds if
size is also zero.
- Counting File Descriptors: If curl_multi_waitfds is passed a NULL
ufds, or the size of ufds is insufficient, the output parameter
fd_count will return the number of fds needed. This value may be
higher than actually needed but never lower.
Testing:
- Test 2405 has been updated to cover the usage scenarios described
above.
Jay Satiro [Tue, 24 Dec 2024 07:18:37 +0000 (02:18 -0500)]
cookie: fix crash in netscape cookie parsing
- Parse the input string without modifying it.
Prior to this change a segfault could occur if the input string was
const because the tokenizer modified the input string. For example if
the user set CURLOPT_COOKIELIST to a const string then libcurl would
likely cause a crash when modifying that string. Even if the string was
not const or a crash did not occur there was still the incorrect and
unexpected modification of the user's input string.
This issue was caused by 30da1f59 (precedes 8.11.0) which refactored
some options parsing and eliminated the copy of the input string. Also,
an earlier commit f88cc654 incorrectly cast the input pointer when
passing it to strtok.
Co-authored-by: Daniel Stenberg
Closes https://github.com/curl/curl/pull/15826
Viktor Szakats [Tue, 24 Dec 2024 01:43:02 +0000 (02:43 +0100)]
clang-tidy: add to CI, add cmake support, fix fallouts
build:
- autotools: fix to build generated sources for the `tidy` target.
- autotools: allow passing custom clang-tidy options via
`CURL_CLANG_TIDYFLAGS` env.
- cmake: add `CURL_CLANG_TIDY` option to configure for `clang-tidy`.
Also add:
- `CLANG_TIDY` variable to customize the `clang-tidy` tool.
- `CURL_CLANG_TIDYFLAGS` to pass custom options to `clang-tidy`.
- apply `--enable-werror` and `-DCURL_WERROR=ON` to `clang-tidy`.
CI/GHA:
- add clang-tidy job for Linux, using autotools and clang-tidy v18.
This one needs to disable `clang-analyzer-valist.Uninitialized`
to avoid false positives:
https://github.com/llvm/llvm-project/issues/40656
Duration: 5.5 minutes
- add clang-tidy job for macOS, using cmake and clang-tidy v19.
This one also covers tests and examples, and doesn't hit the false
positives seen with llvm v18 and earlier.
Duration: 4.5 minutes
- Linux/macOS: skip installing test dependencies when not building or
running tests.
Viktor Szakats [Thu, 26 Dec 2024 10:34:24 +0000 (11:34 +0100)]
cmake: add `librtmp` Find module
The new detection method also allows to enable librtmp without using
OpenSSL as a curl TLS backend at the same time.
Also:
- implement manual version detection for librtmp.
Version info is in hex. With CMake 3.13 and newer, extract it as a hex
number. With earlier CMake version, just strip the leading zeroes.
Doing more here seems overkill because librtmp has been standing
at 2.3/2.4 for a decade now. Bumping into hex digits seems unlikely
before deprecating CMake 3.13 support.
librtmp advertises v2.4 via its `pkg-config` module, and v2.3 via
its public header. The latter shows up in `curl -V` and either can
be shown at configure-time depending on detection method.
This isn't a curl bug.
- GHA/macos: enable rtmp in a job.
- apply the "half-detection" fix to the Find module.
`librtmp` is also affected (in CI too), because it depends on libssl and
libcrypto.
Viktor Szakats [Sun, 22 Dec 2024 10:30:45 +0000 (11:30 +0100)]
cmake: move `pkg-config` names to Find modules
Make the Find modules set and return their respective `pkg-config`
module name(s) to the CMake build process, which then adds those
to the `Requires:` list.
Before this patch, `pkg-config` module names were maintainted in two
separate places. After this patch, they are maintained in the Find
modules for dependencies that have one (most do).
Re-align existing modules with this change: msh3, mbedtls, rustls.
These modules return their `pkg-config` module name only when
detected via `pkg-config`.
Viktor Szakats [Tue, 24 Dec 2024 09:12:31 +0000 (10:12 +0100)]
cmake/FindLibpsl: protect against `pkg-config` "half-detection"
Same issue as seen before with libssh2: `libpsl`'s pkg-config module
depends on another module, but that's not found. CMake ends up reporting
`LIBPSL_FOUND=YES`, while leaving `LIBPSL_INCLUDE_DIRS` empty. Then
the build fails to find `psl.h`.
The missing dependency in this case is `icu4c`, which is "keg-only",
meaning it's not exposed in the default Homebrew header, pkg-config,
lib, etc locations. It must be added to the `PKG_CONFIG_PATH` env, as
suggested by the warnings messages of `pkgconf`.
To avoid this fallout, let's ensure that `LIBPSL_INCLUDE_DIRS` is
non-empty when detecting via `pkg-config` and fall back to the CMake
detection method otherwise.
This was an issue till Homebrew libpsl 0.21.5_1, fixed in 0.21.5_2, that
no longer depends on `icu4c`.
Example log:
```
-- Checking for module 'libpsl'
-- Found libpsl, version 0.21.5
Package icu-uc was not found in the pkg-config search path.
Perhaps you should add the directory containing `icu-uc.pc'
to the PKG_CONFIG_PATH environment variable
Package 'icu-uc', required by 'libpsl', not found
[...]
-- Found Libpsl (via pkg-config): (found version "0.21.5")
[...]
In file included from curl/_bld/lib/CMakeFiles/libcurl_static.dir/Unity/unity_0_c.c:4:
In file included from curl/lib/altsvc.c:32:
In file included from curl/lib/urldata.h:145:
curl/lib/psl.h:28:10: fatal error: 'libpsl.h' file not found
^~~~~~~~~~
1 error generated.
```
Viktor Szakats [Mon, 23 Dec 2024 20:36:23 +0000 (21:36 +0100)]
cmake/FindLDAP: avoid empty 'Requires' item when omitting `pkg-config` module
`list(APPEND ...)` does this automatically, but we're _prepending_ LDAP
(to follow historical code and also autotools). `list(PREPEND ...)`
would likely do that the same, but it requires CMake 3.15 so we do this
manually and the manual method needs to skip the empty value manually.
Viktor Szakats [Wed, 20 Nov 2024 21:01:32 +0000 (22:01 +0100)]
curl-config: tidy up, optimize
- optimize out `cppflag_curl_staticlib` variable.
- optimize out `CPPFLAG_CURL_STATICLIB` variable and simplify logic.
- lowercase local variable name `CURLLIBDIR`.
projects / thirdparty / curl.git / log
