Charlie C [Mon, 6 Nov 2023 21:19:00 +0000 (14:19 -0700)]
cmake: option to disable install & drop `curlu` target when unused
This patch makes the following changes:
- adds the option `CURL_DISABLE_INSTALL` - to disable 'install' targets.
- Removes the target `curlu` when the option `BUILD_TESTING` is set to
`OFF` - to prevent it from being loaded in Visual Studio.
Kai Pastor [Fri, 22 Sep 2023 05:48:36 +0000 (07:48 +0200)]
cmake: fix multiple include of CURL package
Fixes errors on second `find_package(CURL)`. This is a frequent case
with transitive dependencies:
```
CMake Error at ...:
add_library cannot create ALIAS target "CURL::libcurl" because another
target with the same name already exists.
```
Test to reproduce:
```cmake
cmake_minimum_required(VERSION 3.27) # must be 3.18 or higher
Viktor Szakats [Wed, 8 Nov 2023 04:39:54 +0000 (04:39 +0000)]
tidy-up: use `OPENSSL_VERSION_NUMBER`
Uniformly use `OPENSSL_VERSION_NUMBER` to check for OpenSSL version.
Before this patch some places used `OPENSSL_VERSION_MAJOR`.
Also fix `lib/md4.c`, which included `opensslconf.h`, but that doesn't
define any version number in these implementations: BoringSSL, AWS-LC,
LibreSSL, wolfSSL. (Only in mainline OpenSSL/quictls). Switch that to
`opensslv.h`. This wasn't causing a deeper problem because the code is
looking for v3, which is only provided by OpenSSL/quictls as of now.
According to https://github.com/openssl/openssl/issues/17517, the macro
`OPENSSL_VERSION_NUMBER` is safe to use and not deprecated.
Daniel Stenberg [Tue, 7 Nov 2023 09:58:08 +0000 (10:58 +0100)]
content_encoding: make Curl_all_content_encodings allocless
- Fixes a memory leak pointed out by Coverity
- Also found by OSS-Fuzz: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=63947
- Avoids unncessary allocations
David Suter [Mon, 6 Nov 2023 15:45:44 +0000 (15:45 +0000)]
projects: add VC14.20 project files
Windows projects included VC14, VC14.10, VC14.30 but not VC14.20.
OpenSSL and Wolf SSL scripts mention VC14.20 so I don't see a reason why
this is missing. Updated the templates to produce a VC14.20 project.
Project opens in Visual Studio 2019 as expected.
Daniel Stenberg [Mon, 6 Nov 2023 12:24:43 +0000 (13:24 +0100)]
curl: move IPFS code into src/tool_ipfs.[ch]
- convert ensure_trailing into ensure_trailing_slash
- strdup the URL string to own it proper
- use shorter variable names
- combine some expressions
- simplify error handling in ipfs_gateway()
- add MAX_GATEWAY_URL_LEN + proper bailout if maximum is reached
- ipfs-gateway.d polish and simplification
- shorten ipfs error message + make them "synthetic"
Viktor Szakats [Sat, 28 Oct 2023 01:10:01 +0000 (01:10 +0000)]
build: delete support bits for obsolete Windows compilers
- Pelles C: Unclear status, failed to obtain a fresh copy a few months
ago. Possible website is HTTP-only. ~10 years ago I left this compiler
dealing with crashes and other issues with no response on the forum
for years. It has seen some activity in curl back in 2021.
- LCC: Last stable release in September 2002.
- Salford C: Misses winsock2 support, possibly abandoned? Last mentioned
in 2006.
- Borland C++: We dropped Borland C++ support in 2018.
- MS Visual C++ 6.0: Released in 1998. curl already requires VS 2010
(or possibly 2008) as a minimum.
Viktor Szakats [Mon, 6 Nov 2023 02:17:39 +0000 (02:17 +0000)]
build: delete `HAVE_STDINT_H` and `HAVE_INTTYPES_H`
We use `stdint.h` unconditionally in all places except one. These uses
are imposed by external dependencies / features. nghttp2, quic, wolfSSL
and `HAVE_MACH_ABSOLUTE_TIME` do require this C99 header. It means that
any of these features make curl require a C99 compiler. (In case of
MSVC, this means Visual Studio 2010 or newer.)
This patch changes the single use of `stdint.h` guarded by
`HAVE_STDINT_H` to use `stdint.h` unconditionally. Also stop using
`inttypes.h` as an alternative there. `HAVE_INTTYPES_H` wasn't used
anywhere else, allowing to delete this feature check as well.
Stefan Eissing [Mon, 23 Oct 2023 08:33:07 +0000 (10:33 +0200)]
lib: client writer, part 2, accounting + logging
This PR has these changes:
Renaming of unencode_* to cwriter, e.g. client writers
- documentation of sendf.h functions
- move max decode stack checks back to content_encoding.c
- define writer phase which was used as order before
- introduce phases for monitoring inbetween decode phases
- offering default implementations for init/write/close
Add type paramter to client writer's do_write()
- always pass all writes through the writer stack
- writers who only care about BODY data will pass other writes unchanged
add RAW and PROTOCOL client writers
- RAW used for Curl_debug() logging of CURLINFO_DATA_IN
- PROTOCOL used for updates to data->req.bytecount, max_filesize checks and
Curl_pgrsSetDownloadCounter()
- remove all updates of data->req.bytecount and calls to
Curl_pgrsSetDownloadCounter() and Curl_debug() from other code
- adjust test457 expected output to no longer see the excess write
Viktor Szakats [Sun, 5 Nov 2023 23:27:55 +0000 (23:27 +0000)]
rand: fix build error with autotools + LibreSSL
autotools unexpectedly detects `arc4random` because it is also looking
into dependency libs. One dependency, LibreSSL, happens to publish an
`arc4random` function (via its shared lib before v3.7, also via static
lib as of v3.8.2). When trying to use this function in `lib/rand.c`,
its protoype is missing. To fix that, curl included a prototype, but
that used a C99 type without including `stdint.h`, causing:
```
../../lib/rand.c:37:1: error: unknown type name 'uint32_t'
37 | uint32_t arc4random(void);
| ^
1 error generated.
```
This patch improves this by dropping the local prototype and instead
limiting `arc4random` use for non-OpenSSL builds. OpenSSL builds provide
their own random source anyway.
The better fix would be to teach autotools to not link dependency libs
while detecting `arc4random`.
LibreSSL publishing a non-namespaced `arc4random` tracked here:
https://github.com/libressl/portable/issues/928
Enno Boland [Fri, 3 Nov 2023 19:00:38 +0000 (20:00 +0100)]
HTTP: fix empty-body warning
This change fixes a compiler warning with gcc-12.2.0 when
`-DCURL_DISABLE_BEARER_AUTH=ON` is used.
/home/tox/src/curl/lib/http.c: In function 'Curl_http_input_auth':
/home/tox/src/curl/lib/http.c:1147:12: warning: suggest braces around empty body in an 'else' statement [-Wempty-body]
1147 | ;
| ^
Daniel Stenberg [Sat, 4 Nov 2023 22:36:40 +0000 (23:36 +0100)]
openssl: identify the "quictls" backend correctly
Since vanilla OpenSSL does not support the QUIC API I think it helps
users to identify the correct OpenSSL fork in version output. The best
(crude) way to do that right now seems to be to check if ngtcp2 support
is enabled.
Jay Satiro [Wed, 1 Nov 2023 07:18:53 +0000 (03:18 -0400)]
curl_sspi: support more revocation error names in error messages
- Add these revocation errors to sspi error list:
CRYPT_E_NO_REVOCATION_DLL, CRYPT_E_NO_REVOCATION_CHECK,
CRYPT_E_REVOCATION_OFFLINE and CRYPT_E_NOT_IN_REVOCATION_DATABASE.
Prior to this change those error codes were not matched to their macro
name and instead shown as "unknown error".
Before:
schannel: next InitializeSecurityContext failed:
Unknown error (0x80092013) - The revocation function was
unable to check revocation because the revocation server was offline.
After:
schannel: next InitializeSecurityContext failed:
CRYPT_E_REVOCATION_OFFLINE (0x80092013) - The revocation function was
unable to check revocation because the revocation server was offline.
Bug: https://github.com/curl/curl/issues/12239 Reported-by: Niracler Li
Closes https://github.com/curl/curl/pull/12241
Jay Satiro [Thu, 2 Nov 2023 22:56:06 +0000 (18:56 -0400)]
strdup: don't allow Curl_strndup to read past a null terminator
- Use malloc + strncpy instead of Curl_memdup to dupe the string before
null terminating it.
Prior to this change if Curl_strndup was passed a length longer than
the allocated string then it could copy out of bounds.
This change is for posterity. Curl_strndup was added in the parent
commit and currently none of the calls to it pass a length that would
cause it to read past the allocated length of the input.
Stefan Eissing [Tue, 31 Oct 2023 14:25:08 +0000 (15:25 +0100)]
vtls: late clone of connection ssl config
- perform connection cache matching against `data->set.ssl.primary`
and proxy counterpart
- fully clone connection ssl config only when connection is used
Daniel Stenberg [Sat, 21 Oct 2023 12:32:30 +0000 (14:32 +0200)]
http: consider resume with CURLOPT_FAILONERRROR and 416 to be fine
Finding a 'Content-Range:' in the response changed the handling.
Add test case 1475 to verify -C - with 416 and Content-Range: header,
which is almost exactly like test 194 which instead uses a fixed -C
offset. Adjusted test 194 to also be considered fine.
Viktor Szakats [Sat, 28 Oct 2023 10:52:23 +0000 (10:52 +0000)]
build: require Windows XP or newer
After this patch we assume availability of `getaddrinfo` and
`freeaddrinfo`, first introduced in Windows XP. Meaning curl
now requires building for Windows XP as a minimum.
Jay Satiro [Sat, 28 Oct 2023 04:23:40 +0000 (00:23 -0400)]
curl_setup: disallow Windows IPv6 builds missing getaddrinfo
- On Windows if IPv6 is enabled but getaddrinfo is missing then #error
the build.
curl can be built with IPv6 support (ENABLE_IPV6) but without the
ability to resolve hosts to IPv6 addresses (HAVE_GETADDRINFO). On
Windows this is highly unlikely and should be considered a bad build
configuration.
Such a bad configuration has already given us a bug that was hard to
diagnose. See #12134 and #12136 for discussion.
Nico Rieck [Mon, 23 Oct 2023 10:38:19 +0000 (12:38 +0200)]
openssl: make CURLSSLOPT_NATIVE_CA import Windows intermediate CAs
- If CURLSSLOPT_NATIVE_CA on Windows then import from intermediate CA
"CA" store after importing from root CA "ROOT" store.
This change allows curl to work in situations where a server does not
send all intermediate certs and they are present in the "CA" store (the
store with intermediate CAs). This is already allowed by the Schannel
backend.
Also this change makes partial chain verification possible for those
certs since we allow partial chain verification by default for OpenSSL
(unless CURLSSLOPT_NO_PARTIALCHAIN). This is not allowed by the Schannel
backend.
Prior to this change CURLSSLOPT_NATIVE_CA only imported "ROOT" certs.
Viktor Szakats [Fri, 27 Oct 2023 10:20:47 +0000 (10:20 +0000)]
Makefile.mk: restore `_mingw.h` for default `_WIN32_WINNT`
In 8.4.0 we deleted `_mingw.h` as part of purging old-mingw support.
Turns out `_mingw.h` had the side-effect of setting a default
`_WIN32_WINNT` value expected by `lib/config-win32.h` to enable
`getaddrinfo` support in `Makefile.mk` mingw-w64 builds. This caused
disabling support for this unless specifying the value manually.
Restore this header and update its comment to tell why we continue
to need it.
This triggered a regression in official Windows curl builds starting
with 8.4.0_1. Fixed in 8.4.0_6. (8.5.0 will be using CMake.)
Stefan Eissing [Thu, 26 Oct 2023 15:02:45 +0000 (17:02 +0200)]
cfilter: provide call to tell connection to forget a socket
- fixed libssh.c workaround for a socket being closed by
the library
- eliminate the terrible hack in cf-socket.c to guess when
this happened and try not closing the socket again.
- fixes race in eyeballing when socket could have failed to
be closed for a discarded connect attempt
Viktor Szakats [Thu, 26 Oct 2023 08:28:20 +0000 (08:28 +0000)]
cmake: speed up threads setup for Windows
Win32 threads are always available. We enabled them unconditionally
(with `ENABLE_THREADED_RESOLVER`). CMake built-in thread detection
logic has this condition hard-coded for Windows as well (since at least
2007).
Instead of doing all the work of detecting pthread combinations on
Windows, then discarding those results, skip these efforts and assume
built-in thread support when building for Windows.
Viktor Szakats [Wed, 25 Oct 2023 23:37:48 +0000 (23:37 +0000)]
cmake: speed up zstd detection
Before this patch we detected the presence of a specific zstd API to
see if we can use the library. zstd published that API in its first
stable release: v1.0.0 (2016-08-31).
Replace that method by detecting the zstd library version instead and
accepting if it's v1.0.0 or newer. Also display this detected version
and display a warning if the zstd found is unfit for curl.
We use the same version detection method as zstd itself, via its public
C header.
This deviates from autotools which keeps using the slow method of
looking for the API by building a test program. The outcome is the same
as long as zstd keeps offering this API.
Stefan Eissing [Wed, 25 Oct 2023 10:31:34 +0000 (12:31 +0200)]
urldata: move async resolver state from easy handle to connectdata
- resolving is done for a connection, not for every transfer
- save create/dup/free of a cares channel for each transfer
- check values of setopt calls against a local channel if no
connection has been attached yet, when needed.
Stefan Eissing [Mon, 4 Sep 2023 10:06:07 +0000 (12:06 +0200)]
lib: introduce struct easy_poll_set for poll information
Connection filter had a `get_select_socks()` method, inspired by the
various `getsocks` functions involved during the lifetime of a
transfer. These, depending on transfer state (CONNECT/DO/DONE/ etc.),
return sockets to monitor and flag if this shall be done for POLLIN
and/or POLLOUT.
Due to this design, sockets and flags could only be added, not
removed. This led to problems in filters like HTTP/2 where flow control
prohibits the sending of data until the peer increases the flow
window. The general transfer loop wants to write, adds POLLOUT, the
socket is writeable but no data can be written.
This leads to cpu busy loops. To prevent that, HTTP/2 did set the
`SEND_HOLD` flag of such a blocked transfer, so the transfer loop cedes
further attempts. This works if only one such filter is involved. If a
HTTP/2 transfer goes through a HTTP/2 proxy, two filters are
setting/clearing this flag and may step on each other's toes.
Connection filters `get_select_socks()` is replaced by
`adjust_pollset()`. They get passed a `struct easy_pollset` that keeps
up to `MAX_SOCKSPEREASYHANDLE` sockets and their `POLLIN|POLLOUT`
flags. This struct is initialized in `multi_getsock()` by calling the
various `getsocks()` implementations based on transfer state, as before.
After protocol handlers/transfer loop have set the sockets and flags
they want, the `easy_pollset` is *always* passed to the filters. Filters
"higher" in the chain are called first, starting at the first
not-yet-connection one. Each filter may add sockets and/or change
flags. When all flags are removed, the socket itself is removed from the
pollset.
Example:
* transfer wants to send, adds POLLOUT
* http/2 filter has a flow control block, removes POLLOUT and adds
POLLIN (it is waiting on a WINDOW_UPDATE from the server)
* TLS filter is connected and changes nothing
* h2-proxy filter also has a flow control block on its tunnel stream,
removes POLLOUT and adds POLLIN also.
* socket filter is connected and changes nothing
* The resulting pollset is then mixed together with all other transfers
and their pollsets, just as before.
Use of `SEND_HOLD` is no longer necessary in the filters.
All filters are adapted for the changed method. The handling in
`multi.c` has been adjusted, but its state handling the the protocol
handlers' `getsocks` method are untouched.
The most affected filters are http/2, ngtcp2, quiche and h2-proxy. TLS
filters needed to be adjusted for the connecting handshake read/write
handling.
No noticeable difference in performance was detected in local scorecard
runs.
Viktor Szakats [Thu, 5 Oct 2023 23:38:59 +0000 (23:38 +0000)]
cmake: pre-fill rest of detection values for Windows
The goal of this patch is to avoid unnecessary feature detection work
when doing Windows builds with CMake. Do this by pre-filling well-known
detection results for Windows and specifically for mingw-w64 and MSVC
compilers. Also limit feature checks to platforms where the results are
actually used. Drop a few redundant ones. And some tidying up.
- pre-fill remaining detection values in Windows CMake builds.
Based on actual detection results observed in CI runs, preceding
similar work over libssh2 and matching up values with
`lib/config-win32.h`.
This brings down CMake configuration time from 58 to 14 seconds on the
same local machine.
On AppVeyor CI this translates to:
- 128 seconds -> 50 seconds VS2022 MSVC with OpenSSL (per CMake job):
https://ci.appveyor.com/project/curlorg/curl/builds/48208419/job/4gw66ecrjpy7necb#L296
https://ci.appveyor.com/project/curlorg/curl/builds/48217440/job/8m4fwrr2fe249uo8#L186
- 62 seconds -> 16 seconds VS2017 MINGW (per CMake job):
https://ci.appveyor.com/project/curlorg/curl/builds/48208419/job/s1y8q5ivlcs7ub29?fullLog=true#L290
https://ci.appveyor.com/project/curlorg/curl/builds/48217440/job/pchpxyjsyc9kl13a?fullLog=true#L194
The formula is about 1-3 seconds delay for each detection. Almost all
of these trigger a full compile-link cycle behind the scenes, slow
even today, both cross and native, mingw-w64 and apparently MSVC too.
Enabling .map files or other custom build features slows it down
further. (Similar is expected for autotools configure.)
- stop detecting `idn2.h` if idn2 was deselected.
autotools does this.
- stop detecting `idn2.h` if idn2 was not found.
This deviates from autotools. Source code requires both header and
lib, so this is still correct, but faster.
- limit `ADDRESS_FAMILY` detection to Windows.
- normalize `HAVE_WIN32_WINNT` value to lowercase `0x0a12` format.
- pre-fill `HAVE_WIN32_WINNT`-dependent detection results.
Saving 4 (slow) feature-detections in most builds: `getaddrinfo`,
`freeaddrinfo`, `inet_ntop`, `inet_pton`
- fix pre-filled `HAVE_SYS_TIME_H`, `HAVE_SYS_PARAM_H`,
`HAVE_GETTIMEOFDAY` for mingw-w64.
Luckily this do not change build results, as `WIN32` took
priority over `HAVE_GETTIMEOFDAY` with the current source
code.
- limit `HAVE_CLOCK_GETTIME_MONOTONIC_RAW` and
`HAVE_CLOCK_GETTIME_MONOTONIC` detections to non-Windows.
We're not using these in the source code for Windows.
- reduce compiler warning noise in CMake internal logs:
- fix to include `winsock2.h` before `windows.h`.
Apply it to autotools test snippets too.
- delete previous `-D_WINSOCKAPI_=` hack that aimed to fix the above.
- cleanup `CMake/CurlTests.c` to emit less warnings.
- delete redundant `HAVE_MACRO_SIGSETJMP` feature check.
It was the same check as `HAVE_SIGSETJMP`.
- delete 'experimental' marking from `CURL_USE_OPENSSL`.
- show CMake version via `CMakeLists.txt`.
Credit to the `zlib-ng` project for the idea:
https://github.com/zlib-ng/zlib-ng/blob/61e181c8ae93dbf56040336179c9954078bd1399/CMakeLists.txt#L7
- make `CMake/CurlTests.c` pass `checksrc`.
- `CMake/WindowsCache.cmake` tidy-ups.
- replace `WIN32` guard with `_WIN32` in `CMake/CurlTests.c`.
Alex Bozarth [Tue, 3 Oct 2023 22:30:13 +0000 (17:30 -0500)]
OpenSSL: Include SIG and KEM algorithms in verbose
Currently the verbose output does not include which algorithms are used
for the signature and key exchange when using OpenSSL. Including the
algorithms used will enable better debugging when working on using new
algorithm implementations. Know what algorithms are used has become more
important with the fast growing research into new quantum-safe
algorithms.
This implementation includes a build time check for the OpenSSL version
to use a new function that will be included in OpenSSL 3.2 that was
introduced in openssl/openssl@6866824
Based-on-patch-by: Martin Schmatz <mrt@zurich.ibm.com>
Closes #12030
Daniel Stenberg [Sun, 22 Oct 2023 09:22:08 +0000 (11:22 +0200)]
http2: provide an error callback and failf the message
Getting nghttp2's error message helps users understand what's going
on. For example when the connection is brought down due a forbidden
header is used - as that header is then not displayed by curl itself.
Example:
curl: (92) Invalid HTTP header field was received: frame type: 1,
stream: 1, name: [upgrade], value: [h2,h2c]
Viktor Szakats [Thu, 19 Oct 2023 21:12:48 +0000 (21:12 +0000)]
cmake: fix OpenSSL quic detection in quiche builds
An orphan call to `CheckQuicSupportInOpenSSL()` remained after a recent
update when checking QUIC for quiche. Move back QUIC detection to
a function and fixup callers to use that. Also make sure that quiche
gets QUIC from BoringSSL, because it doesn't support other forks at this
time.
Daniel Stenberg [Fri, 20 Oct 2023 09:15:40 +0000 (11:15 +0200)]
openldap: move the alloc of ldapconninfo to *connect()
Fixes a minor memory leak on LDAP connection reuse.
Doing the allocation already in *setup_connection() is wrong since that
connect struct might get discarded early when an existing connection is
reused instead.
Daniel Stenberg [Mon, 16 Oct 2023 12:46:36 +0000 (14:46 +0200)]
gnutls: support CURLSSLOPT_NATIVE_CA
Remove the CURL_CA_FALLBACK logic. That build option was added to allow
primarily OpenSSL to use the default paths for loading the CA certs. For
GnuTLS it was instead made to load the "system certs", which is
different and not desirable.
The native CA store loading is now asked for with this option.
Stefan Eissing [Sat, 7 Oct 2023 13:13:09 +0000 (15:13 +0200)]
RTSP: improved RTP parser
- fix HTTP header parsing to report incomplete
lines it buffers as consumed!
- re-implement the RTP parser for interleave RTP
messages for robustness. It is now keeping its
state at the connection
- RTSP protocol handler "readwrite" implementation
now tracks if the response is before/in/after
header parsing or "in" a bod by calling
"Curl_http_readwrite_headers()" itself. This
allows it to know when non-RTP bytes are "junk"
or HEADER or BODY.
- tested with #12035 and various small receive
sizes where current master fails
projects / thirdparty / curl.git / log
