Viktor Szakats [Thu, 4 Apr 2024 10:45:01 +0000 (10:45 +0000)]
cmake: fix `CURL_WERROR=ON` for old CMake and use it in GHA/linux-old
- cmake: fix `-pedantic-errors` for old CMake with `CURL_WERROR=ON` set.
`-pedantic-errors` option throws a warning with GCC (all versions) and
makes `check_symbol_exists()` fail in CMake versions older than
v3.23.0 (2022-03-29), when CMake introduced a workaround:
Viktor Szakats [Thu, 4 Apr 2024 10:45:01 +0000 (10:45 +0000)]
GHA: enable unity mode for cmake jobs + tidy-ups
Unity mode is not supported by CMake v3.7.2 used in linux-old, but
enable it anyway for consistency and to kick in automatically once
migrating to a newer old Linux in the future.
Stefan Eissing [Wed, 3 Apr 2024 11:18:01 +0000 (13:18 +0200)]
request: paused upload on completed download, assess connection
A transfer with a completed download that is still uploading needs to
check the connection state when it is PAUSEd, since connection
close/errors would otherwise go unnoticed.
Viktor Szakats [Wed, 27 Mar 2024 15:54:08 +0000 (15:54 +0000)]
cmake: speed up libcurl doc building again
This time limit the number of files per command to avoid exceeding
limitations of certain OS/shell envs.
Such known env is Windows with the `cmd.exe` shell, which features an
8K command-line length limit to this day.
Allowlisting `UNIX` to have no limit and using a limit of 200 for other
envs to be safe. If there is a way to detect `cmd.exe` and/or we know
which precise envs are sensitive to this, we can tweak these conditions
further.
Even with the low limit, this patch reduces external commands by 200x,
making builds much faster.
Fabian Keil [Sun, 17 Mar 2024 14:49:58 +0000 (15:49 +0100)]
wolfssl: plug memory leak in wolfssl_connect_step2()
Fixes:
test 2034...[simple HTTPS GET with DER public key pinning]
==61829== 22,610 (3,744 direct, 18,866 indirect) bytes in 1 blocks are definitely lost in loss record 51 of 54
==61829== at 0x484BB74: malloc (vg_replace_malloc.c:446)
==61829== by 0x4B53A80: wolfSSL_Malloc (memory.c:344)
==61829== by 0x4C1C8E1: wolfSSL_X509_new (x509.c:5326)
==61829== by 0x4C3977D: d2i_X509orX509REQ (x509.c:3628)
==61829== by 0x4C1D1F4: wolfSSL_X509_d2i (x509.c:3664)
==61829== by 0x4C1C37B: wolfSSL_X509_dup (x509.c:13425)
==61829== by 0x4C197DB: wolfSSL_get_peer_certificate (ssl.c:18765)
==61829== by 0x33297C: wolfssl_connect_step2 (wolfssl.c:875)
==61829== by 0x331669: wolfssl_connect_common (wolfssl.c:1287)
==61829== by 0x3303E9: wolfssl_connect_nonblocking (wolfssl.c:1319)
==61829== by 0x32FE89: ssl_connect_nonblocking (vtls.c:510)
==61829== by 0x32DBE5: ssl_cf_connect (vtls.c:1679)
==61829== by 0x27ABD7: Curl_conn_cf_connect (cfilters.c:307)
==61829== by 0x27D9CF: cf_setup_connect (connect.c:1199)
==61829== by 0x27ABD7: Curl_conn_cf_connect (cfilters.c:307)
==61829== by 0x283CEA: cf_hc_baller_connect (cf-https-connect.c:135)
Viktor Szakats [Wed, 3 Apr 2024 03:06:16 +0000 (03:06 +0000)]
appveyor: OpenSSL 3 no longer found by CMake, revert to 1.1.1
OpenSSL moved directories, and bumped versions in AppVeyor CI.
Downgrading is not an ideal solution, but however trivial the solution
may be, I failed to come with anything that made CMake recognize either
OpenSSL 3.1 or 3.2.
Stefan Eissing [Mon, 1 Apr 2024 13:41:18 +0000 (15:41 +0200)]
http: with chunked POST forced, disable length check on read callback
- when an application forces HTTP/1.1 chunked transfer encoding
by setting the corresponding header and instructs curl to use
the CURLOPT_READFUNCTION, disregard any POST length information.
- this establishes backward compatibility with previous curl versions
Applications are encouraged to not force "chunked", but rather
set length information for a POST. By setting -1, curl will
auto-select chunked on HTTP/1.1 and work properly on other HTTP
versions.
Daniel McCarney [Sun, 31 Mar 2024 20:19:10 +0000 (16:19 -0400)]
m4: reposition USE_RUSTLS="yes" for pkg-config
It's necessary to set this var to "yes" _after_ AC_DEFINE and AC_SUBST
in order for a later `test` to pass so that `check_for_ca_bundle=1` ends
up being set. This is in turn required for the default CA certificate
bundle to be set when building w/ rustls & pkg-config.
Reported-by: Matt Jolly
Fixes #13248
Closes #13251
Daniel Stenberg [Sat, 30 Mar 2024 09:41:21 +0000 (10:41 +0100)]
curl: use curl_getenv instead of the curlx_ version
The curlx one was once introduced when we still considered dropping the
libcurl function at some point. To reduce confusion and to make it
easier to understand when curl_free() should be used, use the actual
libcurl function call directly instead.
Matt Jolly [Wed, 27 Mar 2024 12:52:26 +0000 (22:52 +1000)]
m4: fix rustls pkg-config codepath
The previous pkg-config code would successfully detect rustls but did
not set all appropriate variables and call the right macros to properly
configure cURL.
Reported-by: kpcyrd on github
Fixes #13200
Closes #13202
Daniel McCarney [Sat, 30 Mar 2024 18:18:45 +0000 (14:18 -0400)]
deps: update librustls 0.12.0 -> 0.13.0
This commit updates the optional rustls-ffi librustls dependency from
0.12.0 to 0.13.0. This version is based on the latest available rustls
release (0.23.4).
The breaking API changes from 0.12.0 to 0.13.0 are in API surface unused
by curl, so this is an in-place update without any code changes.
The `RUSTLS.md` documentation is updated to reflect the new version in
use, and to clarify that `cbindgen` isn't required to build `librustls`
- it's only used by developers to update the vendored `rustls.h` header
file maintained upstream.
Internally, libssh2 dereferences the NULL pointer if length is non-zero.
The callback function cannot return the error condition, so at least
prevent subsequent crash.
Daniel Stenberg [Sat, 23 Mar 2024 13:43:35 +0000 (14:43 +0100)]
DISTROS: add document with distro pointers
Lots of organizations distribute curl packages to end users. This is a
collection of pointers to where to learn more about curl on and with
each distro.
Assisted-by: Alan Coopersmith Assisted-by: Andrew Kaster Assisted-by: Andy Fiddaman Assisted-by: Arjan van de Ven Assisted-by: Brian Clemens Assisted-by: chrysos349 on github Assisted-by: Dan Fandrich Assisted-by: Dan McDonald Assisted-by: Gaelan Steele Assisted-by: graywolf on github Assisted-by: Jan Macku Assisted-by: John Marshall Assisted-by: Jonathan Perkin Assisted-by: Kevin Daudt Assisted-by: Marcus Müller Assisted-by: Michał Górny Assisted-by: Outvi V Assisted-by: Ross Burton Assisted-by: Sean Molenaar Assisted-by: Till Wegmüller Assisted-by: Viktor Szakats Assisted-by: Winni Neessen
Closes #13178
Fabian Keil [Sun, 17 Mar 2024 13:30:33 +0000 (14:30 +0100)]
wolfSSL: do not call the stub function wolfSSL_BIO_set_init()
Calling the function isn't necessary and causes the build
to fail when wolfSSL has been compiled with NO_WOLFSSL_STUB:
Making all in opts
CCLD curl
ld: error: undefined symbol: wolfSSL_BIO_set_init
>>> referenced by wolfssl.c:235 (vtls/wolfssl.c:235)
>>> libcurl_la-wolfssl.o:(wolfssl_bio_cf_create) in archive ../lib/.libs/libcurl.a
cc: error: linker command failed with exit code 1 (use -v to see invocation)
*** Error code 1
Matt Jolly [Sun, 24 Mar 2024 09:00:09 +0000 (19:00 +1000)]
curl-rustls.m4: add pkg-config support to rustls detection
Based on the existing openssl pkg-config detection, this commit tries to
use pkg-config to find `rustls` then falls back to the current approach
if that fails.
We use the following logic:
- if no path is provided, just use pkg-config, if it's not there we have
a problem!
- if a path is provided, try pkg-config
+ if pkg-config fails, try and find rustls directly
Daniel Stenberg [Thu, 21 Mar 2024 14:32:26 +0000 (15:32 +0100)]
docs/libcurl: cleanups
- CURLINFO_TLS_SESSION.md: remove mention of NSS
- CURLINFO_TLS_SSL_PTR.md: remove NSS leftover
- CURLOPT_CAINFO.md: drop mention of backends not supporting this
- CURLOPT_CAPATH.md: wolfSSL also supports this
Daniel Stenberg [Thu, 21 Mar 2024 10:50:20 +0000 (11:50 +0100)]
docs: make each libcurl man specify protocol(s)
The mandatory header now has a mandatory list of protocols for which the
manpage is relevant.
Most man pages already has a "PROTOCOLS" section, but this introduces a
stricter way to specify the relevant protocols.
cd2nroff verifies that at least one protocol is mentioned (which can be
`*`).
This information is not used just yet, but A) the PROTOCOLS section can
now instead get generated and get a unified wording across all manpages
and B) this allows us to more reliably filter/search for protocol
specific manpages/options.
Stefan Eissing [Mon, 11 Mar 2024 16:23:15 +0000 (17:23 +0100)]
http: separate response parsing from response action
- move code that triggers on end-of-response into separate function from
parsing
- simplify some headp/headerlen usage
- add `httpversion` to SingleRequest to indicate the version of the
current response
Stefan Eissing [Mon, 18 Mar 2024 11:46:43 +0000 (12:46 +0100)]
http: improve response header handling, save cpu cycles
Saving some cpu cycles in http response header processing:
- pass the length of the header line along
- use string constant sizeof() instead of strlen()
- check line length if prefix is possible
- switch on first header char to limit checks
Stefan Eissing [Mon, 11 Mar 2024 16:23:15 +0000 (17:23 +0100)]
http: expect 100 rework
Move all handling of HTTP's `Expect: 100-continue` feature into a client
reader. Add sending flag `KEEP_SEND_TIMED` that triggers transfer
sending on general events like a timer.
HTTP installs a `CURL_CR_PROTOCOL` reader when announcing `Expect:
100-continue`. That reader works as follows:
- on first invocation, records time, starts the `EXPIRE_100_TIMEOUT`
timer, disables `KEEP_SEND`, enables `KEEP_SEND_TIMER` and returns 0,
eos=FALSE like a paused upload.
- on subsequent invocation it checks if the timer has expired. If so, it
enables `KEEP_SEND` and switches to passing through reads to the
underlying readers.
Transfer handling's `readwrite()` will be invoked when a timer expires
(like `EXPIRE_100_TIMEOUT`) or when data from the server arrives. Seeing
`KEEP_SEND_TIMER`, it will try to upload more data, which triggers
reading from the client readers again. Which then may lead to a new
pausing or cause the upload to start.
Flags and timestamps connected to this have been moved from
`SingleRequest` into the reader's context.
Stefan Eissing [Fri, 8 Mar 2024 08:37:27 +0000 (09:37 +0100)]
TLS: start shutdown only when peer did not already close
- When curl sees a TCP close from the peer, do not start a TLS shutdown.
TLS shutdown is a handshake and if the peer already closed the
connection, it is not interested in participating.
Reported-by: dfdity on github Assisted-by: Jiří Bok Assisted-by: Pēteris Caune
Fixes #10290
Closes #13087
Daniel Stenberg [Thu, 14 Mar 2024 11:56:39 +0000 (12:56 +0100)]
GHA/linux: add sysctl trick to work-around GitHub runner issue
The GitHub image runner update from 20240304.1.0 to 20240310.1
introduces a problem for clang-14. The issue is caused by
incompatibility between llvm 14 provided in ubuntu-22.04 image and the
much newer kernel configured with high-entropy ASLR.
As a work-around, we issue a sysctl command to lower the entropy and get
clang-14 to work again.
Daniel Stenberg [Tue, 12 Mar 2024 07:45:22 +0000 (08:45 +0100)]
managen: remove space before protocols
For options that are listed for specific protocols, the protocols (shown
first within parentheses) are now output without the leading space in the
manpage output.
Jay Satiro [Thu, 7 Mar 2024 00:18:46 +0000 (19:18 -0500)]
mbedtls: properly cleanup the thread-shared entropy
- Store the state of the thread-shared entropy for global init/cleanup.
- Use curl's thread support of mbedtls for all Windows builds instead of
just when the threaded resolver is used via USE_THREADS_WIN32.
Prior to this change on global cleanup curl builds that have curl thread
support for mbedtls freed the entropy (8b1d2298) but failed to mark that
it had been freed, which caused problems on subsequent init + transfer.
Stefan Eissing [Mon, 11 Mar 2024 11:35:07 +0000 (12:35 +0100)]
lib: move 'done' parameter to SingleRequests
A transfer may do several `SingleRequest`s for its success. This happens
regularly for authentication, follows and retries on failed connections.
The "readwrite()" calls and functions connected to those carried a `bool
*done` parameter to indicate that the current `SingleRequest` is over.
This may happen before `upload_done` or `download_done` bits of
`SingleRequest` are set.
The problem with that is now `write_resp()` protocol handlers are
invoked in places where the `bool *done` cannot be passed up to the
caller. Instead of being a bool in the call chain, it needs to become a
member of `SingleRequest`, reflecting its state.
This removes the `bool *done` parameter and adds the `done` bit to
`SingleRequest` instead. It adds `Curl_req_soft_reset()` for using a
`SingleRequest` in a follow up, clearing `done` and other
flags/counters.