Daniel Stenberg [Wed, 17 Apr 2024 09:39:25 +0000 (11:39 +0200)]
urlapi: add CURLU_GET_EMPTY for empty queries and fragments
By default the API inhibits empty queries and fragments extracted.
Unless this new flag is set.
This also makes the behavior more consistent: without it set, zero
length queries and fragments are considered not present in the URL. With
the flag set, they are returned as a zero length strings if they were in
fact present in the URL.
This applies when extracting the individual query and fragment
components and for the full URL.
Daniel Stenberg [Wed, 17 Apr 2024 08:42:28 +0000 (10:42 +0200)]
urlapi: fix relative redirects to fragment-only
Using the URL API for a redirect URL when the redirected-to string
starts with a hash, ie is only a fragment, the API would produce the
wrong final URL.
Adjusted test 1560 to test for several new redirect cases.
Stefan Eissing [Wed, 27 Mar 2024 14:01:06 +0000 (15:01 +0100)]
http/2, http/3: decouple stream state from easy handle
- add `Curl_hash_offt` as hashmap between a `curl_off_t` and
an object. Use this in h2+h3 connection filters to associate
`data->id` with the internal stream state.
- changed implementations of all affected connection filters
- removed `h2_ctx*` and `h3_ctx*` from `struct HTTP` and thus
the easy handle
- solves the problem of attaching "foreign protocol" easy handles
during connection shutdown
idn: add native AppleIDN (icucore) support for macOS/iOS
I implemented the IDN functions for macOS and iOS using Unicode
libraries coming with macOS and iOS.
Builds and runs here on macOS 14.2.1. Also verified to load and
run on older macOS version 10.13.
Build requires macOS SDK 13 or equivalent.
Set `-DUSE_APPLE_IDN=ON` CMake option to enable it.
With autotools and other build tools, set these manual options:
```
CPPFLAGS=-DUSE_APPLE_IDN
LIBS=-licucore
```
Completes TODO 1.6.
TODO: add autotools option and feature-detection.
Refs: #5330 #5371 Co-authored-by: Viktor Szakats
Closes #13246
Stefan Eissing [Mon, 15 Apr 2024 12:34:32 +0000 (14:34 +0200)]
http3: extend download abort tests, fixes in ngtcp2
- fix flow handling in ngtcp2 to ACK data on streams
we abort ourself.
- extend test_02_23* cases to also run for h3
- skip test_02_23* for OpenSSL QUIC as it gets stalled
on progressing the connection
Viktor Szakats [Sat, 13 Apr 2024 09:51:36 +0000 (09:51 +0000)]
openvms: look for `USE_IPV6` in `config.h` (was: `ENABLE_IPV6`)
The OpenVMS script `config_h.com` is parsing the config header
generated by autotools. Let's make it look for the macro name we now
use universally across the codebase.
Viktor Szakats [Mon, 15 Apr 2024 18:39:15 +0000 (18:39 +0000)]
mbedtls: fix building with v3 in CMake Unity mode
Before this patch the internal feature detection macro
`HAS_MBEDTLS_RESULT_CODE_BASED_FUNCTIONS` was defined in three files,
with an incomplete logic in one of them. In Unity mode that spilled
into another source file and broke the build.
Daniel Stenberg [Fri, 5 Apr 2024 11:07:16 +0000 (13:07 +0200)]
multi: introduce SETUP state for better timeouts
Since we can go to the CONNECT state from PENDING, potentially multiple
times for a single transfer, this change introdues a SETUP state that
happens before CONNECT when doing a new transfer.
Now, doing a redirect on a handle goes back to SETUP (not CONNECT like
before) and we initilize the connect timeout etc in SETUP. Previously,
we would do it in CONNECT but that would make it unreliable in cases
where a transfer goes in and out between CONNECT and PENDING multiple
times.
SETUP is transient, so the handle never actually stays in that state.
Additionally: take care of timeouts of PENDING transfers in
curl_multi_perform()
docs: clarify CURLOPT_MAXFILESIZE and CURLOPT_MAXFILESIZE_LARGE
The bounds of the size parameter were not specified, and nor was it
specified how to disable the maximum file size check.
The documentation also incorrectly stated that CURLOPT_MAXFILESIZE
always returns CURLE_OK and that CURLOPT_MAXFILESIZE_LARGE only returns
CURLE_OK or CURLE_UNKNOWN_OPTION.
It also did not mention what the default value is, which is zero. This
commit updates the documentation to make note of all these things.
Daniel Stenberg [Thu, 4 Apr 2024 09:14:44 +0000 (11:14 +0200)]
multi: timeout handles even without connection
When there is a "change" in a multi handle and pending handles are moved
back to the main list to be retested if they can proceed further (for
example a previous transfer completed or a connection has a confirmed
multiplexed state), the timeout check in multi_runsingle() would not
trigger because it required an established connection.
This could make a pending tranfer go back to pending state even though
it had been "in progress" for a longer time than permitted. By removing
the requirement for an associated connection, the timeout check will be
done proper even for transfers that has not yet been assigned one.
Ref #13227 Reported-by: Rahul Krishna M
Closes #13276
Patrick Monnerat [Sun, 14 Apr 2024 12:20:28 +0000 (14:20 +0200)]
mprintf: check fputc error rather than matching returned character
OS/400 ascii fputc wrapper deviates from the posix standard by the
fact that it returns the ebcdic encoding of the original ascii
character. Testing for a matching value for success will then always
fail.
This commit replaces the chariacter comparison by an explicit error
return check.
Viktor Szakats [Sat, 13 Apr 2024 10:42:34 +0000 (10:42 +0000)]
ci: add CMake build variation, fixup libssh detection in `linux-old`
To test without c-ares and hit `easy_lock.h` on an old system. Use this
new build step to introduce small variations, and also test libssh2.
Also add workaround to existing job to enable libssh. (CMake's generic
auto-detection doesn't seem to work here.):
```
CMake Warning at CMakeLists.txt:908 (find_package):
Could not find a package configuration file provided by "libssh" with any
of the following names:
Viktor Szakats [Thu, 11 Apr 2024 16:35:08 +0000 (16:35 +0000)]
lib: merge `ENABLE_QUIC` C macro into `USE_HTTP3`
Before this patch `lib/curl_setup.h` defined these two macros right
next to each other, then the source code used them interchangeably.
After this patch, `USE_HTTP3` guards all HTTP/3 / QUIC features.
(Like `USE_HTTP2` does for HTTP/2.) `ENABLE_QUIC` is no longer used.
This patch doesn't change the way HTTP/3 is enabled via autotools
or CMake. Builders who enabled HTTP/3 manually by defining both of
these macros via `CPPFLAGS` can now delete `-DENABLE_QUIC`.
Before this patch, two macros were used to guard IPv6 features in curl
sources: `ENABLE_IPV6` and `USE_IPV6`. This patch makes the source use
the latter for consistency with other similar switches.
`-DENABLE_IPV6` remains accepted for compatibility as a synonym for
`-DUSE_IPV6`, when passed to the compiler.
`ENABLE_IPV6` also remains the name of the CMake and `Makefile.vc`
options to control this feature.
docs: add CURLOPT_NOPROGRESS to CURLOPT_XFERINFOFUNCTION example
It's important to set `CURLOPT_NOPROGRESS` to `0` if you want your
transfer callback function, set by `CURLOPT_XFERINFOFUNCTION`, getting
called. To emphasize this to the users, add this to the code example.
Limited to x64, because for build testing the additional CPUs don't add
much value compared to the extra build time. They can be enabled easily
if deemed useful.
To the extent of curl-for-win configuration options, it's trivial to add
further build combinations.
Viktor Szakats [Sun, 7 Apr 2024 12:39:59 +0000 (12:39 +0000)]
OS400: fix shellcheck warnings in scripts
- use `$()` instead of backticks, and re-arrange double-quotes inside.
- add missing `|| exit 1` to `cd` calls. (could be dropped by using `set -eu`.)
- add `-n` to a few `if`s.
- shorten redirections by using `{} >` (as shellcheck recommended).
- silence warnings where variables were detected as unused (SC2034).
- a couple misc updates to silence warnings.
- switch to bash shebang for `-ot` feature.
- split two lines to unbreak syntax highlighting in my editor. (`$(expr \`, `$(dirname \`)
Stefan Eissing [Wed, 10 Apr 2024 11:30:16 +0000 (13:30 +0200)]
gnutls: lazy init the trust settings
- delay loading of trust anchors and CRLs after the ClientHello
has been sent off
- add tracing to IO operations
- on IO errors, return the CURLcode of the underlying filter
Viktor Szakats [Tue, 9 Apr 2024 19:08:18 +0000 (19:08 +0000)]
appveyor: make VS2010 job build-only, enable Schannel, fix compiler warnings
Tests were consistently flaky for a while.
Also fix compiler warnings in `CertOpenStore()` calls for old MSVC compilers:
```
C:/projects/curl/lib/vtls/schannel.c(688):
warning C4306: 'type cast' : conversion from 'int' to 'LPCSTR' of greater size
C:/projects/curl/lib/vtls/schannel_verify.c(642):
warning C4306: 'type cast' : conversion from 'int' to 'LPCSTR' of greater size
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/49580310/job/ywu2y44kymgc0nif#L106
Daniel Stenberg [Mon, 8 Apr 2024 07:13:17 +0000 (09:13 +0200)]
projects: drop MSVC project files for recent versions
We encourage users to generate visual studio project files using CMake.
We keep project files in git for ancient visual studio versions that
cmake cannot generate files for, but we no longer ship the project files
in the tarballs.
appveyor: switch VisualStudioSolution job to VC12 (Visual Studio 2013)
Co-Authored-by: Viktor Szakats Co-Authored-by: Jay Satiro
Closes #13311
Stefan Eissing [Tue, 9 Apr 2024 07:41:19 +0000 (09:41 +0200)]
tests: stabilitze test_02_23*
- h2-download now always opens the output file on first write callback
invocation, if it will pause the transfer or not.
- Checks on output files then does not depend on the amount of data curl
has collected for the first write.
Viktor Szakats [Tue, 9 Apr 2024 06:46:35 +0000 (06:46 +0000)]
dist: add reproducible dir entries to tarballs
In the initial implementation of reproducible tarballs, they were
missing directory entries, while .zip archives had them. It meant
that on extracting the tarball, on-disk directory entries got the
current timestamp.
This patch fixes this by including directory entries in the tarball,
with reproducible timestamps. It also moves sorting inside tar,
to ensure reproducible directory entry timestamps on extract
(without the need of `--delay-directory-restore` option, when
extracting with GNU tar. BSD tar got that right by default.)
Daniel Stenberg [Mon, 8 Apr 2024 15:14:35 +0000 (17:14 +0200)]
release-tools.sh: store the timestamp and release tag too
When maketgz invokes this script to generate the docs/RELEASE-TOOLS.md
file that gets bundled in the release, it now also passes on the exact
timestamp and version number so that those details also get mentioned in
the document. They will help users reproduce an identical tarball.
Stefan Eissing [Fri, 5 Apr 2024 13:38:11 +0000 (15:38 +0200)]
http2: emit RST when client write fails
- When the writing of response data fails, reset the stream
and do not return a callback error to nghttp2. That would
be a fatal error for the connection and harm other requests.
- add test cases for various abort scenarios
Reported-by: Konstantin Kuzov
Fixes #13292
Closes #13298
Daniel Stenberg [Thu, 4 Apr 2024 16:00:33 +0000 (18:00 +0200)]
docs/cmdline-opts: invoke managen using a relative path
... no need to use an absolute path, that makes the build unncessarily
fail if invoked using a different mount point. managen now takes options
to find the input files.
Update test1478 to provide the dir arguments to managen
Viktor Szakats [Fri, 5 Apr 2024 13:27:05 +0000 (13:27 +0000)]
dist: `set -eu`, fix shellcheck, make reproducible and smaller tarballs
- set bash `-eu` and fix fallouts.
- fix shellcheck warnings.
- set and use `SOURCE_DATE_EPOCH` for reproducibility. Authored-by: Daniel J. H.
Ref: #13280
- set `TZ=UTC` and `LC_ALL=C` for reproducibility.
- make file timestamps in tarball/zip reproducible.
- make directory timestamps in zip reproducible.
- make timestamps of tarballs/zip reproducible.
- make file order in tarball/zip reproducible.
- omit extra file metadata from zip for reproducibility.
- use maximum zip compression.
- use POSIX `ustar` tarball format to avoid supply chain vulnerability:
https://seclists.org/oss-sec/2021/q4/0
- make uid/gid in tarball reproducible.
- omit owner user/group names from tarball for reproducibility and privacy.
- omit current timestamp from .gz header for reproducibility.
- display SHA-256 hashes of produced tarballs/zip.
- fix whitespace.
`.tar.gz` also became smaller in the process: 4,462,311 -> 4,148,249 bytes (8.7.1)
Requires GNU tar, GNU date, `sha256sum`.
Reviewed-by: Daniel Stenberg
Ref: #13250
Closes #13299
Stefan Eissing [Fri, 5 Apr 2024 10:11:48 +0000 (12:11 +0200)]
vquic: use new curl_int64_t type
- add curl_int64_t signed 64-bit type for lib use
- define CURL_PRId64, CURL_PRIu64 format ids
- use curl_int64_t in vquic
curl_int64_t signed complements the existing curl_uint64_t unsigned.
Note that `curl_int64_t` and `int64_t` are assignable from each other
but not identical. Some platforms with 64 long type defint int64_t as
"long long" (staring at macOS) which messes up things like pointers and
format identifiers.
Jay Satiro [Tue, 9 Jan 2024 08:24:57 +0000 (03:24 -0500)]
lib: use multi instead of multi_easy for the active multi
- Use data->multi and not data->multi_easy to refer to the active multi.
The easy handle's active multi is always data->multi.
This is a follow up to 757dfdf which changed curl so that an easy handle
used with the easy interface and then multi interface cannot have two
different multi handles associated with it at the same time
(data->multi_easy from the easy interface and data->multi from the multi
interface).
Daniel Stenberg [Fri, 5 Apr 2024 11:01:58 +0000 (13:01 +0200)]
makefile: remove the sorting from the vc-ide action
This target generates the MSVC project files. This change removes the
extra sorting and instead makes the script use the order of the files as
listed in the variables - which are mostly sorted anyway.
This is an attempt to make the project file generation more easily
reproducible.