Dan Fandrich [Sun, 5 Feb 2023 00:05:35 +0000 (16:05 -0800)]
tool_operate: Fix error codes on bad URL & OOM
curl would erroneously report CURLE_OUT_OF_MEMORY in some cases instead
of CURLE_URL_MALFORMAT. In other cases, it would erroneously return
CURLE_URL_MALFORMAT instead of CURLE_OUT_OF_MEMORY. Add a test case to
test the former condition.
Kvarec Lezki [Thu, 2 Feb 2023 11:37:42 +0000 (16:37 +0500)]
http: fix "part of conditional expression is always false"
[CWE-570] V560: A part of conditional expression is always false: conn->bits.authneg.
[CWE-570] V560: A part of conditional expression is always false: conn->handler->protocol & (0 | 0).
Stefan Eissing [Wed, 1 Feb 2023 16:13:12 +0000 (17:13 +0100)]
connections: introduce http/3 happy eyeballs
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.
HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.
HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue
HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.
GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.
Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.
HTTP/3 documentation on how https eyeballing works.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy
pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.
Daniel Stenberg [Wed, 1 Feb 2023 08:20:09 +0000 (09:20 +0100)]
CODEOWNERS: remove the peeps mentioned as CI owners
These owners do not have the bandwidth/energy to do the reviews which
makes PRs stall and this ownership claim flawed. We can bring people
back when the situation is different.
Martin D'Aloia [Thu, 2 Feb 2023 01:27:23 +0000 (22:27 -0300)]
write-out.d: add 'since version' to %{header_json} documentation
The documentation of `%{header_json}` missed to mention since which
version this variable for `--write-out` is present.
Based on commit https://github.com/curl/curl/commit/4133a69f2daa476bb
we can determine from the tags were this commit is present that the
first version to include it was `7.83.0`.
This could be also checked with:
`git tag --contains 4133a69f2daa476bb6d902687f1dd6660ea9c3c5`
Daniel Stenberg [Tue, 31 Jan 2023 23:17:28 +0000 (00:17 +0100)]
urlapi: avoid Curl_dyn_addf() for hex outputs
Inspired by the recent fixes to escape.c, we should avoid calling
Curl_dyn_addf() in loops, perhaps in particular when adding something so
simple as %HH codes - for performance reasons. This change makes the
same thing for the URL parser's two URL-encoding loops.
Daniel Stenberg [Wed, 1 Feb 2023 07:26:08 +0000 (08:26 +0100)]
urlapi: skip path checks if path is just "/"
As a miniscule optimization, treat a path of the length 1 as the same as
non-existing, as it can only be a single leading slash, and that's what
we do for no paths as well.
Viktor Szakats [Wed, 1 Feb 2023 09:44:57 +0000 (09:44 +0000)]
tls: fixes for wolfssl + openssl combo builds
1. Add `USE_WOLFSSL` to the TLS backend priority list in
`lib/curl_ntlm_core.c`.
2. Fix `lib/curl_ntlm_core.h` to respect TLS backend priority, bringing
it in sync with the above list and `lib/curl_ntlm_core.c` itself.
Reported-by: Mark Roszko
Ref: https://github.com/curl/curl/issues/10321
3. Allow enabling both wolfSSL and OpenSSL at the same time in
`lib/Makefile.mk` bringing this in line with cmake/autotools builds.
Update logic to select the crypto-specific lib for `ngtcp2`, which
supports a single TLS backend at the same time.
Stefan Eissing [Mon, 30 Jan 2023 15:03:00 +0000 (16:03 +0100)]
cf-socket: improvements in socket I/O handling
- Curl_write_plain/Curl_read_plain have been eliminated. Last code use
now uses Curl_conn_send/recv so that requests use conn->send/revc
callbacks which defaults to cfilters use.
- Curl_recv_plain/Curl_send_plain have been internalized in cf-socket.c.
- USE_RECV_BEFORE_SEND_WORKAROUND (active on Windows) has been moved
into cf-socket.c. The pre_recv buffer is held at the socket filter
context. `postponed_data` structures have been removed from
`connectdata`.
- the hanger in HTTP/2 request handling was a result of read buffering
on all sends and the multi handling is not prepared for this. The
following happens:
- multi preforms on a HTTP/2 easy handle
- h2 reads and processes data
- this leads to a send of h2 data
- which receives and buffers before the send
- h2 returns
- multi selects on the socket, but no data arrives (its in the buffer already)
the workaround now receives data in a loop as long as there is something in
the buffer. The real fix would be for multi to change, so that `data_pending`
is evaluated before deciding to wait on the socket.
io_buffer, optional, in cf-socket.c, http/2 sets state.drain if lower
filter have pending data.
This io_buffer is only available/used when the
-DUSE_RECV_BEFORE_SEND_WORKAROUND is active, e.g. on Windows
configurations. It also maintains the original checks on protocol
handler being HTTP and conn->send/recv not being replaced.
The HTTP/2 (nghttp2) cfilter now sets data->state.drain when it finds
out that the "lower" filter chain has still pending data at the end of
its IO operation. This prevents the processing from becoming stalled.
Jay Satiro [Mon, 23 Jan 2023 08:44:16 +0000 (03:44 -0500)]
write-out.d: clarify Windows % symbol escaping
- Clarify that in Windows batch files the % must be escaped as %%, and
at the command prompt it cannot be escaped which could lead to
incorrect expansion.
Prior to this change the doc implied % must be escaped as %% in win32
always.
---
Examples showing how a write-out argument is received by curl:
If curl --write-out "%{http_code}" is executed in a batch file:
{http_code}
If curl --write-out "%%{http_code}" is executed in a batch file:
%{http_code}
If curl --write-out "%{http_code}" is executed from the command prompt:
%{http_code}
If curl --write-out "%%{http_code}" is executed from the command prompt:
%%{http_code}
At the command prompt something like "%{speed_download}%{http_code}"
would first be parsed by the command interpreter as %{speed_download}%
and would be expanded as environment variable {speed_download} if it
existed, though that's highly unlikely since Windows environment names
don't use braces.
---
Reported-by: Muhammad Hussein Ammari
Ref: https://github.com/bagder/everything-curl/pull/279
Ryan Schmidt [Thu, 26 Jan 2023 01:05:02 +0000 (19:05 -0600)]
cf-socket: Fix build when not HAVE_GETPEERNAME
Remove remaining references to conn and sockfd, which were removed from
the function signature when conninfo_remote was renamed to
conn_set_primary_ip in 6a8d7ef.
Stop explicitly defining `SECURITY_WIN32` in CMake builds.
No other build systems define this macro, because it's unconditionally
defined in `lib/curl_sspi.h` already. This is the only curl source using
the `sspi.h` and `security.h` Win32 headers, and no other Win32 headers
need this macro.
Dan Fandrich [Fri, 20 Jan 2023 19:35:48 +0000 (11:35 -0800)]
CI: Add a workflow to automatically label pull requests
The labeler language is quite restrictive right now so labels are added
quite conservatively, meaning that many PRs won't get labels when it's
"obvious" they should. It will still save some manual work on those
that it can label.
Jay Satiro [Fri, 20 Jan 2023 23:37:44 +0000 (18:37 -0500)]
system.h: assume OS400 is always built with ILEC compiler
Prior to this change the OS400 types were only defined when __ILEC400__.
That symbol is only defined by IBM's C compiler and not their C++
compiler, which led to missing types when users on OS400 would compile a
C++ application that included curl.
The IBM C and C++ compilers are the only native compilers on the
platform.
Assisted-by: Jon Rumsey Reported-by: John Sherrill
Fixes https://github.com/curl/curl/issues/10305
Closes https://github.com/curl/curl/pull/10329
xgladius [Tue, 17 Jan 2023 18:08:10 +0000 (12:08 -0600)]
cmake: Remove deprecated symbols check
curl stopped use of CMAKE_USE_ as a prefix for its own build symbols in
2021 and added a check, meant to last 1 year, to fatally error on those
symbols. This commit removes that check.
Jay Satiro [Thu, 5 Jan 2023 23:32:27 +0000 (18:32 -0500)]
openssl: Don't ignore CA paths when using Windows CA store (redux)
.. and remove 'experimental' designation from CURLSSLOPT_NATIVE_CA.
This commit restores the behavior of CURLSSLOPT_NATIVE_CA so that it
does not override CURLOPT_CAINFO / CURLOPT_CAPATH, or the hardcoded
default locations. Instead the native Windows CA store can be used at
the same time.
---
This behavior was originally added over two years ago in abbc5d60
(#5585) but then 83393b1a (#7892) broke it over a year ago, I assume
inadvertently.
The CURLSSLOPT_NATIVE_CA feature was marked experimental and likely
rarely used.
Stefan Eissing [Thu, 12 Jan 2023 15:43:35 +0000 (16:43 +0100)]
openssl: make the BIO_METHOD a local variable in the connection filter
This avoids UAF issues when `curl_global_cleanup()` is called before all
transfers have been completely handled. Unfortunately this seems to be a
more common pattern than we like.
easyoptions: Fix header printing in generation script
The optiontable.pl script prints the header comment when generating
easyoptions.c, but it wasn't escaping all characters which jumbled the
curl ascii logo. Fix by escaping.
Stefan Eissing [Wed, 11 Jan 2023 09:30:42 +0000 (10:30 +0100)]
curl_log: for failf/infof and debug logging implementations
- new functions and macros for cfilter debugging
- set CURL_DEBUG with names of cfilters where debug logging should be
enabled
- use GNUC __attribute__ to enable printf format checks during compile
Stefan Eissing [Thu, 5 Jan 2023 16:01:42 +0000 (17:01 +0100)]
ngtcp2: add CURLOPT_SSL_CTX_FUNCTION support for openssl+wolfssl
Using common method for SSL_CTX initialization of verfiy peer and CA
settings. This also provides X509_STORE sharing to become available for
ngtcp2+openssl HTTP/3.
Reported-by: violetlige on github
Fixes #10222
Closes #10239
projects / thirdparty / curl.git / log
