Dan Fandrich [Sat, 11 Feb 2023 19:05:42 +0000 (11:05 -0800)]
CONTRIBUTE: More formally specify the commit description
This codifies what people have actually used in git commits over the
past 6 years. I've left off some lesser-used headers that appear to
duplicate others and tried to describe a consistent use for several
others that were used more arbitrarily.
This makes it easier for new committers to find out the kinds of things
we want to acknowledge, makes it easier to perform statistical analysis
on commits, and opens the possibility of performing lint checks on
descriptions before submission.
Reviewed-by: Daniel Stenberg Reviewed-by: Jay Satiro
Closes #10478
Stefan Eissing [Mon, 13 Feb 2023 15:15:20 +0000 (16:15 +0100)]
openssl: test and fix for forward proxy handling (non-tunneling).
- adding pytest test_10 cases for proxy httpd setup tests
- fixing openssl bug in https: proxy hostname verification that
used the hostname of the request and not the proxy name.
Viktor Szakats [Sun, 12 Feb 2023 12:52:59 +0000 (12:52 +0000)]
windows: always use curl's basename() implementation
The `basename()` [1][2] implementation provided by mingw-w64 [3] makes
assumptions about input encoding and may break with non-ASCII strings.
`basename()` was auto-detected with CMake, autotools and since 68fa9bf3f5d7b4fcbb57619f70cb4aabb79a51f6 (2022-10-13), also in
`Makefile.mk` after syncing its behaviour with the mainline build
methods. A similar patch for curl-for-win broke official Windows
builds earlier, in release 7.83.1_4 (2022-06-15).
This patch forces all Windows builds to use curl's internal
`basename()` implementation to avoid such problems.
Dmitry Atamanov [Fri, 10 Feb 2023 17:44:36 +0000 (22:44 +0500)]
cmake: fix dev warning due to mismatched arg
The package name passed to find_package_handle_standard_args (BROTLI)
does not match the name of the calling package (Brotli). This can lead
to problems in calling code that expects find_package result variables
(e.g., _FOUND) to follow a certain pattern.
Changes numeric error codes into verbose error codes in two places.
Adds a prefix indicating that the error came from rustls, and in some
places which function it came from.
Adds special handling for RUSTLS_RESULT_UNEXPECTED_EOF, since the
default message of "Unexpected EOF" is insufficiently explanatory.
Stefan Eissing [Thu, 9 Feb 2023 09:49:04 +0000 (10:49 +0100)]
vquic: stabilization and improvements
vquic stabilization
- udp send code shared between ngtcp2 and quiche
- quiche handling of data and events improved
ngtcp2 and pytest improvements
- fixes handling of "drain" situations, discovered in scorecard
tests with the Caddy server.
- improvements in handling transfers that have already data or
are already closed to make an early return on recv
pytest
- adding caddy tests when available
scorecard improvemnts.
- using correct caddy port
- allowing tests for only httpd or caddy
Dan Fandrich [Thu, 9 Feb 2023 20:06:12 +0000 (12:06 -0800)]
CI: Retry failed downloads to reduce spurious failures
A temporary error with a remote server shouldn't cause a CI run to fail.
Also, put a cap on the time to download to fail faster on a misbehaving
server or connection and use HTTP compression where possible to reduce
download times.
Stefan Eissing [Wed, 8 Feb 2023 09:26:58 +0000 (10:26 +0100)]
HTTP/[23]: continue upload when state.drain is set
- as reported in #10433, HTTP/2 uploads may stall when a response is
received before the upload is done. This happens when the
data->state.drain is set for such a transfer, as the special handling
in transfer.c from then on only cared about downloads.
- add continuation of uploads, if applicable, in this case.
- add pytest case test_07_12_upload_seq_large to reproduce this scenario
(although, current nghttp2 implementation is using drain less often)
Reported-by: Lucas Pardue
Fixes #10433
Closes #10443
openldap: fix missing sasl symbols at build in specific configs
If curl is built with openldap support (USE_OPENLDAP=1) but does not
have also some other protocol (IMAP/SMTP/POP3) enabled that brings
in Curl_sasl_* functions, then the build will fail with undefined
references to various symbols:
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_decode_mech'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_parse_url_auth_option'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_cleanup'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_can_authenticate'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_continue'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_start'
ld: ../lib/.libs/libcurl.so: undefined reference to `Curl_sasl_init'
This was tracked down to these functions bein used in openldap.c but
defined in curl_sasl.c and then forward in two vauth/ files to have
a guard against a set of #define configurations that was now extended
to cover also this case.
Example configuration targeted that could reproduce the problem:
Jay Satiro [Fri, 3 Feb 2023 08:11:49 +0000 (03:11 -0500)]
curl_setup: Disable by default recv-before-send in Windows
Prior to this change a workaround for Windows to recv before every send
was enabled by default. The way it works is a recv is called before
every send and saves the received data, in case send fails because in
Windows apparently that can wipe out the socket's internal received
data buffer.
This feature has led to several bugs because the way libcurl operates
it waits on a socket to read or to write, and may not at all times
check for buffered receive data.
Two recent significant bugs this workaround caused:
- Broken Schannel TLS 1.3 connections (#9431)
- HTTP/2 arbitrary hangs (#10253)
The actual code remains though it is disabled by default. Though future
changes to connection filter buffering could improve the situation IMO
it's just not tenable to manage this workaround.
Stefan Eissing [Wed, 8 Feb 2023 09:37:36 +0000 (10:37 +0100)]
http2: aggregate small SETTINGS/PRIO/WIN_UPDATE frames
add a small buffer to nghttp2 session sending in order to aggregate
small SETTINGS/PRIO/WIN_UPDATE frames that nghttp2 "writes" to the
callback individually.
Dan Fandrich [Sun, 5 Feb 2023 00:05:35 +0000 (16:05 -0800)]
tool_operate: Fix error codes on bad URL & OOM
curl would erroneously report CURLE_OUT_OF_MEMORY in some cases instead
of CURLE_URL_MALFORMAT. In other cases, it would erroneously return
CURLE_URL_MALFORMAT instead of CURLE_OUT_OF_MEMORY. Add a test case to
test the former condition.
Kvarec Lezki [Thu, 2 Feb 2023 11:37:42 +0000 (16:37 +0500)]
http: fix "part of conditional expression is always false"
[CWE-570] V560: A part of conditional expression is always false: conn->bits.authneg.
[CWE-570] V560: A part of conditional expression is always false: conn->handler->protocol & (0 | 0).
Stefan Eissing [Wed, 1 Feb 2023 16:13:12 +0000 (17:13 +0100)]
connections: introduce http/3 happy eyeballs
New cfilter HTTP-CONNECT for h3/h2/http1.1 eyeballing.
- filter is installed when `--http3` in the tool is used (or
the equivalent CURLOPT_ done in the library)
- starts a QUIC/HTTP/3 connect right away. Should that not
succeed after 100ms (subject to change), a parallel attempt
is started for HTTP/2 and HTTP/1.1 via TCP
- both attempts are subject to IPv6/IPv4 eyeballing, same
as happens for other connections
- tie timeout to the ip-version HAPPY_EYEBALLS_TIMEOUT
- use a `soft` timeout at half the value. When the soft timeout
expires, the HTTPS-CONNECT filter checks if the QUIC filter
has received any data from the server. If not, it will start
the HTTP/2 attempt.
HTTP/3(ngtcp2) improvements.
- setting call_data in all cfilter calls similar to http/2 and vtls filters
for use in callback where no stream data is available.
- returning CURLE_PARTIAL_FILE for prematurely terminated transfers
- enabling pytest test_05 for h3
- shifting functionality to "connect" UDP sockets from ngtcp2
implementation into the udp socket cfilter. Because unconnected
UDP sockets are weird. For example they error when adding to a
pollset.
HTTP/3(quiche) improvements.
- fixed upload bug in quiche implementation, now passes 251 and pytest
- error codes on stream RESET
- improved debug logs
- handling of DRAIN during connect
- limiting pending event queue
HTTP/2 cfilter improvements.
- use LOG_CF macros for dynamic logging in debug build
- fix CURLcode on RST streams to be CURLE_PARTIAL_FILE
- enable pytest test_05 for h2
- fix upload pytests and improve parallel transfer performance.
GOAWAY handling for ngtcp2/quiche
- during connect, when the remote server refuses to accept new connections
and closes immediately (so the local conn goes into DRAIN phase), the
connection is torn down and a another attempt is made after a short grace
period.
This is the behaviour observed with nghttpx when we tell it to shut
down gracefully. Tested in pytest test_03_02.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
- new tests/tests-httpd/scorecard.py for testing h3/h2 protocol implementation.
Invoke:
python3 tests/tests-httpd/scorecard.py --help
for usage.
Improvements on gathering connect statistics and socket access.
- new CF_CTRL_CONN_REPORT_STATS cfilter control for having cfilters
report connection statistics. This is triggered when the connection
has completely connected.
- new void Curl_pgrsTimeWas(..) method to report a timer update with
a timestamp of when it happend. This allows for updating timers
"later", e.g. a connect statistic after full connectivity has been
reached.
- in case of HTTP eyeballing, the previous changes will update
statistics only from the filter chain that "won" the eyeballing.
- new cfilter query CF_QUERY_SOCKET for retrieving the socket used
by a filter chain.
Added methods Curl_conn_cf_get_socket() and Curl_conn_get_socket()
for convenient use of this query.
- Change VTLS backend to query their sub-filters for the socket when
checks during the handshake are made.
HTTP/3 documentation on how https eyeballing works.
TLS improvements
- ALPN selection for SSL/SSL-PROXY filters in one vtls set of functions, replaces
copy of logic in all tls backends.
- standardized the infof logging of offered ALPNs
- ALPN negotiated: have common function for all backends that sets alpn proprty
and connection related things based on the negotiated protocol (or lack thereof).
Scorecard with Caddy.
- configure can be run with `--with-test-caddy=path` to specify which caddy to use for testing
- tests/tests-httpd/scorecard.py now measures download speeds with caddy
pytest improvements
- adding Makfile to clean gen dir
- adding nghttpx rundir creation on start
- checking httpd version 2.4.55 for test_05 cases where it is needed. Skipping with message if too old.
- catch exception when checking for caddy existance on system.
Daniel Stenberg [Wed, 1 Feb 2023 08:20:09 +0000 (09:20 +0100)]
CODEOWNERS: remove the peeps mentioned as CI owners
These owners do not have the bandwidth/energy to do the reviews which
makes PRs stall and this ownership claim flawed. We can bring people
back when the situation is different.
Martin D'Aloia [Thu, 2 Feb 2023 01:27:23 +0000 (22:27 -0300)]
write-out.d: add 'since version' to %{header_json} documentation
The documentation of `%{header_json}` missed to mention since which
version this variable for `--write-out` is present.
Based on commit https://github.com/curl/curl/commit/4133a69f2daa476bb
we can determine from the tags were this commit is present that the
first version to include it was `7.83.0`.
This could be also checked with:
`git tag --contains 4133a69f2daa476bb6d902687f1dd6660ea9c3c5`
Daniel Stenberg [Tue, 31 Jan 2023 23:17:28 +0000 (00:17 +0100)]
urlapi: avoid Curl_dyn_addf() for hex outputs
Inspired by the recent fixes to escape.c, we should avoid calling
Curl_dyn_addf() in loops, perhaps in particular when adding something so
simple as %HH codes - for performance reasons. This change makes the
same thing for the URL parser's two URL-encoding loops.
Daniel Stenberg [Wed, 1 Feb 2023 07:26:08 +0000 (08:26 +0100)]
urlapi: skip path checks if path is just "/"
As a miniscule optimization, treat a path of the length 1 as the same as
non-existing, as it can only be a single leading slash, and that's what
we do for no paths as well.
Viktor Szakats [Wed, 1 Feb 2023 09:44:57 +0000 (09:44 +0000)]
tls: fixes for wolfssl + openssl combo builds
1. Add `USE_WOLFSSL` to the TLS backend priority list in
`lib/curl_ntlm_core.c`.
2. Fix `lib/curl_ntlm_core.h` to respect TLS backend priority, bringing
it in sync with the above list and `lib/curl_ntlm_core.c` itself.
Reported-by: Mark Roszko
Ref: https://github.com/curl/curl/issues/10321
3. Allow enabling both wolfSSL and OpenSSL at the same time in
`lib/Makefile.mk` bringing this in line with cmake/autotools builds.
Update logic to select the crypto-specific lib for `ngtcp2`, which
supports a single TLS backend at the same time.
Stefan Eissing [Mon, 30 Jan 2023 15:03:00 +0000 (16:03 +0100)]
cf-socket: improvements in socket I/O handling
- Curl_write_plain/Curl_read_plain have been eliminated. Last code use
now uses Curl_conn_send/recv so that requests use conn->send/revc
callbacks which defaults to cfilters use.
- Curl_recv_plain/Curl_send_plain have been internalized in cf-socket.c.
- USE_RECV_BEFORE_SEND_WORKAROUND (active on Windows) has been moved
into cf-socket.c. The pre_recv buffer is held at the socket filter
context. `postponed_data` structures have been removed from
`connectdata`.
- the hanger in HTTP/2 request handling was a result of read buffering
on all sends and the multi handling is not prepared for this. The
following happens:
- multi preforms on a HTTP/2 easy handle
- h2 reads and processes data
- this leads to a send of h2 data
- which receives and buffers before the send
- h2 returns
- multi selects on the socket, but no data arrives (its in the buffer already)
the workaround now receives data in a loop as long as there is something in
the buffer. The real fix would be for multi to change, so that `data_pending`
is evaluated before deciding to wait on the socket.
io_buffer, optional, in cf-socket.c, http/2 sets state.drain if lower
filter have pending data.
This io_buffer is only available/used when the
-DUSE_RECV_BEFORE_SEND_WORKAROUND is active, e.g. on Windows
configurations. It also maintains the original checks on protocol
handler being HTTP and conn->send/recv not being replaced.
The HTTP/2 (nghttp2) cfilter now sets data->state.drain when it finds
out that the "lower" filter chain has still pending data at the end of
its IO operation. This prevents the processing from becoming stalled.
projects / thirdparty / curl.git / log
