Casey Bodley [Wed, 15 Feb 2023 15:47:04 +0000 (10:47 -0500)]
aws_sigv4: fall back to UNSIGNED-PAYLOAD for sign_as_s3
all s3 requests default to UNSIGNED-PAYLOAD and add the required
x-amz-content-sha256 header. this allows CURLAUTH_AWS_SIGV4 to correctly
sign s3 requests to amazon with no additional configuration
Viktor Szakats [Tue, 14 Mar 2023 11:57:31 +0000 (11:57 +0000)]
wolfssl: add quic/ngtcp2 detection in cmake, and fix builds
- add QUIC/ngtcp2 detection in CMake with wolfSSL.
Because wolfSSL uses zlib if available, move compression detection
before TLS detection. (OpenSSL might also need this in the future.)
- wolfSSL 5.5.0 started using C99 types in its `quic.h` header, but it
doesn't #include the necessary C99 header itself, breaking builds
(unless another dependency pulled it by chance.) Add local workaround
for it. For this to work with all build tools, we had to fix our
header detection first. Ref: #10745
Stefan Eissing [Thu, 9 Mar 2023 09:50:55 +0000 (10:50 +0100)]
tests/http: add pytest to GHA and improve tests
- added to: ngtcp2-quictls, ngtcp2-gnutls and the linux varians
quiche, bearssl, libressl, mbedtls, openssl3, rustls
- added disabled in ngtcp2-wolfssl due to weird SSL_connect() errors
not reproducable locally
Improvements on pytest:
- handling of systems with nghttpx in $PATH
- configure will seach $PATH got nghttpx used in pytest
- pytest fixes for managing nghttpx without h3 support
- ngtcp2-wolfssl: use a fully enabled wolfssl build
- lower parallel count for http/1.1 tests, since we do not
want to test excessive connections.
- check built curl for HTTPS-proxy support in proxy tests
- bearssl does not like one of our critical cert extensions, making
it non-critical now
- bearssl is too slow for test_12, skipping
- making sure we do h3 tests only when curl and server support is there
Viktor Szakats [Mon, 13 Mar 2023 10:46:38 +0000 (10:46 +0000)]
build: fix stdint/inttypes detection with non-autotools
Fix `stdint.h` and `inttypes.h` detection with non-autotools builds on
Windows. (autotools already auto-detected them accurately.)
`lib/config-win32.h` builds (e.g. `Makefile.mk`):
- set `HAVE_STDINT_H` where supported.
- set `HAVE_INTTYPES_H` for MinGW.
CMake:
- auto-detect them on Windows. (They were both force-disabled.)
- delete unused `CURL_PULL_STDINT_H`.
- delete unused `CURL_PULL_INTTYPES_H`.
- stop detecting `HAVE_STDINT_H` twice.
Present since the initial CMake commit: 4c5307b45655ba75ab066564afdc0c111a8b9291
curl doesn't use these C99 headers, we need them now to workaround
broken wolfSSL builds. Ref: #10739
Once that clears up, we can delete these detections and macros (unless
we want to keep them for future us.)
Dan Fandrich [Mon, 13 Mar 2023 01:56:22 +0000 (18:56 -0700)]
tests: make first.c the same for both lib tests and unit tests
The only difference used to be global variable used in unittest tests.
After cb7ed5a removed individual flag overrides for the unittests, first.c
was no longer recompiled for unit tests to include the flag, so whether it
worked or gave a link error depended on whether it was compiled in
libtest or unittest first. This way also speeds up the build by
eliminating 40 identical compile invocations.
Jay Satiro [Sat, 4 Mar 2023 09:07:24 +0000 (04:07 -0500)]
tool: improve --stderr handling
- freopen stderr with the user-specified file (--stderr file) instead of
using a separate 'errors' stream.
- In tool_setup.h override stdio.h's stderr macro as global variable
tool_stderr.
Both freopen and overriding the stderr macro are necessary because if
the user-specified filename is "-" then stdout is assigned to
tool_stderr and no freopen takes place. See the PR for more information.
Dan Fandrich [Fri, 10 Mar 2023 21:15:43 +0000 (13:15 -0800)]
http: don't send 100-continue for short PUT requests
This is already how curl is documented to behave in Everything curl, but
in actuality only short POSTs skip this. This should knock 30 seconds
off a full run of the test suite since the 100-continue timeout will no
longer be hit.
Dan Fandrich [Thu, 9 Mar 2023 21:17:44 +0000 (13:17 -0800)]
tests: hack to build most unit tests under cmake
These are only built when a libcurl static library is available, since
we're not building a special libcurlu library yet and these tests rely
on private symbols that aren't available in the shared library. A few
unit tests do require libcurlu, so those are not built.
Dan Fandrich [Thu, 9 Mar 2023 20:23:14 +0000 (12:23 -0800)]
tests: keep cmake unit tests names in sync
Put only the test names into Makefile.inc so they can be used by both
cmake and automake. This will prevent the list of tests from becoming
out of date when they are also built under cmake.
Viktor Szakats [Fri, 10 Mar 2023 22:24:24 +0000 (22:24 +0000)]
lib: silence clang/gcc -Wvla warnings in brotli headers
brotli v1.0.0 throughout current latest v1.0.9 and latest master [1]
trigger this warning.
It happened with CMake and GNU Make. autotools builds avoid it with
the `convert -I options to -isystem` macro.
llvm/clang:
```
In file included from ./curl/lib/content_encoding.c:36:
./brotli/x64-ucrt/usr/include/brotli/decode.h:204:34: warning: variable length array used [-Wvla]
const uint8_t encoded_buffer[BROTLI_ARRAY_PARAM(encoded_size)],
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./brotli/x64-ucrt/usr/include/brotli/port.h:253:34: note: expanded from macro 'BROTLI_ARRAY_PARAM'
^~~~~~
In file included from ./curl/lib/content_encoding.c:36:
./brotli/x64-ucrt/usr/include/brotli/decode.h:206:48: warning: variable length array used [-Wvla]
uint8_t decoded_buffer[BROTLI_ARRAY_PARAM(*decoded_size)]);
~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~~~~
./brotli/x64-ucrt/usr/include/brotli/port.h:253:35: note: expanded from macro 'BROTLI_ARRAY_PARAM'
~^~~~~
```
gcc:
```
In file included from ./curl/lib/content_encoding.c:36:
./brotli/x64-ucrt/usr/include/brotli/decode.h:204:5: warning: ISO C90 forbids variable length array 'encoded_buffer' [-Wvla]
204 | const uint8_t encoded_buffer[BROTLI_ARRAY_PARAM(encoded_size)],
| ^~~~~
./brotli/x64-ucrt/usr/include/brotli/decode.h:206:5: warning: ISO C90 forbids variable length array 'decoded_buffer' [-Wvla]
206 | uint8_t decoded_buffer[BROTLI_ARRAY_PARAM(*decoded_size)]);
| ^~~~~~~
```
Daniel Stenberg [Tue, 7 Mar 2023 23:33:33 +0000 (00:33 +0100)]
headers: make curl_easy_header and nextheader return different buffers
By letting curl_easy_header() and curl_easy_nextheader() store the
header data in their own struct storage when they return a pointer to
it, it makes it possible for applications to use them both in a loop.
Like the curl tool does.
Reported-by: Boris Okunskiy
Fixes #10704
Closes #10707
Current test for curl_free() may produce warnings with strict compiler
flags or even with default compiler flags with upcoming versions.
These warning could turned into errors by -Werror or similar flags.
Such warnings/errors are avoided by this patch.
Stefan Eissing [Mon, 6 Mar 2023 11:44:45 +0000 (12:44 +0100)]
url: fix logic in connection reuse to deny reuse on "unclean" connections
- add parameter to `conn_is_alive()` cfilter method that returns
if there is input data waiting on the connection
- refrain from re-using connnection from the cache that have
input pending
- adapt http/2 and http/3 alive checks to digest pending input
to check the connection state
- remove check_cxn method from openssl as that was just doing
what the socket filter now does.
- add tests for connection reuse with special server configs
Stefan Eissing [Mon, 6 Mar 2023 16:16:01 +0000 (17:16 +0100)]
http2: fix handling of RST and GOAWAY to recognize partial transfers
- a reset transfer (HTTP/2 RST) did not always lead to the proper
error message on receiving its response, leading to wrong reports
of a successful transfer
- test_05_02 was able to trigger this condition with increased transfer
count. The simulated response errors did not carry a 'Content-Length'
so only proper RST handling could detect the abort
- When doing such transfers in parallel, a connection could enter the
state where
a) it had been closed (GOAWAY received)
b) the RST had not been "seen" for the transfer yet
or c) the GOAWAY announced an error and the last successful
stream id was not checked against ongoing transfers
Stefan Eissing [Mon, 6 Mar 2023 15:11:11 +0000 (16:11 +0100)]
tests: use dynamic ports numbers in pytest suite
- necessary ports are bound at start of test suite and then
given to server fixtures for use.
- this make parallel use of pytest (in separate directories),
practically safe for use as OS tend to not reuse such port numbers
for a while
Stefan Eissing [Fri, 3 Mar 2023 16:54:44 +0000 (17:54 +0100)]
connect: fix time_connect and time_appconnect timer statistics
- time_connect was not updated when the overall connection failed,
e.g. when SSL verification was unsuccessful, refs #10670
- rework gather those values to interrogate involved filters,
also from all eyeballing attempts, to report the maximum of
those values.
- added 3 test cases in test_06 to check reported values on
successful, partially failed and totally failed connections.
Harry Sintonen [Sat, 4 Mar 2023 07:02:14 +0000 (09:02 +0200)]
rand: use arc4random as fallback when available
Normally curl uses cryptographically strong random provided by the
selected SSL backend. If compiled without SSL support, a naive built-in
function was used instead.
Generally this was okay, but it will result in some downsides for non-
SSL builds, such as predictable temporary file names.
This change ensures that arc4random will be used instead, if available.
Grisha Levit [Sat, 4 Mar 2023 21:32:56 +0000 (16:32 -0500)]
tool: dump headers even if file is write-only
The fixes in #10079 brought a (seemingly unrelated) change of open mode
from `wb`/`ab` to `wb+`/`ab+` for the headerfile. This makes it no
longer possible to write the header file to e.g. a pipe, like:
curl -D >(grep ...) file:///dev/null
Which presently results in `Warning: Failed to open /dev/fd/63`
Viktor Szakats [Sun, 5 Mar 2023 19:55:14 +0000 (19:55 +0000)]
cmake: fix enabling LDAPS on Windows
Before this patch, enabling LDAPS required a manual C flag:
https://github.com/curl/curl-for-win/blob/c1cfc31cfc04f24f7a4f946564d6f0e1b4d7dd36/curl-cmake.sh#L105
Fix this and enable LDAPS automatically when using `wldap32` (and
when not explicitly disabled). This matches autotools and `Makefile.mk`
behavior. Also remove issue from KNOWN_BUGS.
Add workaround for MSVS 2010 warning triggered by LDAPS now enabled
in more CI tests:
`ldap.c(360): warning C4306: 'type cast' : conversion from 'int' to 'void *' of greater size`
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/46408284/job/v8mwl9yfbmoeqwlr#L312
Reported-by: JackBoosY on github Reviewed-by: Jay Satiro Reviewed-by: Marcel Raad
Fixes #6284
Closes #10674
For OpenLDAP builds this macro is not enough to enable LDAPS, and
OpenLDAP is not an option in `Makefile.mk`. For Novell LDAP it might
have helped, but it's also not an option anymore in `Makefile.mk`.
The future for LDAPS is that we should enable it by default without
extra build knobs.
Jan Engelhardt [Thu, 2 Mar 2023 15:08:22 +0000 (16:08 +0100)]
build: drop the use of XC_AMEND_DISTCLEAN
Because automake used to delete depdirs at once (.deps) and there was an issue
with portability, curl's XC_AMEND_DISTCLEAN greps the Makefiles in an attempt
to build a list of all depfiles and delete them individually instead.
Since commit 08849db866b44510f6b8fd49e313c91a43a3dfd3, automake switched from
deleting directories to individual files. curl's custom logic now finds a lot
more results with the grep (the filtering of these results isn't great), which
causes a massive bloating of the Makefile in the order of O(n^2).
Also remove now-unused XC_AMEND_DISTCLEAN macro group
Stefan Eissing [Wed, 1 Mar 2023 12:05:09 +0000 (13:05 +0100)]
socket: detect "dead" connections better, e.g. not fit for reuse
- refs #10646 where reuse was attempted on closed connections in the
cache, leading to an exhaustion of retries on a transfer
- the mistake was that poll events like POLLHUP, POLLERR, etc
were regarded as "not dead".
- change cf-socket filter check to regard such events as inidication
of corpsiness.
- vtls filter checks: fixed interpretation of backend check result
when inconclusive to interrogate status further down the filter
chain.
Reported-by: SendSonS on github
Fixes #10646
Closes #10652
cf-socket: if socket is already connected, return CURLE_OK
In 7.87.0, if callback method for CURLOPT_SOCKOPTFUNCTION returns
CURL_SOCKOPT_ALREADY_CONNECTED then curl library used to return
CURLE_OK. n 7.88.0, now even if callback returns
CURL_SOCKOPT_ALREADY_CONNECTED, curl library still tries to connect to
socket by invoking method do_connect().
This is regression caused by commit
https://github.com/curl/curl/commit/71b7e0161032927cdfb
Fix: Check if we are already connected and return CURLE_OK.
projects / thirdparty / curl.git / log
