Stefan Eissing [Thu, 23 May 2024 10:21:46 +0000 (12:21 +0200)]
http: write last header line late
- HEADERFUNCTIONS might inspect response properties like
CURLINFO_CONTENT_LENGTH_DOWNLOAD_T on seeing the last header line. If
the line is being written before this is initialized, values are not
available.
- write the last header line late when analyzing a HTTP response so that
all information is available at the time of the writing.
- add test1485 to verify that CURLINFO_CONTENT_LENGTH_DOWNLOAD_T works
on seeing the last header.
Fixes #13752 Reported-by: Harry Sintonen
Closes #13757
Dan Fandrich [Fri, 24 May 2024 21:46:19 +0000 (14:46 -0700)]
tests: use exec when spawning nghttpx
This stops keeping perl and shell processes around that are no longer
needed, plus it eliminates an unneeded shell message when the server is
later terminated.
Viktor Szakats [Thu, 23 May 2024 12:54:49 +0000 (14:54 +0200)]
GHA: stop deleting curl in non-native workflows
We do it in Cirrus CI, but for some platforms it's not possible to
delete it and tests work anyway.
The test runner also runs `../src/curl` by default, which is always the
one freshly built. The runner may also need the system curl to talk to
APIs when needed.
Also:
- stop setting `CURL` env. This isn't picked up by the runners,
and works out of the box anyway.
- quote an option just in case.
Jay Satiro [Fri, 17 May 2024 08:01:35 +0000 (04:01 -0400)]
openssl: stop duplicate ssl key logging for legacy OpenSSL
- Don't call the keylog function if it has already logged the key.
For old OpenSSL versions and its forks that do not have support for
OpenSSL's keylog callback, libcurl has its own legacy key logging
function that logs the TLS 1.2 (and earlier) key (client random + master
key) on a single line.
Prior to this change, since e7de80e8 (precedes 8.8.0), the legacy key
logging function could write the same key line more than once (usually
twice) due to some incorrect logic.
Viktor Szakats [Thu, 23 May 2024 14:44:40 +0000 (16:44 +0200)]
cmake: fix brotli lib order
Fix root cause that caused missing symbols when linking brotli
statically with e.g. binutils `ld` (and any other "picky" linker,
or "traditional" linker as CMake now calls them).
Also drop existing workaround that added brotli libs twice to the lib
list.
```
x86_64-w64-mingw32-ld: .../curl/brotli/_bld/usr/lib/libbrotlidec.a(decode.c.obj):decode.c:(.text$ProcessCommands[ProcessCommands]+0xbb5): undefined reference to `BrotliTransformDictionaryWord'
x86_64-w64-mingw32-ld: .../curl/brotli/_bld/usr/lib/libbrotlidec.a(decode.c.obj):decode.c:(.text$SafeProcessCommands[SafeProcessCommands]+0xe8a): undefined reference to `BrotliTransformDictionaryWord'
x86_64-w64-mingw32-ld: .../curl/brotli/_bld/usr/lib/libbrotlidec.a(decode.c.obj):decode.c:(.rdata$.refptr._kBrotliContextLookupTable[.refptr._kBrotliContextLookupTable]+0x0): undefined reference to `_kBrotliContextLookupTable'
x86_64-w64-mingw32-ld: .../curl/brotli/_bld/usr/lib/libbrotlidec.a(decode.c.obj):decode.c:(.rdata$.refptr._kBrotliPrefixCodeRanges[.refptr._kBrotliPrefixCodeRanges]+0x0): undefined reference to `_kBrotliPrefixCodeRanges'
x86_64-w64-mingw32-ld: .../curl/brotli/_bld/usr/lib/libbrotlidec.a(state.c.obj):state.c:(.text$BrotliDecoderStateInit[BrotliDecoderStateInit]+0x21): undefined reference to `BrotliDefaultAllocFunc'
x86_64-w64-mingw32-ld: .../curl/brotli/_bld/usr/lib/libbrotlidec.a(state.c.obj):state.c:(.text$BrotliDecoderStateInit[BrotliDecoderStateInit]+0x2f): undefined reference to `BrotliDefaultFreeFunc'
x86_64-w64-mingw32-ld: .../curl/brotli/_bld/usr/lib/libbrotlidec.a(state.c.obj):state.c:(.text$BrotliDecoderStateInit[BrotliDecoderStateInit]+0x10e): undefined reference to `BrotliSharedDictionaryCreateInstance'
x86_64-w64-mingw32-ld: .../curl/brotli/_bld/usr/lib/libbrotlidec.a(state.c.obj):state.c:(.text$BrotliDecoderStateCleanup[BrotliDecoderStateCleanup]+0xf4): undefined reference to `BrotliSharedDictionaryDestroyInstance'
collect2: error: ld returned 1 exit status
```
Breakage reproducible with curl-for-win config "`win-gcc`" and deleting
the `LDFLAGS+=' -Wl,--start-group'` line from its `curl.sh` script.
(Above line still required for some non-brotli cases, e.g. libssh2 and
zlib.)
Assisted-by: Kai Pastor
Ref: https://github.com/curl/curl/pull/10857#discussion_r1611714989
Follow-up to 1e3319a167d2f32d295603167486e9e88af9bb4e #10857
Closes #13761
Jay Satiro [Wed, 22 May 2024 06:46:13 +0000 (02:46 -0400)]
winbuild: remove outdated WIN32 defines
- Remove all instances in the makefile of compiler option /DWIN32.
This is a follow-up to e9a7d4a1 which replaced all defined(WIN32) checks
with defined(_WIN32) in the codebase, since only the latter is
automatically defined by all compilers for Windows builds.
Bug: https://github.com/curl/curl/pull/13739#issuecomment-2123937859 Reported-by: Viktor Szakats
Closes https://github.com/curl/curl/pull/13742
Pavel P [Tue, 21 May 2024 23:23:34 +0000 (01:23 +0200)]
asyn-thread: avoid using GetAddrInfoExW with impersonation
Multiple reports suggest that GetAddrInfoExW fails when impersonation is
used. This PR checks if thread is impersonating and avoids using
GetAddrInfoExW api.
Stefan Eissing [Wed, 22 May 2024 14:52:16 +0000 (16:52 +0200)]
transfer: conn close on paused upload
- add 2 variations on test_07_42 which PAUSEs uploads
and response connections terminating either right away
or after the 100-continue response
- when detecting the connection being closed in transfer.c
readwrite_data(), clear ALL send bits in data->req.keepon.
It no longer makes send to wait for a KEEP_SEND_PAUSE or HOLD.
- in the protocol client writer add the check for incomplete
response bodies. When an EOS is seen and the length is known,
check that and fail if bytes are missing.
Stefan Eissing [Wed, 22 May 2024 08:35:51 +0000 (10:35 +0200)]
mbedtls: send close-notify on close
- send the TLS close notify message when cloding down
the mbedtls connection filter
- this is a "least" effort version and, as other TLS filters,
is lacking a graceful send/receive/timeout for a really
clean shutdown.
Stefan Eissing [Tue, 21 May 2024 10:17:33 +0000 (12:17 +0200)]
pytest: add ftp upload tests
- refs #13556
- allow anon uploads on vsftpd test server
- add test_30_05 for plain upload of 1k, 100k, 1m
- add test_31_05 for SSL upload of 1k, 100k, 1m
- verify file size and contents
Jay Satiro [Wed, 22 May 2024 05:39:44 +0000 (01:39 -0400)]
winbuild: fix PE version info debug flag
- Only set PE file flag VS_FF_DEBUG if curl.exe and libcurl.dll were
built with winbuild option DEBUG=yes which builds with debug info.
VS_FF_DEBUG is a PE flag (Portable Executable file flag - dll, exe, etc)
that indicates the file contains or was built with debug info.
Prior to this change when winbuild was used to build curl, curl.exe
and libcurl.dll always had VS_FF_DEBUG set, regardless of build option
DEBUG=yes/no, due to some bad logic.
Stefan Eissing [Tue, 21 May 2024 09:21:14 +0000 (11:21 +0200)]
http: HEAD response body tolerance
- as reported in #13725, some servers wrongly send body bytes in
responses to a HEAD request. This used to be tolerated in curl
8.4 and before and leads to failed transfers in newer versions.
- restore previous behaviour for HTTP/1.1 and HTTP/2:
* 1.1: do not add 'Transfer-Encoding' writers from HEAD
responses. RFC 9112 says they do not apply.
* 2: when the transfer expects 'no_body', to not report stream
resets as error when all response headers have been received.
Jay Satiro [Fri, 17 May 2024 04:07:21 +0000 (00:07 -0400)]
openssl: revert keylog_callback support for LibreSSL
- Revert to the legacy TLS 1.2 key logging code for LibreSSL.
- Document SSLKEYLOGFILE for LibreSSL is TLS 1.2 max.
Prior to this change if the user specified a filename in the
SSLKEYLOGFILE environment variable and was using LibreSSL 3.5.0+ then
an empty file would be created and no keys would be logged.
This is effectively a revert of e43474b4 which changed openssl.c to use
SSL_CTX_set_keylog_callback for LibreSSL 3.5.0+. Unfortunately LibreSSL
added that function only as a stub that doesn't actually do anything.
Viktor Szakats [Fri, 10 May 2024 13:29:21 +0000 (15:29 +0200)]
GHA: add NetBSD, OpenBSD, FreeBSD/arm64 and OmniOS jobs
Add these jobs to GHA:
- NetBSD, cmake-unity, clang, OpenSSL, x86_64, with tests, w/o python,
no parallelism (was flaky sometimes)
- OpenBSD, cmake-unity, clang, LibreSSL, x86_64, with tests,
with python, -j8, TFTP results ignored due to #13623.
- FreeBSD, cmake-unity and autotools, clang, OpenSSL, arm64
(Tests disabled for arm64, because they are slow. It's available for
x86_64 with python, -j12.)
Configuration matches our existing Cirrus CI one.
- OmniOS, autotools, gcc, OpenSSL, x86_64, with tests, -j12.
Viktor Szakats [Sat, 18 May 2024 12:37:57 +0000 (14:37 +0200)]
cmake: do not pass linker flags to the static library tool
Do not add linker flags to the global CMake static library tool (aka
"static linker") (e.g. `ar`) flags list. They don't mix well. This was
only done after successfully detecting GSSAPI.
Linker flags seen on Old Linux CI:
```
-- |GSS_LINKER_FLAGS|-Wl,--enable-new-dtags -Wl,-rpath -Wl,/usr/lib/x86_64-linux-gnu/heimdal|
-- |CMAKE_STATIC_LINKER_FLAGS| -Wl,--enable-new-dtags -Wl,-rpath -Wl,/usr/lib/x86_64-linux-gnu/heimdal|
```
Ref: https://github.com/curl/curl/actions/runs/9138988036/job/25130791712#step:6:85
This problem is invisible at the moment because of another bug (#13698)
that misses building unit tests when not using either the
`ENABLE_DEBUG=ON` or `ENABLE_CURLDEBUG=ON` options (to set
`-DCURLDEBUG`):
```
test 1300 SKIPPED: curl lacks unittest support
```
Ref: https://github.com/curl/curl/actions/runs/9135571781/job/25123104557#step:9:2883
With that fixed, this becomes the next issue.
It's possible this bug also required an older CMake version and/or
a specific OS environment which uses linker flags in GSSAPI that are not
playing well with `ar` options, to reproduce.
Viktor Szakats [Sat, 11 May 2024 13:29:23 +0000 (15:29 +0200)]
GHA: add MSYS, mingw-w64, Cygwin jobs
- re-implement autotools MSYS and Cygwin AppVeyor jobs in GHA.
Now build with SSL and PSL to improve test coverage.
- re-implement MSYS2 mingw-w64 gcc 13 AppVeyor job in GHA.
`CMake, mingw-w64, gcc 13, Debug, x64, Schannel, Static, Unicode`
- add new cmake Cygwin job (build-only).
- enable `-j14` parallelism when running tests.
- delete the 5 migrated jobs from AppVeyor CI.
- add 2 build-only mingw-w64 builds, gcc Release and clang OpenSSL.
- also enable brotli, libssh2, nghttp2 for more test coverage.
These jobs offer better performance, more flexibility and
parallelization compared to the AppVeyor ones they replace. It also
offloads AppVeyor, allowing to iterate faster. They also appear more
reliable than e.g. Azure Windows jobs, where runners are prone to fail
[1].
Stefan Eissing [Wed, 15 May 2024 12:20:11 +0000 (14:20 +0200)]
pytest: fixes for recent python, add FTP tests
Fixes:
- in uds tests, abort also silently on os errors
- be conservative on the h3 goaway duration
- detect curl debug build and use in checks
- fix caddy version check for slight difference under linux
- set caddy default path fitting for linux
- fix deprecation warnings in valid time checks
FTP tests:
- add '--with-test-vsftpd=path' to configure
- use vsftpd default path suitable for linux
- add test_30 with plain FTP tests
- add test_31 with --ssl-reqd FTP tests
- add vsftpd to linux GHA for pytest workflows
Stefan Eissing [Fri, 17 May 2024 11:23:19 +0000 (13:23 +0200)]
rustls: fix handshake done handling
- rustls report it has finished the TLS handshake *before*
all relevant data has been sent off, e.g. it FINISHED message
- On connections the send data immediately, this was never noticed
as the FINISHED in rustls buffers was send with the app data
- On passive FTP connections, curl does not send any data after
the handshake, leaving FINISHED unsent and the server never
responded as it was waiting on this.
Viktor Szakats [Sat, 11 May 2024 21:38:25 +0000 (23:38 +0200)]
lib: fix compiler warnings (gcc)
Seen when setting `ENABLE_DEBUG=ON` and `-DDEBUGBUILD` for mingw-w64
gcc 13.2.0 CMake unity builds in 'Release' configurations.
```
curl/lib/curl_gethostname.c:71:5: error: 'strncpy' specified bound 1025 equals destination size [-Werror=stringop-truncation]
71 | strncpy(name, force_hostname, namelen);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
In file included from curl/_bld/lib/CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c:175:
In function 'hostcache_timestamp_remove',
inlined from 'Curl_hash_clean_with_criterium' at curl/lib/hash.c:265:19,
inlined from 'Curl_hash_clean_with_criterium' at curl/lib/hash.c:247:1,
inlined from 'hostcache_prune' at curl/lib/hostip.c:228:3,
inlined from 'Curl_hostcache_prune' at curl/lib/hostip.c:256:21:
curl/lib/hostip.c:205:12: error: 'now' may be used uninitialized [-Werror=maybe-uninitialized]
205 | time_t age = prune->now - c->timestamp;
| ^~~
curl/lib/hostip.c: In function 'Curl_hostcache_prune':
curl/lib/hostip.c:241:10: note: 'now' was declared here
241 | time_t now;
| ^~~
In function 'hostcache_timestamp_remove',
inlined from 'fetch_addr' at curl/lib/hostip.c:310:8:
curl/lib/hostip.c:205:23: error: 'user.now' may be used uninitialized [-Werror=maybe-uninitialized]
205 | time_t age = prune->now - c->timestamp;
| ~~~~~^~~~~
curl/lib/hostip.c: In function 'fetch_addr':
curl/lib/hostip.c:304:33: note: 'user' declared here
304 | struct hostcache_prune_data user;
| ^~~~
In file included from curl/_bld/lib/CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c:40:
curl/lib/cf-socket.c: In function 'cf_socket_send':
curl/lib/cf-socket.c:1294:10: error: 'c' may be used uninitialized [-Werror=maybe-uninitialized]
1294 | if(c >= ((100-ctx->wblock_percent)*256/100)) {
| ~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
curl/lib/cf-socket.c:1292:19: note: 'c' was declared here
1292 | unsigned char c;
| ^
In file included from curl/_bld/lib/CMakeFiles/libcurl_object.dir/Unity/unity_0_c.c:364:
In function 'tftp_state_timeout',
inlined from 'tftp_multi_statemach' at curl/lib/tftp.c:1230:27:
curl/lib/tftp.c:1208:5: error: 'current' may be used uninitialized [-Werror=maybe-uninitialized]
1208 | if(current > state->rx_time + state->retry_time) {
| ^
curl/lib/tftp.c: In function 'tftp_multi_statemach':
curl/lib/tftp.c:1192:10: note: 'current' was declared here
1192 | time_t current;
| ^~~~~~~
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/49792835/job/91c8dj5qb36spfe0#L112
Ref: https://github.com/curl/curl/actions/runs/9082968838/job/24960616145#step:12:62
Stefan Eissing [Thu, 16 May 2024 12:59:25 +0000 (14:59 +0200)]
rustls: fix partial send handling
When TLS bytes could not completely sent off, the amount of plain bytes
already added to rustls were forgotten. This lead to send those byte
duplicate, corrupting the request send to the server.
Viktor Szakats [Thu, 16 May 2024 00:56:37 +0000 (02:56 +0200)]
cmake: initialize `BUILD_TESTING` before first use
Before this patch `BUILD_TESTING` was used once, then initialized, then
used again. This caused the `curlu` library not being built when relying
on an implicit `BUILD_TESTING=ON` setting, and ending up with a link
error when building the `testdeps` target.
It did not cause issues when `BUILD_TESTING` was explicitly set.
Move the initialization before the first use to fix it.
Max Dymond [Thu, 16 May 2024 08:02:36 +0000 (09:02 +0100)]
ci: disable Renovate dashboard
The Renovate dashboard insists on an open issue,
which is a problem. Disable the dashboard. Status
can still be seen at https://developer.mend.io/github/curl/curl.
Max Dymond [Tue, 14 May 2024 16:16:14 +0000 (17:16 +0100)]
ci: fix renovate config for WolfSSL/WolfSSH tagging scheme
WolfSSL/WolfSSH use a different versioning scheme;
stable builds end with `-stable`. Renovate requires
some extra configuration to extract the version
from these types of tags.
Daniel Stenberg [Mon, 6 May 2024 21:27:41 +0000 (23:27 +0200)]
DEPRECATE.md: TLS libraries without 1.3 support
curl drops support for TLS libraries without TLS 1.3 capability after
May 2025.
It requires that a curl build using the library should be able to
negotiate and use TLS 1.3, or else it is not good enough. We support a
vast amount of other TLS libraries that are likely to satisfy users
better.
autotools: Only probe for SGI MIPS compilers on IRIX
MIPSPro and the predecessor compiler which was part of the IDO (IRIS
Development Option) were only ever shipped on the SGI IRIX operating
system (with MIPSPro on 6.0+ which was released in 1994). Limit the
autoconf check to IRIX when probing for these compilers to save some
cycles on other platforms.
Closes: #13611 Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Viktor Szakats [Mon, 13 May 2024 20:45:56 +0000 (22:45 +0200)]
tests: fix test 1167 to skip digit-only symbols
This avoids mistaking symbols with their numeric value when using
certain C preprocessors which output these numeric values at the
beginning of the line as part of an expression.
Seen on OpenBSD 7.5 + clang.
Example `test1167.pl -v` output, before this patch:
```
Source: cpp /home/runner/work/curl/curl/tests/../include/curl/curl.h
Symbol: 20000
Line #3835: 20000 + 142,
[...]
Bad symbols in public header files:
20000
[...]
```
Ref: https://github.com/curl/curl/actions/runs/9069136530/job/24918015357#step:3:7513
projects / thirdparty / curl.git / log
