]> git.ipfire.org Git - thirdparty/curl.git/log
thirdparty/curl.git
19 months agomprintf: overhaul and bugfixes
Daniel Stenberg [Fri, 22 Dec 2023 08:51:20 +0000 (09:51 +0100)] 
mprintf: overhaul and bugfixes

In a test case using lots of snprintf() calls using many commonly used
%-codes per call, this version is around 30% faster than previous
version.

It also fixes the #12561 bug which made it not behave correctly when
given unknown %-sequences. Fixing that flaw required a different take on
the problem, which resulted in the new two-arrays model.

lib557: extended - Verify the #12561 fix and test more printf features

unit1398: fix test: It used a <num>$ only for one argument, which is not
supported.

Fixes #12561
Closes #12563

19 months agoappveyor: replace PowerShell with bash + parallel autotools
Viktor Szakats [Thu, 21 Dec 2023 03:59:20 +0000 (03:59 +0000)] 
appveyor: replace PowerShell with bash + parallel autotools

PowerShell works (after a steep development curve), but one property of
it stuck and kept causing unresolvable usability issues: With
`$ErrorActionPreference=Stop`, it does abort on failures, but shows only
the first line of the error message. In `Continue` mode, it shows the
full error message, but doesn't stop on all errors. Another issue is
PowerShell considering any stderr output as if the command failed (this
has been improved in 7.2 (2021-Nov), but fixed versions aren't running
in CI and will not be for a long time in all test images.)

Thus, we're going with bash.

Also:
- use `-j2` with autotools tests, making them finish 5-15 minutes per
  job faster.
- omit `POSIX_PATH_PREFIX`.
- use `WINDIR`.
- prefer forward slashes.

Follow-up to: 75078a415d9c769419aed4153d3d525a8eba95af #11999
Ref: #12444

Fixes #12560
Closes #12572

19 months agoasyn-thread: use GetAddrInfoExW on >= Windows 8
Pavel P [Fri, 17 Nov 2023 08:56:08 +0000 (10:56 +0200)] 
asyn-thread: use GetAddrInfoExW on >= Windows 8

For doing async DNS resolution instead of starting a thread for each
request.

Fixes #12481
Closes #12482

19 months agostrerror: repair get_winsock_error()
Daniel Stenberg [Thu, 21 Dec 2023 16:50:29 +0000 (17:50 +0100)] 
strerror: repair get_winsock_error()

It would try to read longer than the provided string and crash.

Follow-up to ff74cef5d4a0cf60106517a1c7384
Reported-by: calvin2021y on github
Fixes #12578
Closes #12579

19 months agoCURLOPT_SSH_*_KEYFILE: clarify
Daniel Stenberg [Tue, 19 Dec 2023 07:48:21 +0000 (08:48 +0100)] 
CURLOPT_SSH_*_KEYFILE: clarify

Closes #12554

19 months agongtcp2: put h3 at the front of alpn
ivanfywang [Thu, 21 Dec 2023 09:06:47 +0000 (17:06 +0800)] 
ngtcp2: put h3 at the front of alpn

Closes #12576

19 months agotest460: verify a command line using --expand with no argument
Daniel Stenberg [Thu, 21 Dec 2023 08:52:24 +0000 (09:52 +0100)] 
test460: verify a command line using --expand with no argument

This verifies the fix for #12565

19 months agotool_getparam: do not try to expand without an argument
Daniel Stenberg [Thu, 21 Dec 2023 08:50:00 +0000 (09:50 +0100)] 
tool_getparam: do not try to expand without an argument

This would lead to a segfault.

Fixes #12565
Reported-by: Geeknik Labs
Closes #12575

19 months agoRELEASE-NOTES: synced
Daniel Stenberg [Thu, 21 Dec 2023 10:33:02 +0000 (11:33 +0100)] 
RELEASE-NOTES: synced

Bumped version to 8.6.0 because of changes

19 months agoMakefile.am: fix the MSVC project generation
Daniel Stenberg [Wed, 20 Dec 2023 22:10:38 +0000 (23:10 +0100)] 
Makefile.am: fix the MSVC project generation

It made the vcxproj files not get included in dist tarballs.

Regression since 74423b5df4c8117891eb89 (8.5.0)

Reported-by: iAroc on github
Fixes #12564
Closes #12567

19 months agoaltsvc: free 'as' when returning error
zengwei2000 [Thu, 21 Dec 2023 02:09:21 +0000 (02:09 +0000)] 
altsvc: free 'as' when returning error

Closes #12570

Signed-off-by: zengwei <zengwei1@uniontech.com>
19 months agobuild: fix `-Wconversion`/`-Wsign-conversion` warnings 12571/head 12574/head
Viktor Szakats [Tue, 19 Dec 2023 19:16:03 +0000 (19:16 +0000)] 
build: fix `-Wconversion`/`-Wsign-conversion` warnings

Fix remaining warnings in examples and tests which are not suppressed
by the pragma in `lib/curl_setup.h`.

Silence a toolchain issue causing warnings in `FD_SET()` calls with
older Cygwin/MSYS2 builds. Likely fixed on 2020-08-03 by:
https://cygwin.com/git/?p=newlib-cygwin.git;a=commitdiff;h=5717262b8ecfed0f7fab63e2c09c78991e36f9dd

Follow-up to 2dbe75bd7f3c36837aa06fd87a442bdf3fb7faef #12492

Closes #12557

19 months agobuild: fix some `-Wsign-conversion`/`-Warith-conversion` warnings
Viktor Szakats [Sat, 9 Dec 2023 02:45:19 +0000 (02:45 +0000)] 
build: fix some `-Wsign-conversion`/`-Warith-conversion` warnings

- enable `-Wsign-conversion` warnings, but also setting them to not
  raise errors.
- fix `-Warith-conversion` warnings seen in CI.
  These are triggered by `-Wsign-converion` and causing errors unless
  explicitly silenced. It makes more sense to fix them, there just a few
  of them.
- fix some `-Wsign-conversion` warnings.
- hide `-Wsign-conversion` warnings with a `#pragma`.
- add macro `CURL_WARN_SIGN_CONVERSION` to unhide them on a per-build
  basis.
- update a CI job to unhide them with the above macro:
  https://github.com/curl/curl/actions/workflows/linux.yml -> OpenSSL -O3

Closes #12492

19 months agocmake: tidy-up `OtherTests.cmake`
Viktor Szakats [Mon, 18 Dec 2023 15:40:33 +0000 (15:40 +0000)] 
cmake: tidy-up `OtherTests.cmake`

- make more obvious which detection uses which prep steps.
- merge and streamline conditions.
- these should not alter detection results.

Also align log output messages from
`Macros.cmake` / `curl_internal_test` with rest of the build.

Closes #12551

19 months agoappveyor: switch to out-of-tree builds
Viktor Szakats [Mon, 18 Dec 2023 15:21:21 +0000 (15:21 +0000)] 
appveyor: switch to out-of-tree builds

With cmake and autotools.

Closes #12550

19 months agoDEPRECATE.md: mention that NTLM_WB no longer works
Daniel Stenberg [Tue, 19 Dec 2023 07:21:48 +0000 (08:21 +0100)] 
DEPRECATE.md: mention that NTLM_WB no longer works

Ref: #12479
Closes #12553

19 months agoCURLOPT_SERVER_RESPONSE_TIMEOUT_MS: add
Daniel Stenberg [Mon, 20 Nov 2023 16:22:40 +0000 (17:22 +0100)] 
CURLOPT_SERVER_RESPONSE_TIMEOUT_MS: add

Proposed-by: Yifei Kong
Ref: https://curl.se/mail/lib-2023-11/0023.html
Closes #12369

19 months agobuild: more `-Wformat` fixes
Viktor Szakats [Sat, 16 Dec 2023 17:23:40 +0000 (17:23 +0000)] 
build: more `-Wformat` fixes

- memdebug: update to not trigger `-Wformat-nonliteral` warnings.
- imap: mark `imap_sendf()` with  `CURL_PRINTF()`.
- tool_msgs: mark static function with `CURL_PRINTF()`.

Follow-up to 3829759bd042c03225ae862062560f568ba1a231 #12489

Closes #12540

19 months agowindows: delete redundant headers
Viktor Szakats [Tue, 12 Dec 2023 19:27:44 +0000 (19:27 +0000)] 
windows: delete redundant headers

`winsock2.h` pulls in `windows.h`. `ws2tcpip.h` pulls in `winsock2.h`.
`winsock2.h` and `ws2tcpip.h` are also pulled by `curl/curl.h`.

Keep only those headers that are not already included, or the code under
it uses something from that specific header.

Closes #12539

19 months agocmake: prefill/cache `HAVE_STRUCT_SOCKADDR_STORAGE`
Viktor Szakats [Tue, 12 Dec 2023 22:00:59 +0000 (22:00 +0000)] 
cmake: prefill/cache `HAVE_STRUCT_SOCKADDR_STORAGE`

Also add missing include to `OtherTests.cmake`. It didn't cause an issue
because the parent already included this earlier by chance.

Closes #12537

19 months agorunner.pm: fix perl warning when running tests
Daniel Stenberg [Mon, 18 Dec 2023 13:19:22 +0000 (14:19 +0100)] 
runner.pm: fix perl warning when running tests

    Use of uninitialized value $runner::gdbthis in numeric eq (==) at runner.pm

Follow-up from 3dcf301752a09d9

Closes #12549

19 months agoruntests: support -gl. Like -g but for lldb.
Daniel Stenberg [Mon, 18 Dec 2023 09:11:30 +0000 (10:11 +0100)] 
runtests: support -gl. Like -g but for lldb.

Follow-up to 63b5748

Invokes the test case via lldb instead of gdb. Since using gdb is such a
pain on mac, using lldb is sometimes less quirky.

Closes #12547

19 months agocurl.h: add CURLE_TOO_LARGE
Daniel Stenberg [Mon, 18 Dec 2023 09:34:17 +0000 (10:34 +0100)] 
curl.h: add CURLE_TOO_LARGE

A new error code to be used when an internal field grows too large, like
when a dynbuf reaches its maximum. Previously it would return
CURLE_OUT_OF_MEMORY for this, which is highly misleading.

Ref: #12268
Closes #12269

19 months agoCI/circleci: disable MQTT in the HTTP-only build
Daniel Stenberg [Mon, 18 Dec 2023 08:34:39 +0000 (09:34 +0100)] 
CI/circleci: disable MQTT in the HTTP-only build

And remove the use of configure options that don't actually exist

Closes #12546

19 months agotests: respect $TMPDIR when creating unix domain sockets
Yedaya Katsman [Sun, 17 Dec 2023 20:04:25 +0000 (22:04 +0200)] 
tests: respect $TMPDIR when creating unix domain sockets

When running on termux, where $TMPDIR isn't /tmp, running the tests
failed, since the server config tried creating sockets in /tmp, without
checking the temp dir config. Use the TMPDIR variable that makes it find
the correct directory everywhere [0]

[0] https://perldoc.perl.org/File::Temp#tempfile

Closes #12545

19 months agossh: fix namespace of two local macros
Viktor Szakats [Sun, 17 Dec 2023 12:29:26 +0000 (12:29 +0000)] 
ssh: fix namespace of two local macros

Avoid using the libssh and libssh2 macro namespaces by prefixing
these local macro names with `CURL_`.

Follow-up to 413a0fedd02c8c6df1d294534b8c6e306fcca7a2 #12346

Reviewed-by: Daniel Stenberg
Closes #12544

19 months agocmake: whitespace tidy-up in `OtherTests.cmake`
Viktor Szakats [Tue, 12 Dec 2023 21:46:58 +0000 (21:46 +0000)] 
cmake: whitespace tidy-up in `OtherTests.cmake`

Closes #12538

19 months agocmake: fix generation for system name iOS
Mark Sinkovics [Thu, 14 Dec 2023 00:49:55 +0000 (19:49 -0500)] 
cmake: fix generation for system name iOS

This PR fixes a problem that happens during CMake configuration when
the `CMAKE_SYSTEM_NAME` set to `iOS` and not `Darwin`. This value is
available (as far as I remember) version 3.14. The final solution
(thanks to @vszakats) is to use `APPLE` which contains all the Apple
platforms https://cmake.org/cmake/help/latest/variable/APPLE.html.

This issue was found when during vcpkg installation. Running command
`vcpkg install curl:arm64-ios` and `vcpkg install curl:x64-ios` failed
with message:
```
CMake Error: try_run() invoked in cross-compiling mode, please set the following cache variables appropriately:
   HAVE_H_ERRNO_ASSIGNABLE_EXITCODE (advanced)
```
After this fix, I was able to compile the compile the binary without
any issue.

In addition to that fix, this PR also contains an simplification to
check if the platform is not APPLE.

Co-authored-by: Viktor Szakats
Closes #12515

19 months agoRELEASE-NOTES: synced
Daniel Stenberg [Sat, 16 Dec 2023 21:17:06 +0000 (22:17 +0100)] 
RELEASE-NOTES: synced

19 months agognutls: fix build with --disable-verbose
Baruch Siach [Mon, 11 Dec 2023 18:45:01 +0000 (20:45 +0200)] 
gnutls: fix build with --disable-verbose

infof() parameters must be defined event with --disable-verbose since
commit dac293cfb702 ("lib: apache style infof and trace
macros/functions").

Move also 'ptr' definition under !CURL_DISABLE_VERBOSE_STRINGS.

Fixes the following build failure:

In file included from ../lib/sendf.h:29,
                 from vtls/gtls.c:44:
vtls/gtls.c: In function 'Curl_gtls_verifyserver':
vtls/gtls.c:841:34: error: 'version' undeclared (first use in this function); did you mean 'session'?
  841 |         gnutls_protocol_get_name(version), ptr);
      |                                  ^~~~~~~

Closes #12505

19 months agobuild: delete unused `HAVE_{GSSHEIMDAL,GSSMIT,HEIMDAL}`
Viktor Szakats [Tue, 7 Nov 2023 00:25:18 +0000 (00:25 +0000)] 
build: delete unused `HAVE_{GSSHEIMDAL,GSSMIT,HEIMDAL}`

Stop setting `HAVE_GSSHEIMDAL`, `HAVE_GSSMIT` and `HAVE_HEIMDAL`.
There was no place in the build system or source code that used them.

Reviewed-by: Daniel Stenberg
Closes #12506

19 months agobuild: remove redundant `CURL_PULL_*` settings
Viktor Szakats [Mon, 11 Dec 2023 17:28:35 +0000 (17:28 +0000)] 
build: remove redundant `CURL_PULL_*` settings

These macros were not propagated to the source code from CMake.

autotools set only one of them (`CURL_PULL_SYS_POLL_H`), initially to
address an AIX issue [1]. This later broke when introducing `system.h`
[2] without the logic it enabled. A subsequent fix [3] re-added the
logic, and also enabled it for AIX before its use, directly in
`system.h`.

[1] 2012-11-23: 665adcd4b7bcdb7deb638cdc499fbe71f8d777f2
[2] 2017-03-29: 9506d01ee50d5908138ebad0fd9fbd39b66bd64d #1373
[3] 2017-08-25: 8a84fcc4b59e8b78d2acc6febf44a43d6bc81b59 #1828 #1833

Reviewed-by: Daniel Stenberg
Closes #12502

19 months agosystem.h: sync mingw `CURL_TYPEOF_CURL_SOCKLEN_T` with other compilers
Viktor Szakats [Sun, 10 Dec 2023 02:19:32 +0000 (02:19 +0000)] 
system.h: sync mingw `CURL_TYPEOF_CURL_SOCKLEN_T` with other compilers

Align mingw with the other Windows compilers and use the `int` type for
`CURL_TYPEOF_CURL_SOCKLEN_T` (and thus for `curl_socklent_t`). This
makes it unnecessary to make a mingw-specific trick and pull all Windows
headers early just for this type definition. This type is specific to
Windows, not to the compiler. mingw-w64's Windows header maps it to
`int` too.

With this we also delete all remaining uses of `CURL_PULL_WS2TCPIP_H`.

[ The official solution is to use `socklen_t` for all Windows compilers.
In this case we may want to update `curl/curl.h` to pull in Windows
headers before `system.h`. ]

Reviewed-by: Daniel Stenberg
Reviewed-by: Jay Satiro
Closes #12501

19 months agowindows: simplify detecting and using system headers
Viktor Szakats [Sat, 9 Dec 2023 20:37:11 +0000 (20:37 +0000)] 
windows: simplify detecting and using system headers

- autotools, cmake: assume that if we detect Windows, `windows.h`,
  `winsock2.h` and `ws2tcpip.h` do exist.
- lib: fix 3 outlier `#if` conditions to use `USE_WINSOCK` instead of
  looking for `winsock2.h`.
- autotools: merge 3 Windows check methods into one.
- move Watt-32 and lwIP socket support to `setup-win32.h` from
  `config-win32.h`. It opens up using these with all build tools. Also
  merge logic with Windows Sockets.
- fix to assume Windows sockets with the mingw32ce toolchain.
  Follow-up to: 2748c64d605b19fb419ae56810ad8da36487a2d4
- cmake: delete unused variable `signature_call_conv` since
  eb33ccd5332435fa50f1758e5debb869c6942b7f.
- autotools: simplify `CURL_CHECK_WIN32_LARGEFILE` detection.
- examples/externalsocket: fix header order.
- cmake/OtherTests.cmake: delete Windows-specific `_source_epilogue`
  that wasn't used anymore.
- cmake/OtherTests.cmake: set `WIN32_LEAN_AND_MEAN` for test
  `SIZEOF_STRUCT_SOCKADDR_STORAGE`.

After this patch curl universally uses `_WIN32` to guard
Windows-specific logic. It guards Windows Sockets-specific logic with
`USE_WINSOCK` (this might need further work).

Reviewed-by: Jay Satiro
Closes #12495

19 months agobuild: enable missing OpenSSF-recommended warnings, with fixes
Viktor Szakats [Fri, 8 Dec 2023 13:05:09 +0000 (13:05 +0000)] 
build: enable missing OpenSSF-recommended warnings, with fixes

https://best.openssf.org/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C++.html
as of 2023-11-29 [1].

Enable new recommended warnings (except `-Wsign-conversion`):

- enable `-Wformat=2` for clang (in both cmake and autotools).
- add `CURL_PRINTF()` internal attribute and mark functions accepting
  printf arguments with it. This is a copy of existing
  `CURL_TEMP_PRINTF()` but using `__printf__` to make it compatible
  with redefinting the `printf` symbol:
  https://gcc.gnu.org/onlinedocs/gcc-3.0.4/gcc_5.html#SEC94
- fix `CURL_PRINTF()` and existing `CURL_TEMP_PRINTF()` for
  mingw-w64 and enable it on this platform.
- enable `-Wimplicit-fallthrough`.
- enable `-Wtrampolines`.
- add `-Wsign-conversion` commented with a FIXME.
- cmake: enable `-pedantic-errors` the way we do it with autotools.
  Follow-up to d5c0351055d5709da8f3e16c91348092fdb481aa #2747
- lib/curl_trc.h: use `CURL_FORMAT()`, this also fixes it to enable format
  checks. Previously it was always disabled due to the internal `printf`
  macro.

Fix them:

- fix bug where an `set_ipv6_v6only()` call was missed in builds with
  `--disable-verbose` / `CURL_DISABLE_VERBOSE_STRINGS=ON`.
- add internal `FALLTHROUGH()` macro.
- replace obsolete fall-through comments with `FALLTHROUGH()`.
- fix fallthrough markups: Delete redundant ones (showing up as
  warnings in most cases). Add missing ones. Fix indentation.
- silence `-Wformat-nonliteral` warnings with llvm/clang.
- fix one `-Wformat-nonliteral` warning.
- fix new `-Wformat` and `-Wformat-security` warnings.
- fix `CURL_FORMAT_SOCKET_T` value for mingw-w64. Also move its
  definition to `lib/curl_setup.h` allowing use in `tests/server`.
- lib: fix two wrongly passed string arguments in log outputs.
Co-authored-by: Jay Satiro
- fix new `-Wformat` warnings on mingw-w64.

[1] https://github.com/ossf/wg-best-practices-os-developers/blob/56c0fde3895bfc55c8a973ef49a2572c507b2ae1/docs/Compiler-Hardening-Guides/Compiler-Options-Hardening-Guide-for-C-and-C%2B%2B.md

Closes #12489

19 months agoMakefile.mk: drop Windows support
Viktor Szakats [Sat, 28 Oct 2023 09:48:19 +0000 (09:48 +0000)] 
Makefile.mk: drop Windows support

And DLL-support with it. This leaves `Makefile.mk` for MS-DOS and Amiga.

We recommend CMake instead. With unity mode it's much faster, and about
the same without.

Ref: https://github.com/curl/curl/pull/12221#issuecomment-1783761806
Reviewed-by: Daniel Stenberg
Closes #12224

19 months agocmdline-docs: use .IP consistently
Daniel Stenberg [Sat, 16 Dec 2023 10:46:31 +0000 (11:46 +0100)] 
cmdline-docs: use .IP consistently

Remove use of .TP and some .B. The idea is to reduce nroff syntax as
much as possible and to use it consistently. Ultimately, we should be
able to introduce our own easier-to-use-and-read syntax/formatting and
convert on generation time.

Closes #12535

19 months agohttp: fix off-by-one error in request method length check
Tatsuhiko Miyagawa [Sat, 16 Dec 2023 09:28:55 +0000 (01:28 -0800)] 
http: fix off-by-one error in request method length check

It should allow one more byte.

Closes #12534

19 months agocurl: show ipfs and ipns as supported "protocols"
Daniel Stenberg [Tue, 12 Dec 2023 16:48:22 +0000 (17:48 +0100)] 
curl: show ipfs and ipns as supported "protocols"

They are accepted schemes in URLs passed to curl (the tool, not the
library).

Also makes curl-config show the same list.

Co-Authored-by: Jay Satiro
Reported-by: Chara White
Bug: https://curl.se/mail/archive-2023-12/0026.html
Closes #12508

19 months agoRevert "urldata: move async resolver state from easy handle to connectdata"
Daniel Stenberg [Thu, 14 Dec 2023 22:25:09 +0000 (23:25 +0100)] 
Revert "urldata: move async resolver state from easy handle to connectdata"

This reverts commit 56a4db2e4e2bcb9a0dcb75b83560a78ef231fcc8 (#12198)

We want the c-ares channel to be held in the easy handle, not per
connection - for performance.

Closes #12524

19 months agoopenssl: re-match LibreSSL deinit with init
Viktor Szakats [Fri, 15 Dec 2023 01:50:06 +0000 (01:50 +0000)] 
openssl: re-match LibreSSL deinit with init

Earlier we switched to use modern initialization with LibreSSL v2.7.0
and up, but did not touch deinitialization [1]. Fix it in this patch.

Regression from bec0c5bbf34369920598678161d2df8bea0e243b #11611

[1] https://github.com/curl/curl/pull/11611#issuecomment-1668654014

Reported-by: Mike Hommey
Reviewed-by: Daniel Stenberg
Fixes #12525
Closes #12526

19 months agolibssh: supress warnings without version check
Daniel Stenberg [Thu, 14 Dec 2023 22:07:24 +0000 (23:07 +0100)] 
libssh: supress warnings without version check

Define unconditionally.

Follow-up from d21bd2190c46ad7fa

Closes #12523

19 months agohostip: return error immediately when Curl_ip2addr() fails
Daniel Stenberg [Thu, 14 Dec 2023 15:34:25 +0000 (16:34 +0100)] 
hostip: return error immediately when Curl_ip2addr() fails

Closes #12522

19 months agolibssh: improve the deprecation warning dismissal
Theo [Thu, 14 Dec 2023 12:53:23 +0000 (13:53 +0100)] 
libssh: improve the deprecation warning dismissal

Previous code was compiler dependant, and dismissed all deprecation warnings
indiscriminately.

libssh provides a way to disable the deprecation warnings for libssh only, and
naturally this is the preferred way.

This commit uses that, to prevent the erroneous hiding of potential, unrelated
deprecation warnings.

Fixes #12519
Closes #12520

19 months agotest1474: removed
Daniel Stenberg [Mon, 11 Dec 2023 22:17:26 +0000 (23:17 +0100)] 
test1474: removed

The test was already somewhat flaky and disabled on several platforms,
and after 1da640abb688 even more unstable.

19 months agoreadwrite_data: loop less
Daniel Stenberg [Mon, 11 Dec 2023 18:36:27 +0000 (19:36 +0100)] 
readwrite_data: loop less

This function is made to loop in order to drain incoming data
faster. Completely removing the loop has a measerably negative impact on
transfer speeds.

Downsides with the looping include

- it might call the progress callback much more seldom. Especially if
  the write callback is slow.

- rate limiting becomes less exact

- a single transfer might "starve out" other parallel transfers

- QUIC timers for other connections can't be maintained correctly

The long term fix should be to remove the loop and optimize coming back
to avoid the transfer speed penalty.

This fix lower the max loop count to reduce the starvation problem, and
avoids the loop completely for when rate-limiting is in progress.

Ref: #12488
Ref: https://curl.se/mail/lib-2023-12/0012.html
Closes #12504

19 months agolib: eliminate `conn->cselect_bits`
Stefan Eissing [Wed, 13 Dec 2023 10:25:20 +0000 (11:25 +0100)] 
lib: eliminate `conn->cselect_bits`

- use `data->state.dselect_bits` everywhere instead
- remove `bool *comeback` parameter as non-zero
  `data->state.dselect_bits` will indicate that IO is
  incomplete.

Closes #12512

19 months agoconnect: refactor `Curl_timeleft()`
Stefan Eissing [Thu, 14 Dec 2023 11:59:19 +0000 (12:59 +0100)] 
connect: refactor `Curl_timeleft()`

- less local vars, "better" readability
- added documentation

Closes #12518

19 months agocookie: avoid fopen with empty file name
Dmitry Karpov [Wed, 13 Dec 2023 22:32:53 +0000 (14:32 -0800)] 
cookie: avoid fopen with empty file name

Closes #12514

19 months agotests/server: delete workaround for old-mingw
Viktor Szakats [Tue, 12 Dec 2023 20:38:08 +0000 (20:38 +0000)] 
tests/server: delete workaround for old-mingw

mingw-w64 1.0 comes with w32api v3.12, thus doesn't need this.

Follow-up to 38029101e2d78ba125732b3bab6ec267b80a0e72 #11625

Reviewed-by: Jay Satiro
Closes #12510

19 months agocmake: delete obsolete TODOs more [ci skip]
Viktor Szakats [Tue, 12 Dec 2023 16:37:28 +0000 (16:37 +0000)] 
cmake: delete obsolete TODOs more [ci skip]

- manual completed: 898b012a9bf388590c4be7f526815b5ab74feca1 #1288
- soname completed: 5de6848f104d7cb0017080e31216265ac19d0dde #10023
- bunch of others that are completed
- `NTLM_WB_ENABLED` is implemented in a basic form, and now also
  scheduled for removal, so a TODO at this point isn't useful.

And this 'to-check' item:

Q: "The cmake build selected to run gcc with -fPIC on my box while the
   plain configure script did not."

A: With CMake, since 2ebc74c36a19a1700af394c16855ce144d9878e3 #11546
   and fc9bfb14520712672b4784e8b48256fb29204011 #11627, we explicitly
   enable PIC for libcurl shared lib. Or when building libcurl for
   shared and static lib in a single pass. We do this by default for
   Windows or when enabled by the user via `SHARE_LIB_OBJECT`.
   Otherwise we don't touch this setting. Meaning the default set by
   CMake (if any) or the toolchain is used. On Debian Bookworm, this
   means that PIC is disabled for static libs by default. Some platforms
   (like macOS), has PIC enabled by default.
   autotools supports the double-pass mode only, and in that case
   CMake seems to match PIC behaviour now (as tested on Linux with gcc.)

Follow-up to 5d5dfdbd1a6c40bd75e982b66f49e1fa3a7eeae7 #12500

Reviewed-by: Jay Satiro
Closes #12509

19 months agoCLIENT-WRITERS: design and use documentation
Stefan Eissing [Tue, 12 Dec 2023 11:36:07 +0000 (12:36 +0100)] 
CLIENT-WRITERS: design and use documentation

Closes #12507

19 months agocmake: delete obsolete TODO items [ci skip]
Viktor Szakats [Sun, 10 Dec 2023 03:33:17 +0000 (03:33 +0000)] 
cmake: delete obsolete TODO items [ci skip]

There is always room for improvement, but CMake is up to par now with
autotools, so there is no longer a good reason to keep around these
inline TODO items.

Answering one of questions:

Q: "The gcc command line use neither -g nor any -O options. As a
   developer, I also treasure our configure scripts's --enable-debug
   option that sets a long range of "picky" compiler options."

A: CMake offers the `CMAKE_BUILD_TYPE` variable to control debug info
   and optimization level. E.g.:
   - `Release`    = `-O3` + no debug info
   - `MinSizeRel` = `-Os` + no debug info
   - `Debug`      = `-O0` + debug info

   https://stackoverflow.com/questions/48754619/what-are-cmake-build-type-debug-release-relwithdebinfo-and-minsizerel/59314670#59314670
   https://cmake.org/cmake/help/latest/manual/cmake-buildsystem.7.html#default-and-custom-configurations

   For picky warnings we have the `PICKY_COMPILER` options, enabled by
   default.

Closes #12500

19 months agoCONNECTION-FILTERS: update documentation
Stefan Eissing [Mon, 11 Dec 2023 10:52:26 +0000 (11:52 +0100)] 
CONNECTION-FILTERS: update documentation

Closes #12497

19 months agolib: reduce use of strncpy
Daniel Stenberg [Mon, 11 Dec 2023 15:15:57 +0000 (16:15 +0100)] 
lib: reduce use of strncpy

- bearssl: select cipher without buffer copies
- http_aws_sigv4: avoid strncpy, require exact timestamp length
- http_aws_sigv4: use memcpy isntead of strncpy
- openssl: avoid strncpy calls
- schannel: check for 1.3 algos without buffer copies
- strerror: avoid strncpy calls
- telnet: avoid strncpy, return error on too long inputs
- vtls: avoid strncpy in multissl_version()

Closes #12499

19 months agoCI/distcheck: run full tests
Daniel Stenberg [Mon, 11 Dec 2023 17:57:48 +0000 (18:57 +0100)] 
CI/distcheck: run full tests

To be able to detect missing files better, this now runs the full CI
test suite. If done before, it would have detected #12462 before
release.

Closes #12503

19 months agodocs: clean up Protocols: for cmdline options
Daniel Stenberg [Mon, 11 Dec 2023 07:52:19 +0000 (08:52 +0100)] 
docs: clean up Protocols: for cmdline options

... and some other minor polish.

Closes #12496

19 months agocmdline/gen: fix the sorting of the man page options
Daniel Stenberg [Sat, 9 Dec 2023 23:05:04 +0000 (00:05 +0100)] 
cmdline/gen: fix the sorting of the man page options

They were previously sorted based on the file names, which use a .d
extension, making "data" get placed after "data-binary" etc. Making the
sort ignore the extention fixes the ordering.

Reported-by: Boris Verkhovskiy
Bug: https://curl.se/mail/archive-2023-12/0014.html
Closes #12494

19 months agodoh: remove unused local variable
Daniel Gustafsson [Fri, 8 Dec 2023 20:12:06 +0000 (21:12 +0100)] 
doh: remove unused local variable

The nurl variable is no longer used during probing following
a refactoring, so remove.

Closes #12491

19 months agobuild: fix Windows ADDRESS_FAMILY detection
Jay Satiro [Fri, 1 Dec 2023 23:06:12 +0000 (18:06 -0500)] 
build: fix Windows ADDRESS_FAMILY detection

- Include winsock2.h for Windows ADDRESS_FAMILY detection.

Prior to this change cmake detection didn't work because it included
ws2def.h by itself, which is missing needed types from winsock2.h.

Prior to this change autotools detection didn't work because it did not
include any Windows header.

In both cases libcurl would fall back on unsigned short as the address
family type, which is the same as ADDRESS_FAMILY.

Co-authored-by: Viktor Szakats
Closes https://github.com/curl/curl/pull/12441

19 months agolib: rename Curl_strndup to Curl_memdup0 to avoid misunderstanding
Daniel Stenberg [Fri, 8 Dec 2023 13:27:29 +0000 (14:27 +0100)] 
lib: rename Curl_strndup to Curl_memdup0 to avoid misunderstanding

Since the copy does not stop at a null byte, let's not call it anything
that makes you think it works like the common strndup() function.

Based on feedback from Jay Satiro, Stefan Eissing and Patrick Monnerat

Closes #12490

19 months agoconvsrctest.pl: removed: not used, not shipped in tarballs
Daniel Stenberg [Fri, 8 Dec 2023 09:28:46 +0000 (10:28 +0100)] 
convsrctest.pl: removed: not used, not shipped in tarballs

19 months agotests: rename tests scripts to the test number
Daniel Stenberg [Fri, 8 Dec 2023 08:34:05 +0000 (09:34 +0100)] 
tests: rename tests scripts to the test number

It is hard to name the scripts sensibly. Lots of them are similarly
named and the name did not tell which test that used them.

The new approach is rather to name them based on the test number that
runs them. Also helps us see which scripts are for individual tests
rather than for general test infra.

 - badsymbols.pl -> test1167.pl
 - check-deprecated.pl -> test1222.pl
 - check-translatable-options.pl -> test1544.pl
 - disable-scan.pl -> test1165.pl
 - error-codes.pl -> test1175.pl
 - errorcodes.pl -> test1477.pl
 - extern-scan.pl -> test1135.pl
 - manpage-scan.pl -> test1139.pl
 - manpage-syntax.pl -> test1173.pl
 - markdown-uppercase.pl -> test1275.pl
 - mem-include-scan.pl -> test1132.pl
 - nroff-scan.pl -> test1140.pl
 - option-check.pl -> test1276.pl
 - options-scan.pl -> test971.pl
 - symbol-scan.pl -> test1119.pl
 - version-scan.pl -> test1177.pl

Closes #12487

19 months agosendf: fix compiler warning with CURL_DISABLE_HEADERS_API
MAntoniak [Fri, 8 Dec 2023 00:07:38 +0000 (01:07 +0100)] 
sendf: fix compiler warning with CURL_DISABLE_HEADERS_API

fix MSVC warning C4189: 'htype': local variable is initialized but not
referenced - when CURL_DISABLE_HEADERS_API is defined.

Closes #12485

19 months agotidy-up: whitespace
Viktor Szakats [Thu, 7 Dec 2023 19:06:02 +0000 (19:06 +0000)] 
tidy-up: whitespace

Closes #12484

19 months agotest_02_download: fix paramters to test_02_27
Stefan Eissing [Wed, 6 Dec 2023 11:08:20 +0000 (12:08 +0100)] 
test_02_download: fix paramters to test_02_27

- it is a special client that only ever uses http/2

Closes #12467

19 months agovtls: remove the Curl_cft_ssl_proxy object if CURL_DISABLE_PROXY
MAntoniak [Tue, 5 Dec 2023 22:49:12 +0000 (23:49 +0100)] 
vtls: remove the Curl_cft_ssl_proxy object if CURL_DISABLE_PROXY

Closes #12459

19 months agolib: strndup/memdup instead of malloc, memcpy and null-terminate
Daniel Stenberg [Tue, 5 Dec 2023 14:55:35 +0000 (15:55 +0100)] 
lib: strndup/memdup instead of malloc, memcpy and null-terminate

 - bufref: use strndup
 - cookie: use strndup
 - formdata: use strndup
 - ftp: use strndup
 - gtls: use aprintf instead of malloc + strcpy * 2
 - http: use strndup
 - mbedtls: use strndup
 - md4: use memdup
 - ntlm: use memdup
 - ntlm_sspi: use strndup
 - pingpong: use memdup
 - rtsp: use strndup instead of malloc, memcpy and null-terminate
 - sectransp: use strndup
 - socks_gssapi.c: use memdup
 - vtls: use dynbuf instead of malloc, snprintf and memcpy
 - vtls: use strdup instead of malloc + memcpy
 - wolfssh: use strndup

Closes #12453

19 months agostrdup: remove the memchr check from Curl_strndup
Daniel Stenberg [Tue, 5 Dec 2023 14:55:35 +0000 (15:55 +0100)] 
strdup: remove the memchr check from Curl_strndup

It makes it possible to clone a binary chunk of data.

Closes #12453

19 months agoftp: handle the PORT parsing without allocation
Daniel Stenberg [Tue, 5 Dec 2023 11:15:01 +0000 (12:15 +0100)] 
ftp: handle the PORT parsing without allocation

Also reduces amount of *cpy() calls.

Closes #12456

19 months agoRELEASE-NOTES: synced
Daniel Stenberg [Wed, 6 Dec 2023 22:23:06 +0000 (23:23 +0100)] 
RELEASE-NOTES: synced

Bumped to 8.5.1

19 months agourl: for disabled protocols, mention if found in redirect
Daniel Stenberg [Wed, 6 Dec 2023 13:13:17 +0000 (14:13 +0100)] 
url: for disabled protocols, mention if found in redirect

To help users better understand where the URL (and denied scheme) comes
from. Also removed "in libcurl" from the message, since the disabling
can be done by the application.

The error message now says "not supported" or "disabled" depending on
why it was denied:

 Protocol "hej" not supported
 Protocol "http" disabled

And in redirects:

 Protocol "hej" not supported (in redirect)
 Protocol "http" disabled (in redirect)

Reported-by: Mauricio Scheffer
Fixes #12465
Closes #12469

19 months agosectransp_ make TLSCipherNameForNumber() available in non-verbose config
Stefan Eissing [Wed, 6 Dec 2023 18:03:42 +0000 (19:03 +0100)] 
sectransp_ make TLSCipherNameForNumber() available in non-verbose config

Reported-by: Cajus Pollmeier
Closes #12476
Fixes #12474

19 months agolib: fix variable undeclared error caused by `infof` changes
YX Hao [Wed, 6 Dec 2023 14:42:00 +0000 (22:42 +0800)] 
lib: fix variable undeclared error caused by `infof` changes

`--disable-verbose` yields `CURL_DISABLE_VERBOSE_STRINGS` defined.
`infof` isn't `Curl_nop_stmt` anymore: dac293c.

Follow-up to dac293c

Closes #12470

19 months agotidy-up: fix yamllint whitespace issues in labeler.yml
Viktor Szakats [Wed, 6 Dec 2023 17:54:08 +0000 (17:54 +0000)] 
tidy-up: fix yamllint whitespace issues in labeler.yml

Follow-up to bda212911457c6fadfbba50be61afc4ca513fa56 #12466

Reviewed-by: Dan Fandrich
Closes #12475

19 months agotidy-up: fix yamllint whitespace issues
Viktor Szakats [Wed, 6 Dec 2023 10:05:20 +0000 (10:05 +0000)] 
tidy-up: fix yamllint whitespace issues

Closes #12466

19 months agocmake: fix typo
Chris Sauer [Wed, 6 Dec 2023 09:16:36 +0000 (01:16 -0800)] 
cmake: fix typo

Follow-up to aace27b
Closes #12464

19 months agodist: add tests/errorcodes.pl to the tarball
Daniel Stenberg [Wed, 6 Dec 2023 08:40:30 +0000 (09:40 +0100)] 
dist: add tests/errorcodes.pl to the tarball

Used by test 1477

Reported-by: Xi Ruoyao
Follow-up to 0ca3a4ec9a7
Fixes #12462
Closes #12463

19 months agogithub/labeler: update a missed key in the v5 upgrade
Dan Fandrich [Wed, 6 Dec 2023 08:58:59 +0000 (00:58 -0800)] 
github/labeler: update a missed key in the v5 upgrade

Follow-up to ce03fe3ba

19 months agoRELEASE-NOTES: synced curl-8_5_0
Daniel Stenberg [Wed, 6 Dec 2023 07:11:44 +0000 (08:11 +0100)] 
RELEASE-NOTES: synced

The curl 8.5.0 release.

19 months agogithub/labeler: switch from the beta to labeler v5 12458/head
Dan Fandrich [Tue, 5 Dec 2023 21:17:09 +0000 (13:17 -0800)] 
github/labeler: switch from the beta to labeler v5

Some keys were renamed and the dot option was made default.

Closes #12458

19 months agoDEPRECATE: remove NTLM_WB in June 2024
Daniel Stenberg [Tue, 5 Dec 2023 07:02:57 +0000 (08:02 +0100)] 
DEPRECATE: remove NTLM_WB in June 2024

Ref: https://curl.se/mail/lib-2023-12/0010.html

Closes #12451

19 months agorustls: implement connect_blocking
Jacob Hoffman-Andrews [Fri, 23 Jun 2023 23:58:54 +0000 (16:58 -0700)] 
rustls: implement connect_blocking

Closes #11647

19 months agoexamples/rtsp-options.c: add
Daniel Stenberg [Mon, 4 Dec 2023 16:14:56 +0000 (17:14 +0100)] 
examples/rtsp-options.c: add

Just a bare bones RTSP example using CURLOPT_RTSP_SESSION_ID and
CURLOPT_RTSP_REQUEST set to CURL_RTSPREQ_OPTIONS.

Closes #12452

20 months agongtcp2: ignore errors on unknown streams
Stefan Eissing [Mon, 4 Dec 2023 11:13:25 +0000 (12:13 +0100)] 
ngtcp2: ignore errors on unknown streams

- expecially in is_alive checks on connections, we might
  see incoming packets on streams already forgotten and closed,
  leading to errors reported by nghttp3. Ignore those.

Closes #12449

20 months agodocs: make all examples in all libcurl man pages compile
Daniel Stenberg [Mon, 4 Dec 2023 09:50:42 +0000 (10:50 +0100)] 
docs: make all examples in all libcurl man pages compile

Closes #12448

20 months agochecksrc.pl: support #line instructions
Daniel Stenberg [Mon, 4 Dec 2023 09:50:42 +0000 (10:50 +0100)] 
checksrc.pl: support #line instructions

makes it identify the correct source file and line

20 months agoGHA/man-examples: verify libcurl man page examples
Daniel Stenberg [Mon, 4 Dec 2023 09:50:42 +0000 (10:50 +0100)] 
GHA/man-examples: verify libcurl man page examples

20 months agoverify-examples.pl: verify that all man page examples compile clean
Daniel Stenberg [Mon, 4 Dec 2023 09:50:42 +0000 (10:50 +0100)] 
verify-examples.pl: verify that all man page examples compile clean

20 months agoRELEASE-NOTES: synced
Daniel Stenberg [Sat, 2 Dec 2023 22:21:21 +0000 (23:21 +0100)] 
RELEASE-NOTES: synced

20 months agohttp3: bump ngtcp2 and nghttp3 versions
Graham Campbell [Sat, 2 Dec 2023 18:17:35 +0000 (18:17 +0000)] 
http3: bump ngtcp2 and nghttp3 versions

nghttp3 v1.1.0
ngtcp2 v1.1.0

In docs and CI

Closes #12446

20 months agoCI/quiche: use `3.1.4+quic` consistently in CI workflows
Graham Campbell [Sat, 2 Dec 2023 18:24:19 +0000 (18:24 +0000)] 
CI/quiche: use `3.1.4+quic` consistently in CI workflows

Closes #12447

20 months agotest1545: disable deprecation warnings
Viktor Szakats [Sat, 2 Dec 2023 15:59:21 +0000 (15:59 +0000)] 
test1545: disable deprecation warnings

Fixes:
https://ci.appveyor.com/project/curlorg/curl/builds/48631551/job/bhx74e0i66yrp6pk#L1205

Same with details:
https://ci.appveyor.com/project/curlorg/curl/builds/48662893/job/ol8a78q9gmilb6wt#L1263
```
tests/libtest/lib1545.c:38:3: error: 'curl_formadd' is deprecated: since 7.56.0. Use curl_mime_init() [-Werror=deprecated-declarations]
   38 |   curl_formadd(&m_formpost, &lastptr, CURLFORM_COPYNAME, "file",
      |   ^~~~~~~~~~~~
[...]
```

Follow-up to 07a3cd83e0456ca17dfd8c3104af7cf45b7a1ff5 #12421

Fixes #12445
Closes #12444

20 months agoINSTALL: update list of ports and CPU archs
Daniel Stenberg [Sat, 2 Dec 2023 16:40:48 +0000 (17:40 +0100)] 
INSTALL: update list of ports and CPU archs

20 months agosymbols-in-versions: the CLOSEPOLICY options are deprecated
Daniel Stenberg [Sat, 2 Dec 2023 16:07:34 +0000 (17:07 +0100)] 
symbols-in-versions: the CLOSEPOLICY options are deprecated

The were used with the CURLOPT_CLOSEPOLICY option, which *never* worked.

20 months agobuild: fix builds that disable protocols but not digest auth
z2_ [Sat, 2 Dec 2023 00:09:29 +0000 (01:09 +0100)] 
build: fix builds that disable protocols but not digest auth

- Build base64 functions if digest auth is not disabled.

Prior to this change if some protocols were disabled but not digest auth
then a build error would occur due to missing base64 functions.

Fixes https://github.com/curl/curl/issues/12440
Closes https://github.com/curl/curl/pull/12442

20 months agoconnect: reduce number of transportation providers
MAntoniak [Fri, 1 Dec 2023 17:48:09 +0000 (18:48 +0100)] 
connect: reduce number of transportation providers

Use only the ones necessary - the ones that are built-in. Saves a few
bytes in the resulting code.

Closes #12438

20 months agovtls: consistently use typedef names for OpenSSL structs
David Benjamin [Fri, 1 Dec 2023 19:40:32 +0000 (14:40 -0500)] 
vtls: consistently use typedef names for OpenSSL structs

The foo_st names don't appear in OpenSSL public API documentation. The
FOO typedefs are more common. This header was already referencing
SSL_CTX via <openssl/ssl.h>. There is a comment about avoiding
<openssl/x509v3.h>, but OpenSSL actually declares all the typedefs in
<openssl/ossl_typ.h>, which is already included by <openssl/ssl.h> (and
every other OpenSSL header), so just use that. Though I've included it
just to be explicit.

(I'm also fairly sure including <openssl/ssl.h> already triggers the
Schannel conflicts anyway. The comment was probably just out of date.)

Closes #12439

20 months agolibcurl-security.3: fix typo
Lau [Fri, 1 Dec 2023 14:44:03 +0000 (15:44 +0100)] 
libcurl-security.3: fix typo

Fixed minimal typo.

Closes #12437

20 months agongtcp2: fix races in stream handling
Stefan Eissing [Fri, 1 Dec 2023 13:00:15 +0000 (14:00 +0100)] 
ngtcp2: fix races in stream handling

- fix cases where ngtcp2 invokes callbacks on streams that
  nghttp3 has already forgotten. Ignore the NGHTTP3_ERR_STREAM_NOT_FOUND
  in these cases as it is normal behaviour.

Closes #12435