Daniel Stenberg [Wed, 5 Jun 2024 12:08:02 +0000 (14:08 +0200)]
managen: warn on excessively long help texts
Help texts at 49 characters or longer get a warning displayed because
they make --help output uglier and we should make an effort to keep the
help texts short and succinct.
The warning is only for display, it does not break the build. That is
left for the future if necessary.
I picked 49 because the longest current text is 48.
Stefan Eissing [Tue, 4 Jun 2024 12:00:12 +0000 (14:00 +0200)]
vtls: new io_need flags for poll handling
- decouple need to recv/send from negotiation state, we need
this later in shutdown handling as well
- move ssl enums from urldata.h to vtls_int.h
- implement use of `connssl->io_need` in vtls.c. and all backends
Daniel Stenberg [Tue, 4 Jun 2024 15:00:05 +0000 (17:00 +0200)]
url: allow DoH transfers to override max connection limit
When reaching the set maximum limit of allowed connections, allow a new
connection anyway if the transfer is created for the (internal) purpose
of doing a DoH name resolve. Otherwise, unrelated "normal" transfers can
starve out new DoH requests making it impossible to name resolve for new
transfers.
Viktor Szakats [Mon, 3 Jun 2024 21:06:56 +0000 (23:06 +0200)]
windows: fix UWP builds, add GHA job
Add new job to test building for UWP (aka `CURL_WINDOWS_APP`).
Fix fallouts when building for UWP:
- rand: do not use `BCryptGenRandom()`.
- cmake: disable using win32 LDAP.
- cmake: disable telnet.
- version_win32: fix code before declaration.
- schannel: disable `HAS_MANUAL_VERIFY_API`.
- schannel: disable `SSLSUPP_PINNEDPUBKEY`
and make `schannel_checksum()` a stub.
Ref: e178fbd40a896f2098278ae61e1166c88e7b31d0 #1429
- schannel: make `cert_get_name_string()` a failing stub.
- system_win32: make `Curl_win32_impersonating()` a failing stub.
- system_win32: try to fix `Curl_win32_init()` (untested).
- threads: fix to use `CreateThread()`.
- src: disable searching `PATH` for the CA bundle.
- src: disable bold text support and capability detection.
- src: disable `getfiletime()`/`setfiletime()`.
- tests: make `win32_load_system_library()` a failing stub.
- tests/server/util: make it compile.
- tests/server/sockfilt: make it compile.
- tests/lib3026: fix to use `CreateThread()`.
See individual commits for build error details.
Some of these fixes may have better solutions, and some may not work
as expected. The goal of this patch is to make curl build for UWP.
Orgad Shaneh [Fri, 17 May 2024 11:44:44 +0000 (14:44 +0300)]
socket: support binding to interface *AND* IP
Introduce new notation for CURLOPT_INTERFACE / --interface:
ifhost!<interface>!<host>
Binding to an interface doesn't set the address, and an interface can
have multiple addresses.
When binding to an address (without interface), the kernel is free to
choose the route, and it can route through any device that can access
the target address, not necessarily the one with the chosen address.
Moreover, it is possible for different interfaces to have the same IP
address, on which case we need to provide a way to be more specific.
Factor out the parsing part of interface option, and add unit tests:
1663.
Andy Pan [Sat, 1 Jun 2024 18:26:11 +0000 (02:26 +0800)]
socketpair: add `eventfd` and use `SOCK_NONBLOCK` for `socketpair()`
Currently, we use `pipe` for `wakeup_create`, which requires ***two***
file descriptors. Furthermore, given its complexity inside, `pipe` is a
bit heavyweight for just a simple event wait/notify mechanism.
`eventfd` would be a more suitable solution for this kind of scenario,
kernel also advocates for developers to use `eventfd` instead of `pipe`
in some simple use cases:
Applications can use an eventfd file descriptor instead of a pipe
(see pipe(2) in all cases where a pipe is used simply to signal
events. The kernel overhead of an eventfd file descriptor is much
lower than that of a pipe, and only one file descriptor is required
(versus the two required for a pipe).
This change adds the new backend of `eventfd` for `wakeup_create` and
uses it where available, eliminating the overhead of `pipe`. Also, it
optimizes the `wakeup_create` to eliminate the system calls that make
file descriptors non-blocking by moving the logic of setting
non-blocking flags on file descriptors to `socketpair.c` and using
`SOCK_NONBLOCK` for `socketpair(2)`, `EFD_NONBLOCK` for `eventfd(2)`.
- Remove the locking callback code that demonstrates how to meet
requirements of threading libraries (mainly OpenSSL).
Locking callback code has not been needed for many years. According to
the documentation for OpenSSL and GnuTLS they are thread-safe by design,
assuming support for the underlying OS threading API is built-in.
Viktor Szakats [Tue, 4 Jun 2024 00:41:26 +0000 (02:41 +0200)]
lib: fix thread entry point to return `DWORD` on WinCE
We already do this in `tests/server/util.c`:
https://github.com/curl/curl/blob/97e5e37cc8269660bc5d4a1936f10f2390b97c5a/tests/server/util.c#L604-L606
and in `sockfilt.c`, `lib3026.c`.
Andy Pan [Sun, 2 Jun 2024 06:40:42 +0000 (14:40 +0800)]
socket: use SOCK_NONBLOCK to eliminate extra system call
Every time function `cf_socket_open()` is called to create a socket,
`curlx_nonblock()` is called to make that socket non-blocking. And
`curlx_nonblock()` will cost us 1 or 2 system calls (2 for `fcntl()`, 1
for `ioctl()`, etc.), meanwhile, tucking `SOCK_NONBLOCK` and
`SOCK_CLOEXEC` into the `type` argument for `socket()` is widely
supported across UNIX-like OS: Linux, *BSD, Solaris, etc. With that
ability, we can save 1 or 2 system calls on each socket.
Another change in this PR is to eliminate the redundant
`curlx_nonblock()` call on the socket in `cf_udp_setup_quic()` as that
socket created by `cf_socket_open()` is already non-blocking.
Viktor Szakats [Tue, 4 Jun 2024 00:46:05 +0000 (02:46 +0200)]
GHA: show cmake error log in Windows and non-native workflows
CMake configure doesn't fail often, but when it does, it helps to see
its `CMakeFiles/CMakeConfigureLog.yaml` output. This file is present
since CMake v3.26:
https://cmake.org/cmake/help/v3.26/manual/cmake-configure-log.7.html
(Older CMake versions save similar contend to
`CMakeFiles\CMakeOutput.log` and
`CMakeFiles\CMakeError.log`. This patch doesn't deal with that because
the workflows touched are all running a newer CMake.)
After this patch, we dump the content if cmake fails. Syncing this with
autotools, where we already did that.
Stefan Eissing [Fri, 31 May 2024 11:01:17 +0000 (13:01 +0200)]
mbedtls: v3.6.0 workarounds
- add special sauce to disable unwanted peer verification by mbedtls
when negotiating TLS v1.3
- add special sauce for MBEDTLS_ERR_SSL_RECEIVED_NEW_SESSION_TICKET
return code on *writing* TLS data. We assume the data had not been
written and EAGAIN.
- return correct Curl error code when peer verification failed.
- disable test_08_05 with 50 HTTP/1.1 connections, as mbedtls reports a
memory allocation failed during handshake.
- bump CI mbedtls version to 3.6.0
Stefan Eissing [Mon, 27 May 2024 14:50:15 +0000 (16:50 +0200)]
gnutls: support CA caching
- similar to openssl, use a shared 'credentials' instance
among TLS connections with a plain configuration.
- different to openssl, a connection with a client certificate
is not eligible to sharing.
- document CURLOPT_CA_CACHE_TIMEOUT in man page
Andy Pan [Mon, 3 Jun 2024 07:15:55 +0000 (15:15 +0800)]
tcpkeepalive: support setting TCP keep-alive parameters on Solaris <11.4
Solaris didn't support TCP_KEEPIDLE and TCP_KEEPINTVL until 11.4,
before that it use TCP_KEEPALIVE_THRESHOLD and TCP_KEEPALIVE_ABORT_THRESHOLD
as the substitute. Therefore, for Solaris <11.4 we need to use this substitute
for setting TCP keep-alive parameters.
Viktor Szakats [Sun, 2 Jun 2024 16:22:48 +0000 (18:22 +0200)]
GHA: fix caching old mingw-w64 toolchains in the Windows workflow
- stop altering the `PATH` via `GITHUB_ENV`. This confused the
`actions/cache` post-job, which needs to run in the exact same
environment as its pre-job, to have a consistent cache entry "version"
hash. Altering the `PATH` via `GITHUB_ENV` spills into the the
post-job and breaks this hash. GHA doesn't reset the env automatically
and I have not found a way to do it manually.
- add double-quotes where missing.
- move cache directory under `USERPROFILE` to not rely on absolute
paths.
Andy Pan [Sat, 1 Jun 2024 10:36:04 +0000 (18:36 +0800)]
socket: change TCP keepalive from ms to seconds on DragonFly BSD
DragonFly BSD changed the time unit for TCP keep-alive from milliseconds
to seconds since v5.8, thus setting the keepalive options with
milliseconds with curl/libcurl will result in unexpected behaviors on
DragonFlyBSD 5.8+
Distinguish the DragonFly BSD versions and use the proper time units
accordingly.
Jan Venekamp [Wed, 29 May 2024 11:58:53 +0000 (13:58 +0200)]
sectransp: remove large cipher table
Previously a large table of ciphers was used to determine the default
ciphers and to lookup manually selected ciphers names.
With the lookup of the manually selected cipher names moved to
Curl_cipher_suite_walk_str() the large table is no longer needed for
that purpose.
The list of manually selected cipher can now be intersected with the
ciphers supported by Secure Transport (SSLGetSupportedCiphers()),
instead of using the fixed table for that.
The other use of the table was to filter the list of all supported
ciphers offered by Secure Transport to create a list of ciphers to
use by default, excluding ciphers in the table marked as weak.
Instead of using a complement based approach (exclude weak), switch
to using an intersection with a smaller list of ciphers deemed
appropriate.
The idea is better use of the build cache. Previously, they
independently create caches with the same key. Some of the caches
include source code and intermediate object files, which makes cache
quite large. In this commit, only built artifacts are cached, which
drastically reduces the cache size. OpenSSL v3, mod_h2 and quiche caches
still include all stuff, but they are left for the later improvement.
Because the contents of the cache have been changed, the cache keys are
also changed to include the word "http3".
Matt Jolly [Thu, 30 May 2024 04:47:11 +0000 (14:47 +1000)]
autoconf: remove 'deeper' checks for `AC_CHECK_FUNCS`
The net effect of the deeper checks is to raise implicit function decls
on modern compilers.
These checks appear to have been added ~20 years ago, relating to an
unverifiable claim about HP-UX. Autoconf support for the platform has
grown in leaps and bounds since.
It didn't cause a real problem here, but when investigating a FP this
came up. No evidence has been identified that this was actually broken
in the past, and there is no evidence that this is necessary now.
`-Werror=implicit-function-declarations` is enabled for both checks;
without a working prototype they will both fail regardless. In the
second case there will in fact never be a working prototype and
therefore it will always fail unconditionally.
`AC_CHECK_FUNCS` does effectively the same thing as the removed checks,
except it actually defines a dummy prototype to see if it links.
If `AC_CHECK_FUNCS` is broken on a given platform we have bigger
problems than trying to build cURL. This should also be faster.
Bug: https://bugs.gentoo.org/932827 Reviewed-By: Eli Schwartz <eschwartz93@gmail.com>
Closes #13830
- Save the last set SO_SNDBUF size to compare against so that we can
avoid setsockopt calls every second.
This is a follow-up to 0b520e12 which moved the SO_SNDBUF update check
into cf-socket. This change improves it further by making the function
names easier to understand and reducing the amount of setsockopt calls.
Adjusts the `make -j` flag to match the latest GitHub-hosted runner
hardware specs[^1]:
- `ubuntu-latest` on 4 CPU cores
- `macos-latest` on 3 CPU cores
The processor count is ideally obtained from `nproc`, but setting env
vars from the current CI yaml files is not possible because they expect
literal strings.
Stefan Eissing [Wed, 29 May 2024 15:13:34 +0000 (17:13 +0200)]
multi: fix multi_wait() timeout handling
- determine the actual poll timeout *after* all sockets
have been collected. Protocols and connection filters may
install new timeouts during collection.
- add debug logging to test1533 where the mistake was noticed
Reported-by: Matt Jolly
Fixes #13782
Closes #13825
Viktor Szakats [Tue, 28 May 2024 14:16:01 +0000 (16:16 +0200)]
lib: prefer `var = time(NULL)` over `time(&var)`
Following up on previous occurrences showing up as gcc warnings, replace
the remaining `time(&var)` calls with `var = time(NULL)`, though these
aren't specifically causing compiler warnings. These are in the TFTP
client code (`lib/tftp.c`), except one which is in a debug branch in
`lib/http_aws_sigv4.c`.
What's unexplainable is that this patch seems to mitigate TFTP tests
often hanging or going into an infinite loop on GHA windows workflows
with MSYS2, mingw-w64 and MSVC (Cygwin is unaffected):
https://github.com/curl/curl/pull/13599#issuecomment-2119372376
TFTP hangs did not entirely disappear though, so could be unrelated.
Stefan Eissing [Fri, 24 May 2024 08:09:32 +0000 (10:09 +0200)]
winsock: move SO_SNDBUF update into cf-socket
- Move the code that updates the SO_SNDBUF size for Windows to
cf_socket_send.
Prior to this change the code was in readwrite_upload but the socket
filter is the more appropriate place because it applies to all sends.
Background:
For Windows users SO_SNDBUF (the total per-socket buffer size reserved
by Winsock for sends) is updated dynamically by libcurl during the
transfer. This is because Windows does not do it automatically for
non-blocking sockets and without it the performance of large transfers
may suffer.
Matthias Gatto [Mon, 27 May 2024 12:58:11 +0000 (14:58 +0200)]
aws-sigv4: url encode the canonical path
Refactors canon_query, so it could use the encoding part of the function
to use it in the path.
As the path doesn't encode '/', but encode '=', I had to add some
conditions to know If I was doing the query or path encoding.
Also, instead of adding a `bool in_path` variable, I use `bool
*found_equals` to know if the function was called for the query or path,
as found_equals is used only in query_encoding.
Test 472 verifies.
Reported-by: Alexander Shtuchkin
Fixes #13754
Closes #13814
Daniel Stenberg [Tue, 28 May 2024 06:32:19 +0000 (08:32 +0200)]
cd2nroff: use an empty "##" to signal end of .IP sequence
Like when we list a series of options and then want to add "normal" text
again afterwards.
Without this, the indentation level wrongly continues even after the
final "##" header, making following text wrongly appear to belong to the
header above.
Adjusted several curldown files to use this.
Fixes #13803 Reported-by: Jay Satiro
Closes #13806
Daniel Stenberg [Mon, 27 May 2024 21:12:27 +0000 (23:12 +0200)]
openssl/gnutls: rectify the TLS version checks for QUIC
The versions check wrongly complained and return error if the *minimum*
version was set to something less than 1.3. QUIC is always TLS 1.3, but
that means minimum 1.2 is still fine to ask for.
This also renames the local variable to make the mistake harder to make
in the future.
Stefan Eissing [Mon, 27 May 2024 09:51:03 +0000 (11:51 +0200)]
gnutls: improve TLS shutdown
local ftp upload tests sometimes failed with an invalid TLS record being
reported by gnutls. vsftp did log that the shutdown was not regarded as
clean, failing the control connection thereafter.
These changes make test_31_05 work reliable locally.
- on closing the SSL filter, shutdown READ *and* WRITE
- on closing, try a receive after shutdown is sent
- convert to DEBUGF to CURL_TRC_CF
Viktor Szakats [Mon, 20 May 2024 12:21:05 +0000 (14:21 +0200)]
build: untangle `CURLDEBUG` and `DEBUGBUILD` macros
`CURLDEBUG` is meant to enable memory tracking, but in a bunch of cases,
it was protecting debug features that were supposed to be guarded with
`DEBUGBUILD`.
Replace these uses with `DEBUGBUILD`.
This leaves `CURLDEBUG` uses solely for its intended purpose: to enable
the memory tracking debug feature.
Also:
- autotools: rely on `DEBUGBUILD` to enable `checksrc`.
Instead of `CURLDEBUG`, which worked in most cases because debug
builds enable `CURLDEBUG` by default, but it's not accurate.
- include `lib/easyif.h` instead of keeping a copy of a declaration.
- add CI test jobs for the build issues discovered.
Viktor Szakats [Sat, 18 May 2024 00:15:32 +0000 (02:15 +0200)]
cmake: always build unit tests with the `testdeps` target
Before this patch, the `testdeps` build target required `-DCURLDEBUG`
be set either via `ENABLE_DEBUG=ON` or `ENABLE_CURLDEBUG=ON` to build
the curl unit tests.
After fixing build issues in #13694, we can drop this requirement and
build unit tests unconditionally.
Depends-on: #13694
Depends-on: #13697 (fix unit test issue revealed by Old Linux CI job)
Follow-up to 39e7c22bb459c2e818f079984989a26a09741860 #11446
Closes #13698
Viktor Szakats [Mon, 27 May 2024 13:19:35 +0000 (15:19 +0200)]
CI: disable dependency tracking in most autotools builds
For better build performance. Dependency tracking causes a build
overhead while compiling to help a subsequent build, but in CI there is
never one and the extra work is discarded.
Viktor Szakats [Thu, 16 May 2024 11:49:22 +0000 (13:49 +0200)]
build: untangle `UNITTESTS` and `DEBUGBUILD` macros
- fix `DEBUGBUILD` guards that should be `UNITTESTS`, in libcurl code
used by unit tests.
- fix guards for libcurl functions used in unit tests only.
- sync `UNITTEST` attribute between declarations and definitions.
- drop `DEBUGBUILD` guard from test `unit2600`.
- fix guards for libcurl HSTS code used by both a unit test (`unit1660`)
and `test0446`.
- update an existing AppVeyor CI job to test the issues fixed.
This fixes building tests with `CURLDEBUG` enabled but `DEBUGBUILD`
disabled. This can happen when building tests with CMake with
`ENABLE_DEBUG=ON` in Release config, or with `ENABLE_CURLDEBUG=ON`
and _without_ `ENABLE_DEBUG=ON`. Possibly also with autotools
when using `--enable-curldebug` without `--enable-debug`.
Test results:
- before:
https://ci.appveyor.com/project/curlorg/curl/builds/49835609
https://ci.appveyor.com/project/curlorg/curl/builds/49898529/job/k8qpbs8idby70smw
https://github.com/curl/curl/actions/runs/9259078835/job/25470318167?pr=13798#step:13:821
- after: https://ci.appveyor.com/project/curlorg/curl/builds/49839255
(the two failures are unrelated, subject to PR #13705)
Ref: #13592 (issue discovery)
Ref: #13689 (CI testing this PR with `DEBUGBUILD`/`CURLDEBUG` combinations)
Closes #13694
Viktor Szakats [Mon, 27 May 2024 16:48:32 +0000 (18:48 +0200)]
CI: tidy up skipping tests build/run in Windows jobs
Simplify controlling whether to build and/run tests in a CI job.
Apply the TFLAGS='skipall' (do not build nor run tests) or
'skiprun' (build, but do not run) method already used with old-mingw-w64
and msvc jobs to existing Windows jobs in GHA and AppVeyor.
Also:
- add Cygwin/cmake test build and run steps while here.
- replace `DISABLED_TESTS` with `TFLAGS` in AppVeyor.
Viktor Szakats [Mon, 27 May 2024 12:33:54 +0000 (14:33 +0200)]
cmake: allow `ENABLE_CURLDEBUG=OFF` with `ENABLE_DEBUG=ON`
Before this patch, `ENABLE_CURLDEBUG` (memory tracking) was
unconditionally enabled when `ENABLE_DEBUGBUILD` was set. This made
testing some build configurations complicated. To fix it, this patch
makes `ENABLE_CURLDEBUG` to receive the value of `ENABLE_DEBUG` by
default, while allowing free override by the user.
This allows to use the config:
`ENABLE_DEBUGBUILD=ON ENABLE_CURLDEBUG=OFF`
to enable debug features, without also enabling memory tracking.
This is important because some other build methods allow to set one of
these features but not the other. This patch allows to test any
combination with CMake.
This makes it unnecessary to use the workaround of passing
`-DDEBUGBUILD` via `CMAKE_C_FLAGS`. Which has the disadvantage that our
CMake logic cannot easily detect it, e.g. for disabling symbol hiding on
Windows for `ENABLE_DEBUG`/`DEBUGBUILD` builds.
Viktor Szakats [Sat, 11 May 2024 00:44:10 +0000 (02:44 +0200)]
cmake: `ENABLE_DEBUG=ON` to always set `-DDEBUGBUILD`
Before this patch `ENABLE_DEBUG=ON` always enabled the TrackMemory
(aka `ENABLE_CURLDEBUG=ON`) feature, but required the `Debug` CMake
configration to actually enable curl debug features
(aka `-DDEBUGBUILD`).
Curl debug features do not require compiling with C debug options. This
also made enabling debug features unintuitive and complicated to use.
Due to other issues (subject to PR #13694) it also caused an error in
default (and `Release`/`MinSizeRel`/`RelWithDebInfo`) configs, when
building the `testdeps` target:
```
ld: CMakeFiles/unit1395.dir/unit1395.c.o: in function `test':
unit1395.c:(.text+0x1a0): undefined reference to `dedotdotify'
```
Ref: https://github.com/curl/curl/actions/runs/9037287098/job/24835990826#step:3:2483
Fix it by always defining `DEBUGBUILD` when setting `ENABLE_DEBUG=ON`.
Decoupling this option from the selected CMake configuration.
Note that after this patch `ENABLE_DEBUG=ON` unconditionally enables
curl debug features. These features are insecure and unsuited for
production. Make sure to omit this option when building for production
in default, `Release` (and other not-`Debug`) modes.
Also delete a workaround no longer necessary in GHA CI jobs.
projects / thirdparty / curl.git / log
