Max Faxälv [Thu, 29 Feb 2024 08:12:59 +0000 (09:12 +0100)]
spnego_gssapi: implement TLS channel bindings for openssl
Channel Bindings are used to tie the session context to a specific TLS
channel. This is to provide additional proof of valid identity,
mitigating authentication relay attacks.
Major web servers have the ability to require (None/Accept/Require)
GSSAPI channel binding, rendering Curl unable to connect to such
websites unless support for channel bindings is implemented.
IIS calls this feature Extended Protection (EPA), which is used in
Enterprise environments using Kerberos for authentication.
This change require krb5 >= 1.19, otherwise channel bindings won't be
forwarded through SPNEGO.
Viktor Szakats [Sat, 10 Aug 2024 07:33:18 +0000 (09:33 +0200)]
cmake: allow `pkg-config` in more envs
Before this patch, `pkg-config` was used for `UNIX` builds only (with
a few exceptions like wolfSSL, libssh, gsasl, libuv). This patch extends
`pkg-config` use to all envs except: `MSVC` without vcpkg. Meaning MSVC
with vcpkg will now use it. Also mingw on Windows.
Also apply the new condition to options where `pkg-config` was used
unconditionally (= for all targets). These are:
`-DCURL_USE_WOLFSSL=ON`, `-DCURL_USE_LIBSSH=ON`,
`-DCURL_USE_GSASL=ON` and `-DCURL_USE_LIBUV=ON`
This patch may still cause regressions for cross-builds (e.g. mingw
cross-build from Unix) and potentially other cases. If that happens, we
recommend using some of these methods to explicitly disable `pkg-config`
when using CMake:
- CMake option: `-DPKG_CONFIG_EXECUTABLE=`
(or `-DPKG_CONFIG_EXECUTABLE=nonexistent` or similar)
This is similar to the (curl-specific) `PKG_CONFIG` env for autotools.
- export env: `PKG_CONFIG_LIBDIR=`
(or `PKG_CONFIG_PATH`, `PKG_CONFIG_SYSROOT_DIR`,
or the CMake-specific `PKG_CONFIG`)
We may improve control over this in a future patch, also allowing opting
in MSVC (without vcpkg).
Daniel Stenberg [Fri, 9 Aug 2024 13:38:03 +0000 (15:38 +0200)]
multi: make the "general" list of easy handles a Curl_llist
Instead of having an especially "unique" linked list handler for the
main list of easy handles within the multi handle, this now uses a
regular Curl_llist for this as well.
With this change, it is also clearer that every easy handle added to a
multi handle belongs to one and only one out of three different lists:
process - the general one for normal transfer processing
pending - queued up waiting to get a connection (MSTATE_PENDING)
msgsent - transfer completed (MSTATE_MSGSENT)
An easy handle must therefore be removed from the current list before it
gets added to another.
Viktor Szakats [Tue, 6 Aug 2024 00:59:54 +0000 (02:59 +0200)]
cmake: limit libidn2 `pkg-config` detection to `UNIX`
libidn2 is detected by default, which triggers a `pkg-config` detectio
attempt by default. This in turn may pick up libidn2 inadvertently from
the disk, and append the libidn2 header directory to the include path.
This header directory might contain incompatible system and/or component
headers, causing confusion and failed builds.
Some of these side-effects may be the result of an unknowningly
configured (or misconfigured) `pkg-config`. In another reported case,
it was hit by the `pkg-config` from Strawberry Perl. Until we
investigate the reasons and come up with a technique to avoid these
issues, limit `pkg-config` detection to UNIX platforms, like we already
do in `Find*` modules.
Notice that `-DCURL_USE_LIBSSH=ON`, `-DCURL_USE_GSASL=ON`, and
`-DCURL_USE_LIBUV=ON` options continue to have the above side-effects,
though these options are disabled by default.
Viktor Szakats [Thu, 8 Aug 2024 20:31:24 +0000 (22:31 +0200)]
lib: fix AIX build issues
- memdebug: replace keyword `malloc` with `__malloc__` to
not interfere with envs where `malloc` is redefined. Also apply
the fix to `alloc_size`.
Fixes:
```
lib/memdebug.h:107:13: warning: unknown attribute 'vec_malloc' ignored [-Wunknown-attributes]
CURL_EXTERN ALLOC_FUNC FILE *curl_dbg_fdopen(int filedes, const char *mode,
^~~~~~~~~~
lib/memdebug.h:37:37: note: expanded from macro 'ALLOC_FUNC'
# define ALLOC_FUNC __attribute__((malloc))
^~~~~~
/usr/include/stdlib.h:753:16: note: expanded from macro 'malloc'
#define malloc vec_malloc
^~~~~~~~~~
```
- memdebug: always undef before defining.
Also do this for the rest of functions redefined in the same block.
Avoids warning on AIX:
```
lib/memdebug.h:117:9: warning: 'malloc' macro redefined [-Wmacro-redefined]
#define malloc(size) curl_dbg_malloc(size, __LINE__, __FILE__)
^
/usr/include/stdlib.h:753:9: note: previous definition is here
#define malloc vec_malloc
^
```
- easy: fix `-Wformat` warning on AIX by adding a cast.
```
lib/easy.c:608:47: warning: format specifies type 'int' but the argument has type 'long' [-Wformat]
"%" CURL_FORMAT_SOCKET_T ")", fds[i].fd);
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^~~~~~~~~~
```
- if2ip: silence compiler warning inside AIX system header.
```
/lib/if2ip.c:219:19: warning: signed shift result (0x80000000) sets the sign bit of the shift expression's type ('int') and becomes negative [-Wshift-sign-overflow]
if(ioctl(dummy, SIOCGIFADDR, &req) < 0) {
^~~~~~~~~~~
/usr/include/sys/ioctl.h:401:26: note: expanded from macro 'SIOCGIFADDR'
#define SIOCGIFADDR (int)_IOWR('i',33, struct oifreq) /* get ifnet address */
^~~~~~~~~~~~~~~~~~~~~~~~~~~~
/usr/include/sys/ioctl.h:174:23: note: expanded from macro '_IOWR'
#define _IOWR(x,y,t) (IOC_INOUT|((sizeof(t)&IOCPARM_MASK)<<16)|(x<<8)|y)
^~~~~~~~~
/usr/include/sys/ioctl.h:168:20: note: expanded from macro 'IOC_INOUT'
#define IOC_INOUT (IOC_IN|IOC_OUT)
^~~~~~
/usr/include/sys/ioctl.h:167:28: note: expanded from macro 'IOC_IN'
#define IOC_IN (0x40000000<<1) /* copy in parameters */
~~~~~~~~~~^ ~
```
Ref: https://curl.se/dev/log.cgi?id=20240808180420-3809007 Assisted-by: Dan Fandrich
Closes #14464
Viktor Szakats [Thu, 8 Aug 2024 17:09:56 +0000 (19:09 +0200)]
cmake: more small tidy-ups
- prefix local variables with underscore and convert to lowercase.
- list variables accepted by `libcurl.pc` and `curl-config` templates.
- quote more string literals.
Viktor Szakats [Wed, 7 Aug 2024 22:53:38 +0000 (00:53 +0200)]
cmake: more small tidy-ups and fixes
- tidy up two `MATCHES` expression by avoiding macros expansion and
adding quotes. Then convert then to `STREQUAL` to match other places
in the code doing the same checks.
- fix setting `_ALL_SOURCE` for AIX to match what autotools does.
- delete stray `_ALL_SOURCE` reference from `lib/config_riscos.h`
- simplify/fix two `STREQUAL ""` checks.
The one in the `openssl_check_symbol_exists()` macro succeeded
regardless of the value. The other could return TRUE when
`CMAKE_OSX_SYSROOT` was undefined.
- delete code for CMake versions (<3.7) we no longer support.
- prefer `LIST(APPEND ...)` to extend `CURL_LIBS`.
- use `CURL_LIBS` to add the `network` lib for Haiku.
Before this patch it was done via raw C flags. I could not test this.
- move `_WIN32_WINNT`-related code next to each other.
It also moves detection to the top, allowing more code to use
the result.
- merge two `WIN32` blocks.
- rename internal variables to underscore + lowercase.
Viktor Szakats [Wed, 7 Aug 2024 21:40:05 +0000 (23:40 +0200)]
cmake: tidy up more value comparisons
- unquote numeric value.
- `NOT LESS` -> `GREATER_EQUAL`.
- replace macro with variable name.
It also avoids this error when the variable is undefined:
```
CMake Error at CMakeLists.txt:1529 (if):
if given arguments:
"GREATER" "4"
Unknown arguments specified
```
https://github.com/curl/curl/actions/runs/10289921657/job/28478722584#step:30:356
Stefan Eissing [Wed, 7 Aug 2024 10:42:27 +0000 (12:42 +0200)]
http/2: simplify eos/blocked handling
- rely on the new flush to handle blocked sends. No longer
do simulated EAGAIN on (partially) blocked sends with their
need to handle repeats.
- fix some debug handling CURL_SMALLREQSEND env var
- add some assertings in request.c for affirming we do it right
- enhance assertion output in test_16 for easier analysis
Viktor Szakats [Tue, 6 Aug 2024 23:19:44 +0000 (01:19 +0200)]
cmake: fix `pkg-config`-based detection in `FindGSS.cmake`
Before this patch `pkg-config`-based detection was ignored, and used
solely as a path hint for native detection.
- fix `pkg_search_module()` result prefix to match what code expects:
`_GSS` (was: `_GSS_PKG`). Update variable that were in sync with old
prefix.
- update the pkg-config codepath to use `_GSS_MODULE_NAME` to detect
GSS flavour. This requires CMake 3.16.
Otherwise fall back to the old method. (The old method doesn't seem to
work anymore (?) as of CMake 3.30.1. Documented
`<prefix>_<modulename>_VERSION` variable is defined, but empty.)
- update the pkg-config codepath to use `_GSS_VERSION` set by CMake.
Resort to the old code when this variable is empty. (The old code
doesn't seem to work anymore (?) as of CMake 3.30.1)
- fix pkg-config codepath to set the documented result variables.
- align native detection variable names with those generated by
`pkg_search_module()` in the pkg-config codepath.
- GHA/macos: enable GSS Heimdal in a cmake job.
Uses the native detection.
- GHA/linux: enable GSS Heimdal in cmake and autotools jobs.
CMake uses `pkg-config`-based detection.
- suppress test 2077 and 2078 results on Linux + Heimdal.
```
FAIL-IGNORED 2077: 'curl --fail --negotiate to unauthenticated service fails' HTTP, HTTP GET, GSS-API
FAIL-IGNORED 2078: 'curl --negotiate should not send empty POST request only' HTTP, HTTP GET, GSS-API
```
Failing with valgrind errors in both autotools and cmake builds:
https://github.com/curl/curl/actions/runs/10282222581/job/28453472068?pr=14430#step:38:3638
https://github.com/curl/curl/actions/runs/10282222581/job/28453473398?pr=14430#step:38:7831
Viktor Szakats [Wed, 7 Aug 2024 02:27:49 +0000 (04:27 +0200)]
krb5: fix `-Wcast-align`
```
lib/krb5.c:343:39: warning: cast from 'void **' to 'unsigned char **' increases required alignment from 2 to 8 [-Wcast-align]
(unsigned char **)&_gssresp.value,
^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
Seen on macOS Intel with Apple clang and brew heimdal 7.8.0_1.
Viktor Szakats [Wed, 7 Aug 2024 01:15:04 +0000 (03:15 +0200)]
GHA/macos: tweak toolchain dump steps
- use documented flavour of `xcrun` option.
- show SDK version with a dedicated command.
(Sometimes the SDK path is a symlink and doesn't tell the version.
This is not at the moment the case in CI, but handle it anyway.)
- align group header with reality.
Preinstalled vs. installed Homebrew packages can be recognized
by their directory timestamps. Installed ones have a current date.
Dan Fandrich [Tue, 6 Aug 2024 22:00:05 +0000 (15:00 -0700)]
tests: show snapshot commit in testcurl
This disambiguates the source code being tested. The output format is
the same as when testing out of a git repo, but with no description and
a long hash.
Stefan Eissing [Tue, 6 Aug 2024 09:23:04 +0000 (11:23 +0200)]
curl: fix --test-event --parallel
(in debug-builds)
Fix implementation in curl using libuv to process parallel transfers.
Add pytest capabilities to run test cases with --test-event.
- fix uv_timer handling to carry correct 'data' pointing to uv context.
- fix uv_loop handling to reap and add transfers when possible
- fix return code when a transfer errored
Daniel Stenberg [Mon, 5 Aug 2024 21:54:13 +0000 (23:54 +0200)]
tests: ignore QUIT from FTP protocol comparisons
- QUIT is not an important FTP command
- curl only sends it "best effort", meaning it might not be sent
- it is a known "flaky" thing in test output because of this
Stefan Eissing [Tue, 6 Aug 2024 09:44:23 +0000 (11:44 +0200)]
curl: support repeated use of the verbose option; -vv etc
- make mentioning `-v` on the curl command line increase the
verbosity of the trace output
- related discussion https://github.com/curl/curl/discussions/13810
- make a single -v revert all previous -v+ changes
- make --no-verbose also reset all trace configs
Viktor Szakats [Tue, 6 Aug 2024 09:45:39 +0000 (11:45 +0200)]
cmake: cleanup header paths
- sync build-dir/source-dir header path order with autotools, by
including build-dir first, then source-dir.
This prevents out-of-tree builds breaking due to leftover generated
headers in the source tree.
- tests/unit: move `src` ahead of `libtest` in header path, syncing with
autotools.
- stop adding non-existing generated `include` dir to header path.
There are no generated `include` headers and this directory is either
missing in out-of-tree builds or the same as the one already added
globally via the root `CMakeLists.txt`.
- lib: stop adding a duplicate source include directory to the header
path.
It's already added globally via the root `CMakeLists.txt`.
- lib: stop adding the project root to the header path.
- docs/examples: drop internal header paths.
Examples do not and should not use internal headers.
- replace `curl_setup_once.h` in comments with `curl_setup.h`,
the header actually used, and also referred to in autotools comments.
- add comment why we need `src` in include path for `tests/server`.
Viktor Szakats [Mon, 5 Aug 2024 23:38:32 +0000 (01:38 +0200)]
cmake: use numeric comparison for `HAVE_WIN32_WINNT`
Turns out CMake supports numeric comparison with hexadecimal values.
Confirmed in GHA/linux-old with CMake 3.7.2. I could not find
documentation about this, but our CMakeLists.txt already used it before
this patch.
Extend that method to two more comparisons.
Also pad the value in the existing one to 4 digits.
The padding/lowercasing logic when setting `HAVE_WIN32_WINNT` is no
longer required, but keep it anyway for uniform log output.
Viktor Szakats [Sun, 4 Aug 2024 15:18:39 +0000 (17:18 +0200)]
cmake: add Linux CI job, fix pytest with cmake
- extend existing Linux workflow with CMake support.
Including running pytest the first time with CMake.
- cmake: generate `tests/config` and `tests/http/config.ini`.
Required for pytest tests.
Uses basic detection logic. Feel free to take it from here.
Also dump config files in a CI step for debugging purposes.
- cmake: build `tests/http/clients` programs.
- fix portability issues with `tests/http/clients` programs.
Some of them use `getopt()`, which is not supported by MSVC.
Fix the rest to compile in CI (old-mingw-w64, MSVC, Windows).
- GHA/linux: add CMake job matching an existing autotools one.
- GHA/linux: test `-DCURL_LIBCURL_VERSIONED_SYMBOLS=ON`
in the new CMake job.
- reorder testdeps to build server, client tests first and then
libtests and units, to catch errors in the more complex/unique
sources earlier.
Viktor Szakats [Mon, 5 Aug 2024 16:44:37 +0000 (18:44 +0200)]
cmake: add support for `CURL_USE_LIBUV` option
Also use an `#undef` hack for CMake Unity builds to avoid the previously
included `memdebug.h` header messing up the declarations pulled in by
`uv.h`:
```
In file included from ~/curl/bld/src/CMakeFiles/curl.dir/Unity/unity_0_c.c:88:
In file included from ~/curl/src/tool_operate.c:54:
In file included from /usr/local/Cellar/libuv/1.48.0/include/uv.h:71:
In file included from /usr/local/Cellar/libuv/1.48.0/include/uv/unix.h:34:
/Library/Developer/CommandLineTools/SDKs/MacOSX13.1.sdk/usr/include/netdb.h:269:7: error: expected parameter declarator
void freeaddrinfo(struct addrinfo *);
^
~/curl/lib/memdebug.h:167:31: note: expanded from macro 'freeaddrinfo'
curl_dbg_freeaddrinfo(data, __LINE__, __FILE__)
^
```
Austin Moore [Sun, 4 Aug 2024 03:43:45 +0000 (23:43 -0400)]
aws_sigv4: fix canon order for headers with same prefix
If a request containing two headers that have equivalent prefixes (ex.
"x-amz-meta-test:test" and "x-amz-meta-test-two:test2") AWS expects the
header with the shorter name to come first. The previous implementation
used `strcmp` on the full header. Using the example, this would result
in a comparison between the ':' and '-' chars and sort
"x-amz-meta-test-two" before "x-amz-meta-test", which produces a
different "StringToSign" than the one calculated by AWS.
Jan Venekamp [Sun, 4 Aug 2024 18:07:17 +0000 (20:07 +0200)]
wolfssl: add CURLOPT_TLS13_CIPHERS support
Bring setting ciphers with WolfSSL in line with other SSL backends,
to make the curl interface more consistent across the backends.
Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.
This changes the behaviour for WolfSSL when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.
Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.
Jan Venekamp [Sun, 4 Aug 2024 18:06:27 +0000 (20:06 +0200)]
mbedtls: add CURLOPT_TLS13_CIPHERS support
Bring setting ciphers with mbedTLS in line with other SSL backends,
to make the curl interface more consistent across the backends.
Now the tls1.3 ciphers are set with the --tls13-ciphers option, when
not set the default tls1.3 ciphers are used. The tls1.2 (1.1, 1.0)
ciphers are set with the --ciphers option, when not set the default
tls1.2 ciphers are used. The ciphers available for the connection
are now a union of the tls1.3 and tls1.2 ciphers.
This changes the behaviour for mbedTLS when --ciphers is set, but
--tls13-ciphers is not set. Now the ciphers set with --ciphers
are combined with the default tls1.3 ciphers, whereas before solely
the ciphers of --ciphers were used.
Thus before when no tls1.3 ciphers were specified in --ciphers,
tls1.3 was completely disabled. This might not be what the user
expected, especially as this does not happen with OpenSSL.
Stefan Eissing [Sun, 4 Aug 2024 07:51:26 +0000 (09:51 +0200)]
http2: improved upload eos handling
- replace the counting of upload lengths with the new eos send flag
- improve frequency of stream draining to happen less on events where it
is not needed
- this PR is based on #14220
http2, cf-h2-proxy: fix EAGAINed out buffer
- in adjust pollset and shutdown handling, a non-empty `ctx->outbufq`
must trigger send polling, irregardless of http/2 flow control
- in http2, fix retry handling of blocked GOAWAY frame
test case improvement:
- let client 'upload-pausing' handle http versions
Stefan Eissing [Sun, 4 Aug 2024 11:22:03 +0000 (13:22 +0200)]
wolfssl: improve shutdown handling
Improve handling of shutdown when sending gets blocked.
Add workaround for <https://github.com/wolfSSL/wolfssl/issues/7784>
where wolfSSL keeps on adding close notify messages to its outgoing
buffer on ever attempt.
Stefan Eissing [Sun, 4 Aug 2024 11:08:51 +0000 (13:08 +0200)]
openssl: improve shutdown handling
Make sure that `io_need` is cleared and set at the filter operations.
Add some more tracing for shutdown situations.
Improve shutdown handling for blocked sends. OpenSSL is a bit tricksy
here that it only reports WANT_WRITE on SSL_shutdown(), but never on
SSL_read() on blocked sends. So we need to use both.
At last, set SSL_MODE_ACCEPT_MOVING_WRITE_BUFFER when available since we
are not always retrying sends from the very same address, as testing
showed.
Alex Snast [Tue, 25 Jun 2024 16:34:24 +0000 (09:34 -0700)]
wolfssl: add support for ssl cert blob / ssl key blob options
wolfSSL supports setting certificates/private keys from memory blobs
which allow us to implement both CURLOPT_SSLCERT_BLOB and
CURLOPT_SSLKEY_BLOB options.
Viktor Szakats [Sun, 4 Aug 2024 11:52:44 +0000 (13:52 +0200)]
cmake: add support for versioned symbols option
Implement the `--enable-versioned-symbols` feature available in
`./configure` for CMake.
Enable with `-DCURL_LIBCURL_VERSIONED_SYMBOLS=ON`. Customize the version
prefix with `-DCURL_LIBCURL_VERSIONED_SYMBOLS_PREFIX=MYPREFIX_`. By
default the prefix matches what autotools uses.
When enabled, the feature is detected and a warning shown if
unavailable. (E.g. on Apple and Windows, it isn't.)
Included `HIDDEN {};` to match autotools, though I don't know if it's
necessary, useful or making any difference.
Differences from the autotools implementation:
- soversion is dynamic instead of hard-coded.
- omits referencing non-curl symbols.
- allows prefix/flavour override.
- more universal feature detection.
- doesn't rely on the in-repo `lib/libcurl.vers.in` file.
Also:
- add mbedTLS and BearSSL versioned symbol prefix support to autotools.
- enable this option in an old-linux job.
projects / thirdparty / curl.git / log
