Daniel Stenberg [Mon, 14 Oct 2024 12:09:59 +0000 (14:09 +0200)]
curl.h: remove the struct pointer for CURL/CURLSH/CURLM typedefs
It makes the callbacks get different signnatures when used from within
libcurl vs outside of it by libcurl-using applications (such as the
libtests) and this triggers UndefinedBehaviorSanitizer errors.
Stefan Eissing [Thu, 5 Sep 2024 14:41:53 +0000 (16:41 +0200)]
ftp: move listen handling to socket filter
Move the listen/accept handling of the FTP active data connection
into the socket filter and monitor 'connected' status of that as
with passive connections - more or less.
The advantage is that the socket filter now reports being connected
only when the server has actually called and accept() has been done.
This enables to bootstrap the filter chain on the data connection
just like any other. A require SSL filter can then be added right
at the start and does not need to be patched in later.
Still, the active connection keeps on needing special handling in
ftp.c as the control connection needs to be monitored while waiting
as the server might send error responses this way. So, things did
not turn out quite as squeaky clean as hoped for, but still seems
better to do that way.
Stefan Eissing [Thu, 10 Oct 2024 09:44:39 +0000 (11:44 +0200)]
wolfSSL: fix handling of TLSv1.3 sessions
Register a callback to get notified of new SSL sessions by wolfSSL.
Remove the explicit session retrieval after handshake, since this does
not work for TLSv1.3.
Adjust test expectations now that TLSv1.3 session resumption works
in wolfSSL.
Viktor Szakats [Fri, 11 Oct 2024 08:53:32 +0000 (10:53 +0200)]
smb: do not redefine `getpid` on Windows
Replace with namespaced local macro `Curl_getpid()`.
Redefining symbols can backfire if that symbol is used in system
headers, especially with unity build. We haven't seen a fallout in CI
or supported envs, but do it anyway for good measure.
Viktor Szakats [Sat, 12 Oct 2024 08:45:31 +0000 (10:45 +0200)]
GHA: optimize test prereq steps
- Linux: move test and pytest prereqs right before test run.
- returns build phase results faster.
- allows skipping steps for jobs that don't need them.
- makes dependencies more transparent.
- sync prereq install step names.
- use `tests/requirements.txt` more.
Viktor Szakats [Fri, 11 Oct 2024 16:36:43 +0000 (18:36 +0200)]
cmake: tidy-ups and rebase fixups
- limit `SIZEOF_SA_FAMILY_T` detection to non-Windows.
- make sure `sys/socket.h` exists before detecting `SIZEOF_SA_FAMILY_T`.
- limit `mach_absolute_time()` detection to `APPLE`. Drop from Windows
pre-cache.
- skip `HAVE_LIBSOCKET` detection for Windows, drop pre-cached value.
- drop redundant pre-cached `HAVE_LIBZ` for Windows.
- `curl_required_libpaths()`: stop accepting multiple arguments.
To prepare for `CMAKE_REQUIRED_LINK_DIRECTORIES` support.
Follow-up to 7bab201abe3915a0167c002f9308950cb8a06e4b #15193
- GSS: fix recent rebase mistakes:
- fix variable name.
- do not add a header twice.
Follow-up to 91d451b48809f20415ba8627786f5d4f5aaf8bfe #15157
- GSS: quote a variable.
Dan Fandrich [Wed, 9 Oct 2024 23:47:20 +0000 (16:47 -0700)]
tool_xattr: create the user.creator xattr attribute
This indicates that the file was created by curl which can help a user
determine the origin of a file. Like the other attributes, this is only
enabled with the --xattr option.
- reduce `check_include_file_concat()` use to those headers that either
depend on a previously detected header, or another header or symbol
detection depend on it.
- replace `check_symbol_exists()` with `check_function_exists()` for
functions that are detected with `AC_CHECK_FUNCS()` in `./configure`.
This makes `setmode()` no longer be detected with MSYS, syncing
this with `./configure`. Instead `_setmode()` is used now also in
CMake MSYS builds. This is consistent with Cygwin builds also.
- add comment about which header/symbol detection depends on what
header. Based on `./configure` mainly.
- form `CURL_TEST_DEFINES` manually, and include only those macros which
are actually used in `CMake/CurlTests.c`.
- change `curl_internal_test()` to use `CMAKE_REQUIRED_DEFINITIONS`,
instead of `CMAKE_REQUIRED_FLAGS` to simplify the logic, and to allow
dropping the latter macro completely.
- drop `windows.h` from header and symbol checks.
- `./configure`: add comment about whether `netinet/in6.h`, `sys/un.h`
are indeed meant to be included for all detections. There is a chance
they were added there by accident.
Stefan Eissing [Wed, 9 Oct 2024 12:46:32 +0000 (14:46 +0200)]
TLS: TLSv1.3 earlydata support for curl
Based on #14135, implement TLSv1.3 earlydata support for the curl
command line, libcurl and its implementation in GnuTLS.
If a known TLS session announces early data support, and the feature is
enabled *and* it is not a "connect-only" transfer, delay the TLS
handshake until the first request is being sent.
- Add --tls-earldata as new boolean command line option for curl.
- Add CURLSSLOPT_EARLYDATA to libcurl to enable use of the feature.
- Add CURLINFO_EARLYDATA_SENT_T to libcurl, reporting the amount of
bytes sent and accepted/rejected by the server.
Implementation details:
- store the ALPN protocol selected at the SSL session.
- When reusing the session and enabling earlydata, use exactly
that ALPN protocol for negoptiation with the server. When the
sessions ALPN does not match the connections ALPN, earlydata
will not be enabled.
- Check that the server selected the correct ALPN protocol for
an earlydata connect. If the server does not confirm or reports
something different, the connect fails.
- HTTP/2: delay sending the initial SETTINGS frames during connect,
if not connect-only.
Verification:
- add test_02_32 to verify earlydata GET with nghttpx.
- add test_07_70 to verify earlydata PUT with nghttpx.
- add support in 'hx-download', 'hx-upload' clients for the feature
Viktor Szakats [Sat, 5 Oct 2024 00:12:13 +0000 (02:12 +0200)]
cmake: replace `check_include_file_concat()` for LDAP and GSS detection
Replace `check_include_file_concat()` with `check_include_file()` in
GSS/LDAP detection to avoid these headers spilling into subsequent
feature checks.
- For LDAP, reverse detection order to match with `./configure`.
Though, in current LDAP packages `ldap.h` does include `lber.h`.
- For GSS, align header detection logic with `./configure`, where
`gssapi/gssapi_generic.h` might require `gssapi/gssapi.h`, and
`gssapi/gssapi_krb5.h` might require both.
Viktor Szakats [Sat, 5 Oct 2024 11:59:28 +0000 (13:59 +0200)]
cmake: add comments to feature check options applied globally
Add comments saying when we want values set in feature check option
variables to apply to all feature checks, globally. These are currently:
`ws2_32` and `socket` libraries, and `-D_WIN32_WINNT=` macro.
Also use `list(APPEND ...)` for the libraries to avoid overwriting
potentially existing values.
Viktor Szakats [Sat, 5 Oct 2024 12:01:49 +0000 (14:01 +0200)]
cmake: stop adding dependency headers to global `CMAKE_REQUIRED_INCLUDES`
It was done for `zlib`, `brotli`, `libpsl`, `libssh2`, `wolfssh`
(a copy-paste case for `wolfssh`).
Feature detections should not rely by default on dependency headers.
There is no evidence they do now. If it becomes necessary, headers
should added for the duration of the feature check.
Viktor Szakats [Fri, 4 Oct 2024 23:12:44 +0000 (01:12 +0200)]
cmake: use `cmake_push_check_state()` around feature checks
Enclose
`CMAKE_EXTRA_INCLUDE_FILES`,
`CMAKE_REQUIRED_DEFINITIONS`,
`CMAKE_REQUIRED_FLAGS`,
`CMAKE_REQUIRED_INCLUDES`,
`CMAKE_REQUIRED_LIBRARIES`,
`CMAKE_REQUIRED_LINK_OPTIONS`,
settings within `cmake_push_check_state()`/`cmake_pop_check_state()`
calls. It prevents spilling them into other feature checks. It also
replaces manual resets found in some places (which can have
the undesired side-effect of destroying values meant for global use.)
Also:
- detect and add required system libraries for Rustls on macOS and
non-Windows.
- add Linux CMake jobs for the touched dependencies.
Caveats:
- MSH3 generates a broken `libmsh3.pc`, so needs manual config.
Upstream PR: https://github.com/nibanks/msh3/pull/225
- Rustls `.pc` file missing, so needs manual config.
An internal change worthy of mention is that we are using the lib path
and name information returned by `pkg-config` as-is. Meaning the libname
doesn't include the full path, like it's usual with native cmake
detection. The path comes separately and needs to be rolled separately.
For this we add it to targets via `link_directories()`. We also keep tab
of them in `CURL_LIBDIRS` and use that in `libcurl.pc`. Feature checks
also need to receive these paths. CMake doesn't offer
a `CMAKE_REQUIRED_*` variable for this purpose, only
a `CMAKE_REQUIRED_LINK_OPTIONS` accepting raw linker flags. Add a macro
to convert a list of paths to linker options to solve it. wolfSSL
requires this for now.
Viktor Szakats [Thu, 10 Oct 2024 09:21:09 +0000 (11:21 +0200)]
GHA/linux, http3-linux: add CMake support, sync steps, other improvements
- use shallow clone for submodules.
- reduce total job timeout from 90/60 -> 45 minutes.
- use `$HOME` instead of literal.
- http3-linux: sync step yaml order with linux.yml.
- http3-linux: add cmake + ninja support like in linux.yml.
- http3-linux: dump confgure log, test config, curl -V like in linux.yml.
- http3-linux: skip restoring gnutls and wolfssl when not used.
- dump `curl_config.h`.
- fold a long line.
Viktor Szakats [Thu, 10 Oct 2024 00:22:29 +0000 (02:22 +0200)]
GHA/linux: fix mbedTLS cmake build
CMake builds mbedTLS in Debug mode by default, which was the reason
for these consistent test failures:
```
FAIL 1631: 'FTP through HTTPS-proxy' FTP, HTTPS-proxy
FAIL 1632: 'FTP through HTTPS-proxy, with connection reuse' FTP, HTTPS-proxy
```
Sometimes also:
```
FAIL 303: 'HTTPS with 8 secs timeout' HTTPS, HTTP GET, timeout, FAILURE
```
https://github.com/curl/curl/actions/runs/11260616621/job/31313234198
Fix it by building in `RelWithDebInfo` mode, matching the bare
`Makefile` builds used earlier. (`Release` mode also works.)
Emanuel Komínek [Tue, 8 Oct 2024 16:35:13 +0000 (18:35 +0200)]
multi: make curl_multi_cleanup invalidate magic latter
When a multi handle is being cleaned up, it can still cause user
callbacks to be fired. Notably Curl_cpool_destroy calls socket_callback
on all pooled connections. It's still possible for the callback to call
curl_multi_assign leading to an assert.
This commit moves clearing of a multi handle magic to a point where the
multi handle stops being a valid object.
Viktor Szakats [Tue, 8 Oct 2024 20:41:18 +0000 (22:41 +0200)]
GHA: add Linux and macOS mbedTLS jobs, fix issue
- update mbedTLS repo URL.
- switch local mbedTLS build to use CMake, and Ninja.
CMake build is required to create and install mbedTLS `pkg-config`
files. (as of v3.6.1)
`-DCMAKE_POSITION_INDEPENDENT_CODE=ON` required to avoid this error
when linking mbedtls to `libcurl.so`:
```
/usr/bin/ld: /home/runner/mbedtls/lib/libmbedcrypto.a(cipher.c.o): warning: relocation against `mbedtls_cipher_base_lookup_table' in read-only section `.text'
/usr/bin/ld: /home/runner/mbedtls/lib/libmbedtls.a(ssl_tls.c.o): relocation R_X86_64_PC32 against symbol `mbedtls_x509_crt_profile_suiteb' can not be used when making a shared object; recompile with -fPIC
/usr/bin/ld: final link failed: bad value
```
Ref: https://github.com/curl/curl/actions/runs/11245069259/job/31264386723#step:40:43
- make local mbedTLS build 10x smaller by omitting programs and tests.
- GHA/linux: fix cmake warning by adding `-B .` option.
- GHA/linux: add build-only cmake job for packaged mbedTLS (2.x).
- fix compiler warning when building with mbedTLS 2.x:
```
/home/runner/work/curl/curl/lib/vtls/mbedtls.c:344:1: error: ‘mbed_cipher_suite_get_str’ defined but not used [-Werror=unused-function]
344 | mbed_cipher_suite_get_str(uint16_t id, char *buf, size_t buf_size,
| ^~~~~~~~~~~~~~~~~~~~~~~~~
```
Ref: https://github.com/curl/curl/actions/runs/11244999065/job/31264168295#step:40:50
Viktor Szakats [Tue, 8 Oct 2024 18:18:56 +0000 (20:18 +0200)]
GHA/windows: drop vcpkg shiftmedia-gnutls, replace with mbedtls
GnuTLS vcpkg package broken again with the latest runner image update:
https://github.com/curl/curl/actions/runs/11240011311/job/31248406051?pr=15203#step:5:137
Viktor Szakats [Mon, 7 Oct 2024 14:13:32 +0000 (16:13 +0200)]
cmake: detect GNU GSS
Fix to set `HAVE_GSSGNU` when GNU GSS is detected.
Also set the appropriate `pkg-config` dependency and do version
detection for the GNU GSS flavour.
Tested with `pkg-config` and partly tested without. The latter case
picks up everything else but, in my env. This is likely not the last
word to implement this detection correctly for all build-cases.
GNU GSS doesn't seem to have a Homebrew formula and building
it locally needs manual tweaks to make finish successfully.
Also move a MIT-specific header detection into to MIT-specific `if`
branch.
- `HAVE_CLOSESOCKET_CAMEL`, `HAVE_PROTO_BSDSOCKET_H`
are for AmigaOS.
(Note: `./configure` tries to detect these for all targets, cmake does
it only for AmigaOS, to not inflate configure time.)
Viktor Szakats [Sun, 6 Oct 2024 13:41:48 +0000 (15:41 +0200)]
build: detect and use `_setmode()` with Cygwin/MSYS, also use on Windows
Before this patch `setmode()` was not detected with Cygwin/MSYS, because
it's a macro, not a function, and detection is looking for a function.
Switching to symbol detection doesn't work because it mis-detects it on
BSD systems which features a function with the same name but different
functionality and arguments.
Fix it by looking for a `_setmode()` function on Cygwin/MSYS, and use it
if available.
`_setmode()` is recommended over `setmode()` by Windows documentation so
use that on Windows too. It seems to be available on all supported
compilers, so omit detection.
Officially Windows requires argument `_O_BINARY` with an underscore.
`O_BINARY` is also supported but bound to conditions. Continue to use it
for simplicity. Cygwin supports `O_BINARY` (no underscore).
Dan Fandrich [Thu, 26 Sep 2024 18:33:58 +0000 (11:33 -0700)]
CI: run pytype and ruff on Python code
These ensure a more consistent style and can find some errors statically
that would otherwise only be seen at run-time. Also, bump the Ubuntu
version of some other checks to get newer versions of some linters.
Daniel Stenberg [Thu, 3 Oct 2024 21:53:10 +0000 (23:53 +0200)]
asyn-ares: remove typecast, fix expire
- Use the appropriate variable type for the curlx_tvtoms() return code:
timediff_t and remove the typecast.
- Simplify the function and avoid the odd expire adjustment that
probably is a rest from ancient days when the expire function did not
handle zero millisecond timeouts.
Stefan Eissing [Tue, 1 Oct 2024 10:48:46 +0000 (12:48 +0200)]
test1915: add tracing and connect timeout
Since we see Windows fails of 1915, add tracing and a connect timeout.
The test uses a port no one is supposed to listen on, but Windows has
this weird wait logic. So, set a short timeout.
Daniel Stenberg [Thu, 3 Oct 2024 12:37:00 +0000 (14:37 +0200)]
urlapi: normalize the IPv6 address
As the parsing and address "regeneration" are done anyway, we might as
well use the updated version in the result and thereby A) get a
normalized (and lower cased) version of the address and B) avoid a
strcpy().
Stefan Eissing [Thu, 3 Oct 2024 08:51:26 +0000 (10:51 +0200)]
openssl quic: populate x509 store before handshake
Since OpenSSL does its own send/recv internally, we may miss the moment
to populate the x509 store right before the server response. Do it
instead before we start the handshake, at the loss of the time to set
this up.
Stefan Eissing [Thu, 3 Oct 2024 09:10:29 +0000 (11:10 +0200)]
pytest: improve pytest_07_42a reliability
Due to timings on paused response receive, the error code may vary due
to the location where it is detected that the server closed the transfer
prematurely.
Be more lenient in always allowing PARTIAL_FILE as ok.