Patrick Monnerat [Thu, 25 Jan 2024 12:58:19 +0000 (13:58 +0100)]
sasl: make login option string override http auth
- Use http authentication mechanisms as a default, not a preset.
Consider http authentication options which are mapped to SASL options as
a default (overriding the hardcoded default mask for the protocol) that
is ignored if a login option string is given.
Prior to this change, if some HTTP auth options were given, sasl mapped
http authentication options to sasl ones but merged them with the login
options.
That caused problems with the cli tool that sets the http login option
CURLAUTH_BEARER as a side-effect of --oauth2-bearer, because this flag
maps to more than one sasl mechanisms and the latter cannot be cleared
individually by the login options string.
Daniel Stenberg [Sun, 21 Jan 2024 23:04:47 +0000 (00:04 +0100)]
pingpong: stop using the download buffer
The pingpong logic now uses its own dynbuf for receiving command
response data.
When the "final" response header for a commanad has been received, that
final line is left first in the recvbuf for the protocols to parse at
will. If there is additional data behind the final response line, the
'overflow' counter is indicate how many bytes.
Viktor Szakats [Wed, 24 Jan 2024 12:25:15 +0000 (12:25 +0000)]
cmake: rework options to enable curl and libcurl docs
Rework CMake options for building/using curl tool and libcurl manuals.
- rename `ENABLE_MANUAL` to `ENABLE_CURL_MANUAL`, meaning:
to build man page and built-in manual for curl tool.
- rename `BUILD_DOCS` to `BUILD_LIBCURL_DOCS`, meaning:
to build man pages for libcurl.
- `BUILD_LIBCURL_DOCS` now works without having to enable
`ENABLE_CURL_MANUAL` too.
- drop support for existing CMake-level `USE_MANUAL` option to avoid
confusion. (It used to work with the effect of current
`ENABLE_CURL_MANUAL`, but only by accident.)
Assisted-by: Richard Levitte
Ref: #12771
Closes #12773
Daniel Stenberg [Tue, 23 Jan 2024 14:12:09 +0000 (15:12 +0100)]
GHA: add a job scanning for "bad words" in markdown
This means words, phrases or things we have decided not to use - words that
are spelled right according to the dictionary but we want to avoid. In the
name of consistency and better documentation.
Viktor Szakats [Tue, 23 Jan 2024 06:40:11 +0000 (06:40 +0000)]
cmake: speed up curldown processing, enable by default
- cmake: enable `BUILD_DOCS` by default (this controls converting and
installing `.3` files from `.md` sources)
- cmake: speed up generating `.3` files by using a single command per
directory, instead of a single command per file. This reduces external
commands by about a thousand. (There remains some CMake logic kicking
in resulting in 500 -one per file- external `-E touch_nocreate` calls.)
- cd2nroff: add ability to process multiple input files.
- cd2nroff: add `-k` option to use the source filename to form the
output filename. (instead of the default in-file `Title:` line.)
Daniel Stenberg [Tue, 23 Jan 2024 16:45:22 +0000 (17:45 +0100)]
osslq: remove the TLS library from the version output
Since we only support using a single TLS library at any one time, we
know that the TLS library for QUIC is the same that is also shown for
regular TLS.
Fixes #12763 Reported-by: Viktor Szakats
Closes #12767
Daniel Stenberg [Sat, 20 Jan 2024 22:18:43 +0000 (23:18 +0100)]
docs/cmdline: change to .md for cmdline docs
- switch all invidual files documenting command line options into .md,
as the documentation is now markdown-looking.
- made the parser treat 4-space indents as quotes
- switch to building the curl.1 manpage using the "mainpage.idx" file,
which lists the files to include to generate it, instead of using the
previous page-footer/headers. Also, those files are now also .md
ones, using the same format. I gave them underscore prefixes to make
them sort separately:
_NAME.md, _SYNOPSIS.md, _DESCRIPTION.md, _URL.md, _GLOBBING.md,
_VARIABLES.md, _OUTPUT.md, _PROTOCOLS.md, _PROGRESS.md, _VERSION.md,
_OPTIONS.md, _FILES.md, _ENVIRONMENT.md, _PROXYPREFIX.md,
_EXITCODES.md, _BUGS.md, _AUTHORS.md, _WWW.md, _SEEALSO.md
Daniel Stenberg [Wed, 17 Jan 2024 10:32:44 +0000 (11:32 +0100)]
docs: introduce "curldown" for libcurl man page format
curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:
- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
their man page section is specified)
tools:
- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown
This setup generates .3 versions of all the curldown versions at build time.
CI:
Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...
Viktor Szakats [Mon, 22 Jan 2024 11:21:10 +0000 (11:21 +0000)]
libssh2: use `libssh2_session_callback_set2()` with v1.11.1
To avoid a local hack to pass function pointers and to avoid
deprecation warnings when building with libssh2 v1.11.1 or newer:
```
lib/vssh/libssh2.c:3324:5: warning: 'libssh2_session_callback_set' is deprecated: since libssh2 1.11.1. Use libssh2_session_callback_set2() [-Wdeprecated-declarations]
lib/vssh/libssh2.c:3326:5: warning: 'libssh2_session_callback_set' is deprecated: since libssh2 1.11.1. Use libssh2_session_callback_set2() [-Wdeprecated-declarations]
```
Ref: https://github.com/curl/curl-for-win/actions/runs/7609484879/job/20720821100#step:3:4982
Ref: https://github.com/libssh2/libssh2/pull/1285
Ref: https://github.com/libssh2/libssh2/commit/c0f69548be902147ce014ffa40b8db3cf1d4b0b4 Reviewed-by: Daniel Stenberg
Closes #12754
Stefan Eissing [Thu, 18 Jan 2024 12:07:07 +0000 (13:07 +0100)]
http3: initial support for OpenSSL 3.2 QUIC stack
- HTTP/3 for curl using OpenSSL's own QUIC stack together
with nghttp3
- configure with `--with-openssl-quic` to enable curl to
build this. This requires the nghttp3 library
- implementation with the following restrictions:
* macOS has to use an unconnected UDP socket due to an
issue in OpenSSL's datagram implementation
See https://github.com/openssl/openssl/issues/23251
This makes connections to non-reponsive servers hang.
* GET requests will send the indicator that they have
no body in a separate QUIC packet. This may result
in processing delays or Transfer-Encodings on proxied
requests
* uploads that encounter blocks will use 100% cpu as
detection of these flow control issue is not working
(we have not figured out to pry that from OpenSSL).
Viktor Szakats [Sun, 21 Jan 2024 15:38:09 +0000 (15:38 +0000)]
cmake: fix `ENABLE_MANUAL` option
Fix the `ENABLE_MANUAL` option. Set it to default to `OFF`.
Before this patch `ENABLE_MANUAL=ON` was a no-op, even though it was the
option designed to enable building and using the built-in curl manual.
(`USE_MANUAL=ON` option worked for this instead, by accident).
Viktor Szakats [Tue, 16 Jan 2024 16:30:07 +0000 (16:30 +0000)]
mbedtls: fix `-Wnull-dereference` and `-Wredundant-decls`
- Silence warning in mbedTLS v3.5.1 public headers:
```
./mbedtls/_x64-linux-musl/usr/include/psa/crypto_extra.h:489:14: warning: redundant redeclaration of 'psa_set_key_domain_parameters' [-Wredundant-decls]
./mbedtls/_x64-linux-musl/usr/include/psa/crypto_struct.h:354:14: note: previous declaration of 'psa_set_key_domain_parameters' was here
```
Ref: https://github.com/libssh2/libssh2/commit/ecec68a2c13a9c63fe8c2dc457ae785a513e157c
Ref: https://github.com/libssh2/libssh2/pull/1226
- Fix compiler warnings seen with gcc 9.2.0 + cmake unity:
```
./curl/lib/vtls/mbedtls.c: In function 'mbedtls_bio_cf_read':
./curl/lib/vtls/mbedtls.c:189:11: warning: null pointer dereference [-Wnull-dereference]
189 | nread = Curl_conn_cf_recv(cf->next, data, (char *)buf, blen, &result);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
./curl/lib/vtls/mbedtls.c: In function 'mbedtls_bio_cf_write':
./curl/lib/vtls/mbedtls.c:168:14: warning: null pointer dereference [-Wnull-dereference]
168 | nwritten = Curl_conn_cf_send(cf->next, data, (char *)buf, blen, &result);
| ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
```
Daniel Stenberg [Wed, 17 Jan 2024 13:27:16 +0000 (14:27 +0100)]
docs: cleanup nroff format use
- remove use of .BI for code snippet
- stop using .br, just do a blank line
- remove use of .PP
- remove use for .sp
- remove backslash in .IP
- use .IP instead of .TP
Jay Satiro [Sun, 7 Jan 2024 05:07:55 +0000 (00:07 -0500)]
tool_getparam: stop supporting `@filename` style for --cookie
The `@filename` style was never documented for --cookie <data|filename>
but prior to this change curl would accept it anyway and always treat a
@ prefixed string as a filename.
That's a problem if the string also contains a = sign because then it is
documented to be interpreted as a cookie string and not a filename.
Example:
`--cookie @foo=bar`
Before: Interpreted as load cookies from filename foo=bar.
After: Interpreted as cookie `@foo=bar` (name `@foo` and value `bar`).
Other curl options with a data/filename option-value use the `@filename`
to distinguish filenames which is probably how this happened. The
--cookie option has never been documented that way.
Stefan Eissing [Mon, 15 Jan 2024 12:02:34 +0000 (13:02 +0100)]
websockets: check for negative payload lengths
- in en- and decoding, check the websocket frame payload lengths for
negative values (from curl_off_t) and error the operation in that case
- add test 2307 to verify
Daniel Stenberg [Mon, 15 Jan 2024 15:28:04 +0000 (16:28 +0100)]
tool_operate: stop setting the file comment on Amiga
- the URL is capped at 80 cols, which ruins it if longer
- it does not strip off URL credentials
- it is done unconditonally, not on --xattr
- we don't have Amiga in the CI which makes fixing it blindly fragile
Someone who builds and tests on Amiga can add it back correctly in a
future if there is a desire.
Stefan Eissing [Mon, 15 Jan 2024 10:33:13 +0000 (11:33 +0100)]
rtsp: deal with borked server responses
- enforce a response body length of 0, if the
response has no Content-lenght. This is according
to the RTSP spec.
- excess bytes in a response body are forwarded to
the client writers which will report and fail the
transfer
Daniel Stenberg [Sun, 14 Jan 2024 16:54:51 +0000 (17:54 +0100)]
version: show only the libpsl version, not its dependencies
The libpsl version output otherwise also includes version number for its
dependencies, like IDN lib, but since libcurl does not use libpsl's IDN
functionality those components are not important.
Stefan Eissing [Fri, 1 Dec 2023 12:50:32 +0000 (13:50 +0100)]
lib: replace readwrite with write_resp
This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.
The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.
Curl's protocol handler `readwrite()` method been changed:
The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:
* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
finished
This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.
The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.
HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.
The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:
All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.
All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.
Daniel Stenberg [Thu, 11 Jan 2024 13:11:19 +0000 (14:11 +0100)]
configure: when enabling QUIC, check that TLS supports QUIC
Most importantly perhaps is when using OpenSSL that the used
build/flavor has the QUIC API: the vanilla OpenSSL does not, only
BoringSSL, libressl, AWS-LC and quictls do.
Sergey Markelov [Thu, 11 Jan 2024 00:23:00 +0000 (17:23 -0700)]
multi: remove total timer reset in file_do() while fetching file://
The total timer is properly reset in MSTATE_INIT. MSTATE_CONNECT starts
with resetting the timer that is a start point for further multi states.
If file://, MSTATE_DO calls file_do() that should not reset the total
timer. Otherwise, the total time is always less than the pre-transfer
and the start transfer times.
Stefan Eissing [Tue, 9 Jan 2024 08:29:34 +0000 (09:29 +0100)]
multi: pollset adjust, init with FIRSTSOCKET during connect
- `conn->sockfd` is set by `Curl_setup_transfer()`, but that
is called *after* the connection has been established
- use `conn->sock[FIRSTSOCKET]` instead