]> git.ipfire.org Git - thirdparty/curl.git/log
thirdparty/curl.git
18 months agofile+ftp: use stack buffers instead of data->state.buffer
Stefan Eissing [Thu, 25 Jan 2024 14:06:44 +0000 (15:06 +0100)] 
file+ftp: use stack buffers instead of data->state.buffer

Closes #12789

18 months agovtls: receive max buffer
Stefan Eissing [Fri, 26 Jan 2024 09:10:11 +0000 (10:10 +0100)] 
vtls: receive max buffer

- do not only receive one TLS record, but try to fill
  the passed buffer
- consider <4K remaning space is "filled".

Closes #12801

18 months agodocs: do not start lines/sentences with So, But nor And
Daniel Stenberg [Fri, 26 Jan 2024 09:19:30 +0000 (10:19 +0100)] 
docs: do not start lines/sentences with So, But nor And

Closes #12802

18 months agodocs: remove spurious ampersands from markdown
Daniel Stenberg [Fri, 26 Jan 2024 08:54:48 +0000 (09:54 +0100)] 
docs: remove spurious ampersands from markdown

They were leftovers from the nroff conversion.

Follow-up to eefcc1bda4bccd800f5a5

Closes #12800

18 months agosasl: make login option string override http auth
Patrick Monnerat [Thu, 25 Jan 2024 12:58:19 +0000 (13:58 +0100)] 
sasl: make login option string override http auth

- Use http authentication mechanisms as a default, not a preset.

Consider http authentication options which are mapped to SASL options as
a default (overriding the hardcoded default mask for the protocol) that
is ignored if a login option string is given.

Prior to this change, if some HTTP auth options were given, sasl mapped
http authentication options to sasl ones but merged them with the login
options.

That caused problems with the cli tool that sets the http login option
CURLAUTH_BEARER as a side-effect of --oauth2-bearer, because this flag
maps to more than one sasl mechanisms and the latter cannot be cleared
individually by the login options string.

New test 992 checks this.

Fixes https://github.com/curl/curl/issues/10259
Closes https://github.com/curl/curl/pull/12790

18 months agosocks: use own buffer instead of data->state.buffer
Stefan Eissing [Thu, 25 Jan 2024 13:56:57 +0000 (14:56 +0100)] 
socks: use own buffer instead of data->state.buffer

Closes #12788

18 months agosocks: fix generic output string to say SOCKS instead of SOCKS4
Daniel Stenberg [Thu, 25 Jan 2024 21:15:58 +0000 (22:15 +0100)] 
socks: fix generic output string to say SOCKS instead of SOCKS4

... since it was also logged for SOCKS5.

Closes #12797

18 months agotest742: test SOCKS5 with max length user, password and hostname
Daniel Stenberg [Thu, 25 Jan 2024 21:14:53 +0000 (22:14 +0100)] 
test742: test SOCKS5 with max length user, password and hostname

Adjusted the socksd server accordingly to allow for configuring that
long user name and password.

Closes #12797

18 months agossh: use stack scratch buffer for seeks
Stefan Eissing [Thu, 25 Jan 2024 14:47:08 +0000 (15:47 +0100)] 
ssh: use stack scratch buffer for seeks

- instead of data->state.buffer

Closes #12794

18 months agokrb5: access the response buffer correctly
Daniel Stenberg [Thu, 25 Jan 2024 15:10:03 +0000 (16:10 +0100)] 
krb5: access the response buffer correctly

As the pingpong code no longer uses the download buffer.

Folllow-up to c2d973627bab12ab
Pointed-out-by: Stefan Eissing
Closes #12796

18 months agomqtt: use stack scratch buffer for recv+publish
Stefan Eissing [Thu, 25 Jan 2024 14:33:54 +0000 (15:33 +0100)] 
mqtt: use stack scratch buffer for recv+publish

- instead of data->state.buffer

Closes #12792

18 months agotelnet, use stack scratch buffer for do
Stefan Eissing [Thu, 25 Jan 2024 14:43:14 +0000 (15:43 +0100)] 
telnet, use stack scratch buffer for do

- instead of data->state.buffer

Closes #12793

18 months agohttp, use stack scratch buffer
Stefan Eissing [Thu, 25 Jan 2024 14:14:23 +0000 (15:14 +0100)] 
http, use stack scratch buffer

- instead of data->state.buffer

Closes #12791

18 months agontlm_wb: do not use data->state.buf any longer
Stefan Eissing [Thu, 25 Jan 2024 12:01:28 +0000 (13:01 +0100)] 
ntlm_wb: do not use data->state.buf any longer

Closes #12787

18 months agogitignore: the generated `libcurl-symbols.md`
Stefan Eissing [Thu, 25 Jan 2024 14:49:40 +0000 (15:49 +0100)] 
gitignore: the generated `libcurl-symbols.md`

Closes #12795

18 months agotool: fix the listhelp generation command
Daniel Stenberg [Thu, 25 Jan 2024 10:14:29 +0000 (11:14 +0100)] 
tool: fix the listhelp generation command

The previous command line to generate the tool_listhelp.c source file
broke with 2494b8dd5175cee7.

Make 'make listhelp' invoked in src/ generate it. Also update the
comment in the file to mention the right procedure.

Closes #12786

18 months agohttp: check for "Host:" case insensitively
Daniel Stenberg [Thu, 25 Jan 2024 08:05:19 +0000 (09:05 +0100)] 
http: check for "Host:" case insensitively

When checking if the user wants to replace the header, the check should
be case insensitive.

Adding test 461 to verify

Found-by: Dan Fandrich
Ref: #12782
Closes #12784

18 months agoconfigure: add libngtcp2_crypto_boringssl detection
Tatsuhiro Tsujikawa [Wed, 24 Jan 2024 08:56:24 +0000 (17:56 +0900)] 
configure: add libngtcp2_crypto_boringssl detection

If OpenSSL is found to be BoringSSL or AWS-LC, and ngtcp2 is requested,
try to detect libngtcp2_crypto_boringssl.

Reported-by: ウさん
Fixes #12724
Closes #12769

18 months agohttp: remove comment reference to a removed solution
Daniel Stenberg [Thu, 25 Jan 2024 09:43:24 +0000 (10:43 +0100)] 
http: remove comment reference to a removed solution

Follow-up to 58974d25d

Closes #12785

18 months agopytest: Scorecard tracking CPU and RSS
Stefan Eissing [Fri, 19 Jan 2024 14:37:46 +0000 (15:37 +0100)] 
pytest: Scorecard tracking CPU and RSS

Closes #12765

18 months agoGHA: bump ngtcp2, gnutls, mod_h2, quiche
Graham Campbell [Thu, 25 Jan 2024 02:09:17 +0000 (02:09 +0000)] 
GHA: bump ngtcp2, gnutls, mod_h2, quiche

- ngtcp2 to v1.2.0
- gnutls to 3.8.3
- mod_h2 to 2.0.26
- quiche to 0.20.0

Closes #12778
Closes #12779
Closes #12780
Closes #12781

18 months agoftpserver.pl: send 213 SIZE response without spurious newline
Daniel Stenberg [Tue, 23 Jan 2024 12:01:37 +0000 (13:01 +0100)] 
ftpserver.pl: send 213 SIZE response without spurious newline

18 months agopingpong: stop using the download buffer
Daniel Stenberg [Sun, 21 Jan 2024 23:04:47 +0000 (00:04 +0100)] 
pingpong: stop using the download buffer

The pingpong logic now uses its own dynbuf for receiving command
response data.

When the "final" response header for a commanad has been received, that
final line is left first in the recvbuf for the protocols to parse at
will. If there is additional data behind the final response line, the
'overflow' counter is indicate how many bytes.

Closes #12757

18 months agogen.pl: remove bold from .IP used for ##
Daniel Stenberg [Thu, 25 Jan 2024 00:01:46 +0000 (01:01 +0100)] 
gen.pl: remove bold from .IP used for ##

Reported-by: Viktor Szakats
Fixes #12776
Closes #12777

18 months agocmake: rework options to enable curl and libcurl docs
Viktor Szakats [Wed, 24 Jan 2024 12:25:15 +0000 (12:25 +0000)] 
cmake: rework options to enable curl and libcurl docs

Rework CMake options for building/using curl tool and libcurl manuals.

- rename `ENABLE_MANUAL` to `ENABLE_CURL_MANUAL`, meaning:
  to build man page and built-in manual for curl tool.

- rename `BUILD_DOCS` to `BUILD_LIBCURL_DOCS`, meaning:
  to build man pages for libcurl.

- `BUILD_LIBCURL_DOCS` now works without having to enable
  `ENABLE_CURL_MANUAL` too.

- drop support for existing CMake-level `USE_MANUAL` option to avoid
  confusion. (It used to work with the effect of current
  `ENABLE_CURL_MANUAL`, but only by accident.)

Assisted-by: Richard Levitte
Ref: #12771
Closes #12773

18 months agourlapi: remove assert
Daniel Stenberg [Wed, 24 Jan 2024 12:15:18 +0000 (13:15 +0100)] 
urlapi: remove assert

This assert triggers wrongly when CURLU_GUESS_SCHEME and
CURLU_NO_AUTHORITY are both set and the URL is a single path.

I think this assert has played out its role. It was introduced in a
rather big refactor.

Follow-up to 4cfa5bcc9a

Reported-by: promptfuzz_ on hackerone
Closes #12775

18 months agotests: avoid int/size_t conversion size/sign warnings
Patrick Monnerat [Wed, 24 Jan 2024 08:51:09 +0000 (09:51 +0100)] 
tests: avoid int/size_t conversion size/sign warnings

Closes #12768

18 months agoGHA: add a job scanning for "bad words" in markdown
Daniel Stenberg [Tue, 23 Jan 2024 14:12:09 +0000 (15:12 +0100)] 
GHA: add a job scanning for "bad words" in markdown

This means words, phrases or things we have decided not to use - words that
are spelled right according to the dictionary but we want to avoid. In the
name of consistency and better documentation.

Closes #12764

18 months agocmake: speed up curldown processing, enable by default
Viktor Szakats [Tue, 23 Jan 2024 06:40:11 +0000 (06:40 +0000)] 
cmake: speed up curldown processing, enable by default

- cmake: enable `BUILD_DOCS` by default (this controls converting and
  installing `.3` files from `.md` sources)

- cmake: speed up generating `.3` files by using a single command per
  directory, instead of a single command per file. This reduces external
  commands by about a thousand. (There remains some CMake logic kicking
  in resulting in 500 -one per file- external `-E touch_nocreate` calls.)

- cd2nroff: add ability to process multiple input files.

- cd2nroff: add `-k` option to use the source filename to form the
  output filename. (instead of the default in-file `Title:` line.)

Follow-up to 3f08d80b2244524646ce86915c585509ac54fb4c
Follow-up to ea0b575dab86a3c44dd1d547dc500276266aa382 #12753
Follow-up to eefcc1bda4bccd800f5a56a0fe17a2f44a96e88b #12730

Closes #12762

18 months agodocs: install curl.1 with cmake as well
Richard Levitte [Mon, 22 Jan 2024 21:49:17 +0000 (22:49 +0100)] 
docs: install curl.1 with cmake as well

Closes #12759

18 months agoosslq: remove the TLS library from the version output
Daniel Stenberg [Tue, 23 Jan 2024 16:45:22 +0000 (17:45 +0100)] 
osslq: remove the TLS library from the version output

Since we only support using a single TLS library at any one time, we
know that the TLS library for QUIC is the same that is also shown for
regular TLS.

Fixes #12763
Reported-by: Viktor Szakats
Closes #12767

18 months agoCI: remove unnecessary OpenSSL 3 option `enable-tls1_3`
Stefan Eissing [Mon, 22 Jan 2024 16:31:25 +0000 (17:31 +0100)] 
CI: remove unnecessary OpenSSL 3 option `enable-tls1_3`

.. and switch OpenSSL 3 libdir from lib64 to lib for consistency.

Closes https://github.com/curl/curl/pull/12758

18 months agoGHA: bump nghttp2 version to v1.59.0
Stefan Eissing [Tue, 23 Jan 2024 15:35:08 +0000 (16:35 +0100)] 
GHA: bump nghttp2 version to v1.59.0

- Switch to v1.59.0 for GHA CI jobs that use a specific nghttp2-version.

Closes https://github.com/curl/curl/pull/12766

18 months agoRELEASE-NOTES: synced
Daniel Stenberg [Tue, 23 Jan 2024 13:34:36 +0000 (14:34 +0100)] 
RELEASE-NOTES: synced

18 months agodocs/cmdline: change to .md for cmdline docs
Daniel Stenberg [Sat, 20 Jan 2024 22:18:43 +0000 (23:18 +0100)] 
docs/cmdline: change to .md for cmdline docs

 - switch all invidual files documenting command line options into .md,
   as the documentation is now markdown-looking.

 - made the parser treat 4-space indents as quotes

 - switch to building the curl.1 manpage using the "mainpage.idx" file,
   which lists the files to include to generate it, instead of using the
   previous page-footer/headers. Also, those files are now also .md
   ones, using the same format. I gave them underscore prefixes to make
   them sort separately:
   _NAME.md, _SYNOPSIS.md, _DESCRIPTION.md, _URL.md, _GLOBBING.md,
   _VARIABLES.md, _OUTPUT.md, _PROTOCOLS.md, _PROGRESS.md, _VERSION.md,
   _OPTIONS.md, _FILES.md, _ENVIRONMENT.md, _PROXYPREFIX.md,
   _EXITCODES.md, _BUGS.md, _AUTHORS.md, _WWW.md, _SEEALSO.md

 - updated test cases accordingly

Closes #12751

18 months agoCI: bump actions/cache from 3 to 4
dependabot[bot] [Mon, 22 Jan 2024 14:36:50 +0000 (14:36 +0000)] 
CI: bump actions/cache from 3 to 4

Bumps [actions/cache](https://github.com/actions/cache) from 3 to 4.
- [Release notes](https://github.com/actions/cache/releases)
- [Changelog](https://github.com/actions/cache/blob/main/RELEASES.md)
- [Commits](https://github.com/actions/cache/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/cache
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
Closes #12756

18 months agoopenssl: when verifystatus fails, remove session id from cache
Daniel Stenberg [Mon, 22 Jan 2024 22:54:08 +0000 (23:54 +0100)] 
openssl: when verifystatus fails, remove session id from cache

To prevent that it gets used in a subsequent transfer that skips the
verifystatus check since that check can't be done when the session id is
reused.

Reported-by: Hiroki Kurosawa
Closes #12760

18 months agocmake: add option to disable building docs
Viktor Szakats [Mon, 22 Jan 2024 22:16:54 +0000 (23:16 +0100)] 
cmake: add option to disable building docs

18 months agocmake: use curldown to build man pages
Richard Levitte [Mon, 22 Jan 2024 06:49:08 +0000 (07:49 +0100)] 
cmake: use curldown to build man pages

This throws away the previous HTML and PDF producers, to mimic what
Makefile.am does as faithfully as possible.

Closes #12753

18 months agomksymbolsmanpage.pl: provide references to where the symbol is used
Daniel Stenberg [Fri, 19 Jan 2024 15:18:40 +0000 (16:18 +0100)] 
mksymbolsmanpage.pl: provide references to where the symbol is used

18 months agodocs: introduce "curldown" for libcurl man page format
Daniel Stenberg [Wed, 17 Jan 2024 10:32:44 +0000 (11:32 +0100)] 
docs: introduce "curldown" for libcurl man page format

curldown is this new file format for libcurl man pages. It is markdown
inspired with differences:

- Each file has a set of leading headers with meta-data
- Supports a small subset of markdown
- Uses .md file extensions for editors/IDE/GitHub to treat them nicely
- Generates man pages very similar to the previous ones
- Generates man pages that still convert nicely to HTML on the website
- Detects and highlights mentions of curl symbols automatically (when
  their man page section is specified)

tools:

- cd2nroff: converts from curldown to nroff man page
- nroff2cd: convert an (old) nroff man page to curldown
- cdall: convert many nroff pages to curldown versions
- cd2cd: verifies and updates a curldown to latest curldown

This setup generates .3 versions of all the curldown versions at build time.

CI:

Since the documentation is now technically markdown in the eyes of many
things, the CI runs many more tests and checks on this documentation,
including proselint, link checkers and tests that make sure we capitalize the
first letter after a period...

Closes #12730

18 months agolibssh2: use `libssh2_session_callback_set2()` with v1.11.1
Viktor Szakats [Mon, 22 Jan 2024 11:21:10 +0000 (11:21 +0000)] 
libssh2: use `libssh2_session_callback_set2()` with v1.11.1

To avoid a local hack to pass function pointers and to avoid
deprecation warnings when building with libssh2 v1.11.1 or newer:
```
lib/vssh/libssh2.c:3324:5: warning: 'libssh2_session_callback_set' is deprecated: since libssh2 1.11.1. Use libssh2_session_callback_set2() [-Wdeprecated-declarations]
lib/vssh/libssh2.c:3326:5: warning: 'libssh2_session_callback_set' is deprecated: since libssh2 1.11.1. Use libssh2_session_callback_set2() [-Wdeprecated-declarations]
```
Ref: https://github.com/curl/curl-for-win/actions/runs/7609484879/job/20720821100#step:3:4982

Ref: https://github.com/libssh2/libssh2/pull/1285
Ref: https://github.com/libssh2/libssh2/commit/c0f69548be902147ce014ffa40b8db3cf1d4b0b4
Reviewed-by: Daniel Stenberg
Closes #12754

18 months agotransfer: make the select_bits_paused condition check both directions
Daniel Stenberg [Mon, 22 Jan 2024 15:22:19 +0000 (16:22 +0100)] 
transfer: make the select_bits_paused condition check both directions

If there is activity in a direction that is not paused, return false.

Reported-by: Sergey Bronnikov
Bug: https://curl.se/mail/lib-2024-01/0049.html
Closes #12740

18 months agohttp3: initial support for OpenSSL 3.2 QUIC stack
Stefan Eissing [Thu, 18 Jan 2024 12:07:07 +0000 (13:07 +0100)] 
http3: initial support for OpenSSL 3.2 QUIC stack

- HTTP/3 for curl using OpenSSL's own QUIC stack together
  with nghttp3
- configure with `--with-openssl-quic` to enable curl to
  build this. This requires the nghttp3 library
- implementation with the following restrictions:
  * macOS has to use an unconnected UDP socket due to an
    issue in OpenSSL's datagram implementation
    See https://github.com/openssl/openssl/issues/23251
    This makes connections to non-reponsive servers hang.
  * GET requests will send the indicator that they have
    no body in a separate QUIC packet. This may result
    in processing delays or Transfer-Encodings on proxied
    requests
  * uploads that encounter blocks will use 100% cpu as
    detection of these flow control issue is not working
    (we have not figured out to pry that from OpenSSL).

Closes #12734

18 months agocmake: fix `ENABLE_MANUAL` option
Viktor Szakats [Sun, 21 Jan 2024 15:38:09 +0000 (15:38 +0000)] 
cmake: fix `ENABLE_MANUAL` option

Fix the `ENABLE_MANUAL` option. Set it to default to `OFF`.

Before this patch `ENABLE_MANUAL=ON` was a no-op, even though it was the
option designed to enable building and using the built-in curl manual.
(`USE_MANUAL=ON` option worked for this instead, by accident).

Ref: https://github.com/curl/curl/pull/12730#issuecomment-1902572409
Closes #12749

18 months agoTODO: update broken link to ratelimit-headers draft
Mohammadreza Hendiani [Fri, 19 Jan 2024 13:59:03 +0000 (17:29 +0330)] 
TODO: update broken link to ratelimit-headers draft

Closes #12741

18 months agocmake: when USE_MANUAL=YES, build the curl.1 man page
Daniel Stenberg [Fri, 19 Jan 2024 14:03:50 +0000 (15:03 +0100)] 
cmake: when USE_MANUAL=YES, build the curl.1 man page

Fixes KNOWN_BUG 15.4

Closes #12742

18 months agocmdline-opts/write-out.d: remove spurious double quotes
Daniel Stenberg [Fri, 19 Jan 2024 15:37:14 +0000 (16:37 +0100)] 
cmdline-opts/write-out.d: remove spurious double quotes

18 months agortsp: Convert assertion into debug log
Stefan Eissing [Fri, 19 Jan 2024 09:15:03 +0000 (10:15 +0100)] 
rtsp: Convert assertion into debug log

Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65934

- write excess bytes to the client where the standard excess bytes
  checks will report any wrongness and fail the transfer

Fixes #12738
Closes #12739

18 months agoheaders: remove assert from Curl_headers_push
Daniel Stenberg [Tue, 16 Jan 2024 22:50:02 +0000 (23:50 +0100)] 
headers: remove assert from Curl_headers_push

The fuzzer managed to reach the function without a terminating CR or LF
so let's handle it normally. While there, remove the goto.

Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65839

Closes #12721

18 months agocurl_easy_getinfo.3: remove the wrong time value count
Daniel Stenberg [Wed, 17 Jan 2024 08:42:54 +0000 (09:42 +0100)] 
curl_easy_getinfo.3: remove the wrong time value count

It said "six" time values but they are eight by now. Remove the mention
of the amount.

Closes #12727

18 months agombedtls: fix `-Wnull-dereference` and `-Wredundant-decls`
Viktor Szakats [Tue, 16 Jan 2024 16:30:07 +0000 (16:30 +0000)] 
mbedtls: fix `-Wnull-dereference` and `-Wredundant-decls`

- Silence warning in mbedTLS v3.5.1 public headers:
  ```
  ./mbedtls/_x64-linux-musl/usr/include/psa/crypto_extra.h:489:14: warning: redundant redeclaration of 'psa_set_key_domain_parameters' [-Wredundant-decls]
  ./mbedtls/_x64-linux-musl/usr/include/psa/crypto_struct.h:354:14: note: previous declaration of 'psa_set_key_domain_parameters' was here
  ```
  Ref: https://github.com/libssh2/libssh2/commit/ecec68a2c13a9c63fe8c2dc457ae785a513e157c
  Ref: https://github.com/libssh2/libssh2/pull/1226

- Fix compiler warnings seen with gcc 9.2.0 + cmake unity:
  ```
  ./curl/lib/vtls/mbedtls.c: In function 'mbedtls_bio_cf_read':
  ./curl/lib/vtls/mbedtls.c:189:11: warning: null pointer dereference [-Wnull-dereference]
    189 |   nread = Curl_conn_cf_recv(cf->next, data, (char *)buf, blen, &result);
        |           ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ./curl/lib/vtls/mbedtls.c: In function 'mbedtls_bio_cf_write':
  ./curl/lib/vtls/mbedtls.c:168:14: warning: null pointer dereference [-Wnull-dereference]
    168 |   nwritten = Curl_conn_cf_send(cf->next, data, (char *)buf, blen, &result);
        |              ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ```

- delete stray `#else`.

Closes #12720

18 months agodocs: cleanup nroff format use
Daniel Stenberg [Wed, 17 Jan 2024 13:27:16 +0000 (14:27 +0100)] 
docs: cleanup nroff format use

- remove use of .BI for code snippet
- stop using .br, just do a blank line
- remove use of .PP
- remove use for .sp
- remove backslash in .IP
- use .IP instead of .TP

Closes #12731

18 months agotest2307: fix expected failure code after ws refactoring
Stefan Eissing [Wed, 17 Jan 2024 08:53:38 +0000 (09:53 +0100)] 
test2307: fix expected failure code after ws refactoring

Fixes #12722
Closes #12728

18 months agocf-socket: show errno in tcpkeepalive error messages
Jay Satiro [Wed, 17 Jan 2024 05:10:40 +0000 (00:10 -0500)] 
cf-socket: show errno in tcpkeepalive error messages

- If the socket keepalive options (TCP_KEEPIDLE, etc) cannot be set
  then show the errno in the verbose error messages.

Ref: https://github.com/curl/curl/discussions/12715#discussioncomment-8151652

Closes https://github.com/curl/curl/pull/12726

18 months agotool_getparam: stop supporting `@filename` style for --cookie
Jay Satiro [Sun, 7 Jan 2024 05:07:55 +0000 (00:07 -0500)] 
tool_getparam: stop supporting `@filename` style for --cookie

The `@filename` style was never documented for --cookie <data|filename>
but prior to this change curl would accept it anyway and always treat a
@ prefixed string as a filename.

That's a problem if the string also contains a = sign because then it is
documented to be interpreted as a cookie string and not a filename.

Example:

`--cookie @foo=bar`

Before: Interpreted as load cookies from filename foo=bar.

After: Interpreted as cookie `@foo=bar` (name `@foo` and value `bar`).

Other curl options with a data/filename option-value use the `@filename`
to distinguish filenames which is probably how this happened. The
--cookie option has never been documented that way.

Ref: https://curl.se/docs/manpage.html#-b

Closes https://github.com/curl/curl/pull/12645

18 months agowebsockets: refactor decode chain
Stefan Eissing [Tue, 16 Jan 2024 11:06:57 +0000 (12:06 +0100)] 
websockets: refactor decode chain

- use client writer stack for decoding frames
- move websocket protocol handler to ws.c

Closes #12713

18 months agowebsockets: check for negative payload lengths
Stefan Eissing [Mon, 15 Jan 2024 12:02:34 +0000 (13:02 +0100)] 
websockets: check for negative payload lengths

- in en- and decoding, check the websocket frame payload lengths for
  negative values (from curl_off_t) and error the operation in that case
- add test 2307 to verify

Closes #12707

18 months agodocs: mention env vars not used by schannel
Daniel Stenberg [Mon, 15 Jan 2024 22:39:55 +0000 (23:39 +0100)] 
docs: mention env vars not used by schannel

Ref: #12704

Co-authored-by: Jay Satiro <raysatiro@yahoo.com>
Closes #12711

18 months agotool_operate: make --remove-on-error only remove "real" files
Daniel Stenberg [Mon, 15 Jan 2024 15:49:20 +0000 (16:49 +0100)] 
tool_operate: make --remove-on-error only remove "real" files

Reported-by: Harry Sintonen
Assisted-by: Dan Fandrich
Closes #12710

18 months agourl: don't set default CA paths for Secure Transport backend
Jay Wu [Mon, 15 Jan 2024 04:24:45 +0000 (12:24 +0800)] 
url: don't set default CA paths for Secure Transport backend

As the default for this backend is the native CA store.

Closes #12704

18 months agoasyn-ares: with modern c-ares, use its default timeout
Lin Sun [Mon, 15 Jan 2024 06:58:09 +0000 (06:58 +0000)] 
asyn-ares: with modern c-ares, use its default timeout

Closes #12703

18 months agotool_operate: stop setting the file comment on Amiga
Daniel Stenberg [Mon, 15 Jan 2024 15:28:04 +0000 (16:28 +0100)] 
tool_operate: stop setting the file comment on Amiga

- the URL is capped at 80 cols, which ruins it if longer
- it does not strip off URL credentials
- it is done unconditonally, not on --xattr
- we don't have Amiga in the CI which makes fixing it blindly fragile

Someone who builds and tests on Amiga can add it back correctly in a
future if there is a desire.

Reported-by: Harry Sintonen
Closes #12709

18 months agortsp: deal with borked server responses
Stefan Eissing [Mon, 15 Jan 2024 10:33:13 +0000 (11:33 +0100)] 
rtsp: deal with borked server responses

- enforce a response body length of 0, if the
  response has no Content-lenght. This is according
  to the RTSP spec.
- excess bytes in a response body are forwarded to
  the client writers which will report and fail the
  transfer

Follow-up to d7b6ce6
Fixes #12701
Closes #12706

18 months agoversion: show only the libpsl version, not its dependencies
Daniel Stenberg [Sun, 14 Jan 2024 16:54:51 +0000 (17:54 +0100)] 
version: show only the libpsl version, not its dependencies

The libpsl version output otherwise also includes version number for its
dependencies, like IDN lib, but since libcurl does not use libpsl's IDN
functionality those components are not important.

Ref: https://github.com/curl/curl-for-win/issues/63
Closes #12700

18 months agocurl.h: CURLOPT_DNS_SERVERS is only available with c-ares
bch [Fri, 12 Jan 2024 21:54:40 +0000 (13:54 -0800)] 
curl.h: CURLOPT_DNS_SERVERS is only available with c-ares

Closes #12695

18 months agocmdline-opts/gen.pl: error on initital blank line
Daniel Stenberg [Sun, 14 Jan 2024 13:50:52 +0000 (14:50 +0100)] 
cmdline-opts/gen.pl: error on initital blank line

After the "---" separator, there should be no blank line and this script
now errors out if one is detected.

Ref: #12696
Closes #12698

18 months agocf-h1-proxy: no CURLOPT_USERAGENT in CONNECT with hyper
Daniel Stenberg [Sun, 14 Jan 2024 13:42:08 +0000 (14:42 +0100)] 
cf-h1-proxy: no CURLOPT_USERAGENT in CONNECT with hyper

Follow-up to 693cd1679361828a which was incomplete

Ref #12680
Closes #12697

18 months agocurl_multi_fdset.3: remove mention of null pointer support
Daniel Stenberg [Sat, 13 Jan 2024 22:18:20 +0000 (23:18 +0100)] 
curl_multi_fdset.3: remove mention of null pointer support

... since this funtion has not supported null pointer fd_set arguments since
at least 2006. (That's when I stopped my git blame journey)

Fixes #12691
Reported-by: sfan5 on github
Closes #12692

18 months agodocs/cmdline: remove unnecessary line breaks
Mark Huang [Sun, 14 Jan 2024 08:56:00 +0000 (16:56 +0800)] 
docs/cmdline: remove unnecessary line breaks

Closes #12696

18 months agotransfer: remove warning: Value stored to 'blen' is never read
Daniel Stenberg [Sat, 13 Jan 2024 22:42:19 +0000 (23:42 +0100)] 
transfer: remove warning: Value stored to 'blen' is never read

Detected by scan-build

Follow-up from 1cd2f0072f

Closes #12693

18 months agolib: replace readwrite with write_resp
Stefan Eissing [Fri, 1 Dec 2023 12:50:32 +0000 (13:50 +0100)] 
lib: replace readwrite with write_resp

This clarifies the handling of server responses by folding the code for
the complicated protocols into their protocol handlers. This concerns
mainly HTTP and its bastard sibling RTSP.

The terms "read" and "write" are often used without clear context if
they refer to the connect or the client/application side of a
transfer. This PR uses "read/write" for operations on the client side
and "send/receive" for the connection, e.g. server side. If this is
considered useful, we can revisit renaming of further methods in another
PR.

Curl's protocol handler `readwrite()` method been changed:

```diff
-  CURLcode (*readwrite)(struct Curl_easy *data, struct connectdata *conn,
-                        const char *buf, size_t blen,
-                        size_t *pconsumed, bool *readmore);
+  CURLcode (*write_resp)(struct Curl_easy *data, const char *buf, size_t blen,
+                         bool is_eos, bool *done);
```

The name was changed to clarify that this writes reponse data to the
client side. The parameter changes are:

* `conn` removed as it always operates on `data->conn`
* `pconsumed` removed as the method needs to handle all data on success
* `readmore` removed as no longer necessary
* `is_eos` as indicator that this is the last call for the transfer
  response (end-of-stream).
* `done` TRUE on return iff the transfer response is to be treated as
  finished

This change affects many files only because of updated comments in
handlers that provide no implementation. The real change is that the
HTTP protocol handlers now provide an implementation.

The HTTP protocol handlers `write_resp()` implementation will get passed
**all** raw data of a server response for the transfer. The HTTP/1.x
formatted status and headers, as well as the undecoded response
body. `Curl_http_write_resp_hds()` is used internally to parse the
response headers and pass them on. This method is public as the RTSP
protocol handler also uses it.

HTTP/1.1 "chunked" transport encoding is now part of the general
*content encoding* writer stack, just like other encodings. A new flag
`CLIENTWRITE_EOS` was added for the last client write. This allows
writers to verify that they are in a valid end state. The chunked
decoder will check if it indeed has seen the last chunk.

The general response handling in `transfer.c:466` happens in function
`readwrite_data()`. This mainly operates now like:

```
static CURLcode readwrite_data(data, ...)
{
  do {
    Curl_xfer_recv_resp(data, buf)
    ...
    Curl_xfer_write_resp(data, buf)
    ...
  } while(interested);
  ...
}
```

All the response data handling is implemented in
`Curl_xfer_write_resp()`. It calls the protocol handler's `write_resp()`
implementation if available, or does the default behaviour.

All raw response data needs to pass through this function. Which also
means that anyone in possession of such data may call
`Curl_xfer_write_resp()`.

Closes #12480

18 months agoRELEASE-NOTES: synced
Daniel Stenberg [Sat, 13 Jan 2024 16:02:45 +0000 (17:02 +0100)] 
RELEASE-NOTES: synced

18 months agoTODO: TFTP doesn't convert LF to CRLF for mode=netascii
Daniel Stenberg [Sat, 13 Jan 2024 09:56:47 +0000 (10:56 +0100)] 
TODO: TFTP doesn't convert LF to CRLF for mode=netascii

Closes #12655
Closes #12690

18 months agogen: do italics/bold for a range of letters, not just single word
Daniel Stenberg [Sat, 13 Jan 2024 09:49:20 +0000 (10:49 +0100)] 
gen: do italics/bold for a range of letters, not just single word

Previously it would match only on a sequence of non-space, which made it
miss to highlight for example "public suffix list".

Updated the recent cookie.d edit from 5da57193b732 to use bold instead
of italics.

Closes #12689

18 months agodocs: describe and highlight super cookies
Daniel Stenberg [Fri, 12 Jan 2024 15:50:44 +0000 (16:50 +0100)] 
docs: describe and highlight super cookies

Reported-by: Yadhu Krishna M
Closes #12687

18 months agoconfigure: when enabling QUIC, check that TLS supports QUIC
Daniel Stenberg [Thu, 11 Jan 2024 13:11:19 +0000 (14:11 +0100)] 
configure: when enabling QUIC, check that TLS supports QUIC

Most importantly perhaps is when using OpenSSL that the used
build/flavor has the QUIC API: the vanilla OpenSSL does not, only
BoringSSL, libressl, AWS-LC and quictls do.

Ref: https://github.com/curl/curl/commit/5d044ad9480a9f556f4b6a252d7533b1ba7fe57e#r136780413

Closes #12683

18 months agovquic: extract TLS setup into own source
Stefan Eissing [Tue, 9 Jan 2024 12:21:02 +0000 (13:21 +0100)] 
vquic: extract TLS setup into own source

- separate ngtcp2 specific parts out
- provide callback during init to allow ngtcp2 to apply its defaults

Closes #12678

18 months agomulti: remove total timer reset in file_do() while fetching file://
Sergey Markelov [Thu, 11 Jan 2024 00:23:00 +0000 (17:23 -0700)] 
multi: remove total timer reset in file_do() while fetching file://

The total timer is properly reset in MSTATE_INIT.  MSTATE_CONNECT starts
with resetting the timer that is a start point for further multi states.
If file://, MSTATE_DO calls file_do() that should not reset the total
timer.  Otherwise, the total time is always less than the pre-transfer
and the start transfer times.

Closes #12682

18 months agohttp_proxy: a blank CURLOPT_USERAGENT should not be used in CONNECT
Daniel Stenberg [Wed, 10 Jan 2024 22:20:09 +0000 (23:20 +0100)] 
http_proxy: a blank CURLOPT_USERAGENT should not be used in CONNECT

Extended test 80 to verify this.

Reported-by: Stefan Eissing
Fixes #12680
Closes #12681

18 months agosectransp: do verify_cert without memdup for blobs
Daniel Stenberg [Wed, 10 Jan 2024 14:13:13 +0000 (15:13 +0100)] 
sectransp: do verify_cert without memdup for blobs

Since the information is then already stored in memory, this can avoid
an extra set of malloc + free calls.

Closes #12679

18 months agohsts: remove assert for zero length domain
Daniel Stenberg [Wed, 10 Jan 2024 08:46:19 +0000 (09:46 +0100)] 
hsts: remove assert for zero length domain

A zero length domain can happen if the HSTS parser is given invalid
input data which is not unheard of and is done by the fuzzer.

Follow-up from cfe7902111ae547873

Bug: https://bugs.chromium.org/p/oss-fuzz/issues/detail?id=65661

Closes #12676

18 months agoheaders: make sure the trailing newline is not stored
Daniel Stenberg [Wed, 10 Jan 2024 08:22:59 +0000 (09:22 +0100)] 
headers: make sure the trailing newline is not stored

extended test1940 to verify blank header fields too

Bug: https://curl.se/mail/lib-2024-01/0019.html
Reported-by: Dmitry Karpov
Closes #12675

18 months agocurl_easy_header.3: tiny language fix
Daniel Stenberg [Tue, 9 Jan 2024 23:42:17 +0000 (00:42 +0100)] 
curl_easy_header.3: tiny language fix

Closes #12672

18 months agoexamples/range.c: add
Daniel Stenberg [Tue, 9 Jan 2024 22:47:01 +0000 (23:47 +0100)] 
examples/range.c: add

Closes #12671

18 months agoexamples/netrc.c: add
Daniel Stenberg [Tue, 9 Jan 2024 22:44:24 +0000 (23:44 +0100)] 
examples/netrc.c: add

Closes #12671

18 months agoexamples/ipv6.c: new example showing IPv6-only internet transfer
Daniel Stenberg [Tue, 9 Jan 2024 22:42:17 +0000 (23:42 +0100)] 
examples/ipv6.c: new example showing IPv6-only internet transfer

Closes #12671

18 months agoexamples/address-scope.c: renamed from ipv6.c
Daniel Stenberg [Tue, 9 Jan 2024 22:42:01 +0000 (23:42 +0100)] 
examples/address-scope.c: renamed from ipv6.c

It shows address scope use really

Closes #12671

18 months agomulti: pollset adjust, init with FIRSTSOCKET during connect
Stefan Eissing [Tue, 9 Jan 2024 08:29:34 +0000 (09:29 +0100)] 
multi: pollset adjust, init with FIRSTSOCKET during connect

- `conn->sockfd` is set by `Curl_setup_transfer()`, but that
  is called *after* the connection has been established
- use `conn->sock[FIRSTSOCKET]` instead

Follow-up to a0f94800d507de
Closes #12664

18 months agoWEBSOCKET.md: remove dead link
Daniel Stenberg [Tue, 9 Jan 2024 10:12:02 +0000 (11:12 +0100)] 
WEBSOCKET.md: remove dead link

18 months agoCI: spellcheck/appveyor: invoke configure --without-libpsl
Daniel Stenberg [Tue, 9 Jan 2024 10:09:30 +0000 (11:09 +0100)] 
CI: spellcheck/appveyor: invoke configure --without-libpsl

Follow-up to 2998874bb61ac6

18 months agocmdline/docs/*.d: switch to using ## instead of .IP
Daniel Stenberg [Tue, 9 Jan 2024 09:36:14 +0000 (10:36 +0100)] 
cmdline/docs/*.d: switch to using ## instead of .IP

To make the editing easier. To write and to read.

Closes #12667

18 months agogen.pl: support ## for doing .IP in table-like lists
Daniel Stenberg [Tue, 9 Jan 2024 09:20:48 +0000 (10:20 +0100)] 
gen.pl: support ## for doing .IP in table-like lists

Warn on use of .RS/.IP/.RE

Closes #12667

18 months agocookie.d: Document use of empty string to enable cookie engine
Jay Satiro [Sun, 7 Jan 2024 05:25:13 +0000 (00:25 -0500)] 
cookie.d: Document use of empty string to enable cookie engine

- Explain that --cookie "" can be used to enable the cookie engine
  without reading any initial cookies.

As is documented in CURLOPT_COOKIEFILE.

Ref: https://curl.se/libcurl/c/CURLOPT_COOKIEFILE.html

Bug: https://github.com/curl/curl/issues/12643#issuecomment-1879844420
Reported-by: janko-js@users.noreply.github.com
Closes https://github.com/curl/curl/pull/12646

18 months agosetopt: use memdup0 when cloning COPYPOSTFIELDS
Daniel Stenberg [Sun, 7 Jan 2024 15:11:01 +0000 (16:11 +0100)] 
setopt: use memdup0 when cloning COPYPOSTFIELDS

Closes #12651

18 months agotelnet: use dynbuf instad of malloc for escape buffer
Daniel Stenberg [Sun, 7 Jan 2024 15:28:15 +0000 (16:28 +0100)] 
telnet: use dynbuf instad of malloc for escape buffer

Previously, send_telnet_data() would malloc + free a buffer every time
for escaping IAC codes. Now, it reuses a dynbuf for this purpose.

Closes #12652

18 months agoCI: install libpsl or configure --without-libpsl in builds
Daniel Stenberg [Mon, 8 Jan 2024 15:49:21 +0000 (16:49 +0100)] 
CI: install libpsl or configure --without-libpsl in builds

As a follow-up to the stricted libpsl check in configure

18 months agoconfigure: make libpsl detection failure cause error
Daniel Stenberg [Mon, 8 Jan 2024 15:28:50 +0000 (16:28 +0100)] 
configure: make libpsl detection failure cause error

To force users to explictily disable it if they really don't want it
used and make it harder to accidentally miss it.

--without-libpsl is the option to use if PSL is not wanted.

Closes #12661

18 months agoRELEASE-NOTES: synced
Daniel Stenberg [Mon, 8 Jan 2024 22:00:21 +0000 (23:00 +0100)] 
RELEASE-NOTES: synced

18 months agopop3: replace calloc + memcpy with memdup0
Daniel Stenberg [Sun, 7 Jan 2024 15:06:32 +0000 (16:06 +0100)] 
pop3: replace calloc + memcpy with memdup0

... and make sure to return error on out of memory.

Closes #12650