]> git.ipfire.org Git - thirdparty/curl.git/log
thirdparty/curl.git
2 months agoRELEASE-NOTES: synced
Daniel Stenberg [Mon, 11 Mar 2024 06:30:01 +0000 (07:30 +0100)] 
RELEASE-NOTES: synced

3 months agolib: keep conn IP information together
Stefan Eissing [Fri, 8 Mar 2024 09:45:14 +0000 (10:45 +0100)] 
lib: keep conn IP information together

new struct ip_quadruple for holding local/remote addr+port

- used in data->info and conn and cf-socket.c
- copy back and forth complete struct
- add 'secondary' to conn
- use secondary in reporting success for ftp 2nd connection

Reported-by: DasKutti on github
Fixes #13084
Closes #13090

3 months agoscripts/managen: the new name and home for the manpage generator
Daniel Stenberg [Fri, 8 Mar 2024 10:22:12 +0000 (11:22 +0100)] 
scripts/managen: the new name and home for the manpage generator

It was previously docs/cmdline-opts/gen.pl

Closes #13089

3 months agoVULN-DISCLOSURE-POLICY.md: update detail about CVE requests
Daniel Stenberg [Fri, 8 Mar 2024 10:09:48 +0000 (11:09 +0100)] 
VULN-DISCLOSURE-POLICY.md: update detail about CVE requests

curl is a CNA now

Closes #13088

3 months agolib: client reader polish
Stefan Eissing [Thu, 7 Mar 2024 10:05:53 +0000 (11:05 +0100)] 
lib: client reader polish

- seek_func/seek_client, use transfer values only
    - remove copies held in `struct connectdata`, use only
      ever `data->set.seek_func`
    - resolves possible issues in multiuse connections
    - new mime post reader eliminates need to ever overwriting this

- websockets, remove empty Curl_ws_done() function

Closes #13079

3 months agolib1598: fix `CURLOPT_POSTFIELDSIZE` usage
Marcel Raad [Fri, 8 Mar 2024 07:45:10 +0000 (08:45 +0100)] 
lib1598: fix `CURLOPT_POSTFIELDSIZE` usage

It requires a `long` argument.

Closes https://github.com/curl/curl/pull/13085

3 months agodocs/cmdline-opts: drop the curl.1 from the dist tarball
Daniel Stenberg [Thu, 7 Mar 2024 22:25:07 +0000 (23:25 +0100)] 
docs/cmdline-opts: drop the curl.1 from the dist tarball

Since it is no longer needed for building tool_hugehelp.c and all the
docs is available in readable markdown format in the tarball, the peeps
that don't want to build the manpage still do good.

Removing it also fixes the complexity of out-of-tree builds when the
curl.1 exists in the source tree.

3 months agotest1140/1173: extend wildcards to find curl.1
Daniel Stenberg [Thu, 7 Mar 2024 21:30:50 +0000 (22:30 +0100)] 
test1140/1173: extend wildcards to find curl.1

... in its new build path.

Also update the test scripts to be more precise in error messages to
help us understand CI errors better.

Follow-up to f03c85635f35269f1
Ref: #13029
Closes #13083

3 months agohttp2: minor tweaks to optimize two struct sizes
Daniel Stenberg [Thu, 7 Mar 2024 15:41:06 +0000 (16:41 +0100)] 
http2: minor tweaks to optimize two struct sizes

- use BIT() instead of bool
- place the struct fields in (roughly) size order

Closes #13082

3 months agobuildconf.bat: remove outdated groff/nroff use
Daniel Stenberg [Thu, 7 Mar 2024 09:33:49 +0000 (10:33 +0100)] 
buildconf.bat: remove outdated groff/nroff use

- don't try to generate the real hugehelp file, because it requires
  curl.txt which needs a build
- don't attempt to do anything in a c-ares subdirectory

Follow-up to f03c85635f35269
Closes #13078

3 months agohttp2: memory errors in the push callbacks are fatal
Daniel Stenberg [Thu, 7 Mar 2024 15:02:13 +0000 (16:02 +0100)] 
http2: memory errors in the push callbacks are fatal

Use the correct nghttp2 error code accordingly.

Closes #13081

3 months agomkhelp: rename variable to fix compiler warnings
Viktor Szakats [Thu, 7 Mar 2024 09:25:39 +0000 (09:25 +0000)] 
mkhelp: rename variable to fix compiler warnings

```
src\tool_operate.c(541,33): warning C4459: declaration of 'm' hides global declaration [_bld\src\curl.vcxproj]
  _bld\src\tool_hugehelp.c(8,27):
  see declaration of 'm'
src\tool_paramhlp.c(307,14): warning C4459: declaration of 'm' hides global declaration [_bld\src\curl.vcxproj]
src\tool_progress.c(118,16): warning C4459: declaration of 'm' hides global declaration [_bld\src\curl.vcxproj]
src\tool_writeout.c(288,31): warning C4459: declaration of 'm' hides global declaration [_bld\src\curl.vcxproj]
```
Ref: https://ci.appveyor.com/project/curlorg/curl/builds/49348159/job/51ee75cd2n0wj6lc#L614

Reviewed-by: Daniel Stenberg
Closes #13077

3 months agoKNOWN_BUGS: POP3 issue when reading small chunks
Daniel Stenberg [Thu, 7 Mar 2024 16:40:29 +0000 (17:40 +0100)] 
KNOWN_BUGS: POP3 issue when reading small chunks

Closes #12063

3 months agoRELEASE-NOTES: synced
Daniel Stenberg [Thu, 7 Mar 2024 15:50:59 +0000 (16:50 +0100)] 
RELEASE-NOTES: synced

3 months agoasyn-ares: fix data race warning
Robert Moreton [Mon, 26 Feb 2024 23:16:21 +0000 (18:16 -0500)] 
asyn-ares: fix data race warning

- Store the c-ares version during global init.

Prior to this change several threads could write the same data to a
static int variable at the same time. Though in practice it's not a
problem ThreadSanitizer may warn.

Reported-by: Nikita Taranov
Assisted-by: Jay Satiro
Fixes #13065
Closes #13000

3 months agohyper: implement unpausing via client reader
Stefan Eissing [Thu, 7 Mar 2024 09:08:35 +0000 (10:08 +0100)] 
hyper: implement unpausing via client reader

Just a tidy up to contain 'ifdef' pollution of common
code parts with implementation specifics.

- remove the ifdef hyper unpausing in easy.c
- add hyper client reader for CURL_CR_PROTOCOL phase
  that implements the unpause method for calling
  the hyper waker if it is set

Closes #13075

3 months agongtcp2: no recvbuf for stream
Stefan Eissing [Thu, 7 Mar 2024 08:23:11 +0000 (09:23 +0100)] 
ngtcp2: no recvbuf for stream

- write response data directly to the transfer via
 `Curl_xfer_write_resp()` like we do in HTTP/2.

Closes #13073

3 months agodocs/cmdline-opts/.gitignore: ignore curl.txt
Stefan Eissing [Thu, 7 Mar 2024 09:18:33 +0000 (10:18 +0100)] 
docs/cmdline-opts/.gitignore: ignore curl.txt

Closes #13076

3 months agosha512_256: add support for GnuTLS and OpenSSL
Evgeny Grin (Karlson2k) [Wed, 6 Mar 2024 23:42:04 +0000 (00:42 +0100)] 
sha512_256: add support for GnuTLS and OpenSSL

This is a follow-up for PR #12897.

Add support for SHA-512/256 digest calculation by TLS backends.
Currently only OpenSSL and GnuTLS (actually, nettle) support
SHA-512/256.

Closes #13070

3 months agodigest: add check for hashing error
Evgeny Grin [Thu, 7 Mar 2024 00:35:31 +0000 (01:35 +0100)] 
digest: add check for hashing error

Closes #13072

3 months agocmake: enable `ENABLE_CURL_MANUAL` by default
Viktor Szakats [Wed, 6 Mar 2024 22:24:56 +0000 (22:24 +0000)] 
cmake: enable `ENABLE_CURL_MANUAL` by default

Meaning `curl.1` and `src/tool_hugehelp.c` are built by default,
and `--manual` in curl tool is also enabled by default.

This syncs behaviour with autotools.

For a reproducible `curl.1`, `SOURCE_DATE_EPOCH` needs to be set
to a consistent date, e.g. the timestamp of `CHANGES`.

A pre-built manual (e.g. the one distributed in the official source
tarball) will be ignored and rebuilt after this patch, unless
explicitly disabling this option.

Fixes #13028
Closes #13069

3 months agohttp2: push headers better cleanup
Stefan Eissing [Wed, 6 Mar 2024 08:36:08 +0000 (09:36 +0100)] 
http2: push headers better cleanup

- provide common cleanup method for push headers

Closes #13054

3 months agoGIT-INFO: convert to markdown
Daniel Stenberg [Thu, 7 Mar 2024 08:28:06 +0000 (09:28 +0100)] 
GIT-INFO: convert to markdown

Closes #13074

3 months agocmake: fix libcurl.pc and curl-config library specifications
Richard Levitte [Tue, 13 Feb 2024 05:45:57 +0000 (06:45 +0100)] 
cmake: fix libcurl.pc and curl-config library specifications

Letting CMake figure out where libraries are located gives you full
paths. When generating libcurl.pc and curl-config, getting libraries as
full paths is unusual when one expects to get a list of -l<libname>.

To meet expectations, an effort is made to convert the full paths into
-l<libname>, possibly with -L<libdir> before it.

Fixes #6169
Fixes #12748
Closes #12930

3 months agotest463: HTTP with -d @file with file containing CR, LF and null byte
Daniel Stenberg [Wed, 6 Mar 2024 21:44:41 +0000 (22:44 +0100)] 
test463: HTTP with -d @file with file containing CR, LF and null byte

3 months agoparamhlp: fix CRLF-stripping files with "-d @file"
Daniel Stenberg [Wed, 6 Mar 2024 14:39:09 +0000 (15:39 +0100)] 
paramhlp: fix CRLF-stripping files with "-d @file"

All CR and LF bytes should be stripped, as documented, and all other
bytes are inluded in the data. Starting now, it also excludes null bytes
as they would otherwise also cut the data short.

Reported-by: Simon K
Fixes #13063
Closes #13064

3 months agocmake: fix `CURL_WINDOWS_SSPI=ON` with Schannel disabled
Viktor Szakats [Wed, 6 Mar 2024 11:43:40 +0000 (11:43 +0000)] 
cmake: fix `CURL_WINDOWS_SSPI=ON` with Schannel disabled

Prior to this change `CURL_WINDOWS_SSPI` was accidentally forced `OFF`
when building without the Schannel TLS backend.

This in turn may have caused Kerberos, SPNEGO and SSPI features
disappearing even with `CURL_WINDOWS_SSPI=ON` set.

This patch fixes it by using the `CURL_USE_SCHANNEL` setting as a
default for `CURL_WINDOWS_SSPI`, but allowing a manual override.

Also update the option text to better tell its purpose.

Thanks-to: Andreas Loew
Reviewed-by: Daniel Stenberg
Ref: #13056
Closes #13061

3 months agoKNOWN_BUGS: FTPS server compatibility on Windows with Schannel
Jay Satiro [Sun, 3 Mar 2024 07:15:24 +0000 (02:15 -0500)] 
KNOWN_BUGS: FTPS server compatibility on Windows with Schannel

- Remove "2.12 FTPS with Schannel times out file list operation"

- Remove "7.12 FTPS directory listing hangs on Windows with Schannel"

- Add "7.12 FTPS server compatibility on Windows with Schannel"

This change adds a more generic bug description that explains FTPS with
the latest curl and Schannel is not widely used and may have more bugs
than other TLS backends.

The two removed FTPS Schannel bugs can't be reproduced any longer and
were likely fixed by 24d6c288.

Ref: https://github.com/curl/curl/issues/5284
Ref: https://github.com/curl/curl/issues/9161
Ref: https://github.com/curl/curl/issues/12894

Closes https://github.com/curl/curl/pull/13032

3 months agotrace-config.md: remove the mutexed options list
Jay Satiro [Sat, 2 Mar 2024 22:44:33 +0000 (17:44 -0500)] 
trace-config.md: remove the mutexed options list

- Remove the rendered manpage message that says:
  "[--trace-config] is mutually exclusive to --trace and -v, --verbose".

Actually it can be used with either of those options, which are mutually
exclusive to each other but not to --trace-config.

Ref: https://curl.se/docs/manpage.html#--trace-config

Closes https://github.com/curl/curl/pull/13031

3 months agomkhelp: simplify the generated hugehelp program
Daniel Stenberg [Tue, 5 Mar 2024 16:32:00 +0000 (17:32 +0100)] 
mkhelp: simplify the generated hugehelp program

Use a plain array and puts() every line, also allows us to provide the
strings without ending newlines.

- merge blank lines into the next one as a prefixed newline.
- turn eight consecutive spaces into a tab (since they can only be on the
  left side of text)
- the newly generated tool_hugehelp is 3K lines shorter and 50K smaller
- modifies the top logo layout a little by reducing the indent

Closes #13047

3 months agodocs: ascii version of manpage without nroff
Daniel Stenberg [Mon, 4 Mar 2024 16:22:17 +0000 (17:22 +0100)] 
docs: ascii version of manpage without nroff

Create ASCII version of manpage without nroff

 - build src/tool_hugegelp.c from the ascii manpage
 - move the the manpage and the ascii version build to docs/cmdline-opts
 - remove all use of nroff from the build process
 - should make the build entirely reproducible (by avoiding nroff)

 - partly reverts 2620aa9 to build libcurl option man pages one by one
   in cmake because the appveyor builds got all crazy until I did

The ASCII version of the manpage

 - is built with gen.pl, just like the manpage is
 - has a right-justified column making the appearance similar to the previous
   version
 - uses a 4-space indent per level (instead of the old version's 7)
 - does not do hyphenation of words (which nroff does)

History

  We first made the curl build use nroff for building the hugehelp file in
  December 1998, for curl 5.2.

Closes #13047

3 months agolib: add `void *ctx` to reader/writer instances
Stefan Eissing [Wed, 6 Mar 2024 08:52:43 +0000 (09:52 +0100)] 
lib: add `void *ctx` to reader/writer instances

- `struct Curl_cwriter` and `struct Curl_creader` now carry a
  `void *ctx` member that points to the instance as allocated.
- using `r->ctx` and `w->ctx` as pointer to the instance specific
  struct that has been allocated

Reported-by: Rudi Heitbaum
Fixes #13035
Closes #13059

3 months agohttp: fix dead code in setting post client reader
Stefan Eissing [Wed, 6 Mar 2024 11:09:05 +0000 (12:09 +0100)] 
http: fix dead code in setting post client reader

- postsize was always 0, thus the check's else never happened
  after the mime client reader was introduced

Follow-up to 0ba47146f7ff3d
Closes #13060

3 months agohttp2: fix push discard
Stefan Eissing [Wed, 6 Mar 2024 09:15:15 +0000 (10:15 +0100)] 
http2: fix push discard

- fix logic in discarding a failed pushed stream so that
  stream context is properly cleaned up

Closes #13055

3 months agotransfer.c: break receive loop in speed limited transfers
Stefan Eissing [Tue, 5 Mar 2024 10:08:55 +0000 (11:08 +0100)] 
transfer.c: break receive loop in speed limited transfers

- the change breaks looping in transfer.c receive for transfers that are
  speed limited on having gotten *some* bytes.
- the overall speed limit timing is done in multi.c

Reported-by: Dmitry Karpov
Bug: https://curl.se/mail/lib-2024-03/0001.html
Closes #13050

3 months agomime: add client reader
Stefan Eissing [Thu, 29 Feb 2024 09:12:39 +0000 (10:12 +0100)] 
mime: add client reader

Add `mime` client reader. Encapsulates reading from mime parts, getting
their length, rewinding and unpausing.

- remove special mime handling from sendf.c and easy.c
- add general "unpause" method to client readers
- use new reader in http/imap/smtp
- make some mime functions static that are now only used internally

In addition:
- remove flag 'forbidchunk' as no longer needed

Closes #13039

3 months agoRELEASE-NOTES: synced
Daniel Stenberg [Tue, 5 Mar 2024 16:15:27 +0000 (17:15 +0100)] 
RELEASE-NOTES: synced

3 months agoTODO: remove "build HTTP/3 with OpenSSL and nghttp3 using cmake"
Daniel Stenberg [Tue, 5 Mar 2024 16:09:46 +0000 (17:09 +0100)] 
TODO: remove "build HTTP/3 with OpenSSL and nghttp3 using cmake"

Follow-up to 8e741644a229c37

3 months agocmake: add USE_OPENSSL_QUIC support
Tal Regev [Sun, 3 Mar 2024 19:57:03 +0000 (21:57 +0200)] 
cmake: add USE_OPENSSL_QUIC support

Closes #13034

3 months agoTIMER_STARTTRANSFER: set the same for everyone
Stefan Eissing [Tue, 5 Mar 2024 10:48:16 +0000 (11:48 +0100)] 
TIMER_STARTTRANSFER: set the same for everyone

- set TIMER_STARTTRANSFER on seeing the first response bytes
  in the download client writer, not coming from a CONNECT
- initialized the timer the same way for all protocols
- remove explicit setting of TIMER_STARTTRANSFER in file.c
  and c-hyper.c

Closes #13052

3 months agohttp: better error message for HTTP/1.x response without status line
Michael Kaufmann [Mon, 4 Mar 2024 17:15:08 +0000 (18:15 +0100)] 
http: better error message for HTTP/1.x response without status line

If a response without a status line is received, and the connection is
known to use HTTP/1.x (not HTTP/0.9), report the error "Invalid status
line" instead of "Received HTTP/0.9 when not allowed".

Closes #13045

3 months agoKNOWN_BUGS: fix typo
Viktor Szakats [Tue, 5 Mar 2024 10:52:04 +0000 (10:52 +0000)] 
KNOWN_BUGS: fix typo

Reviewed-by: Daniel Stenberg
Closes #13051

3 months agosmpt: fix starttls
Sebastian Neubauer [Tue, 5 Mar 2024 01:11:46 +0000 (02:11 +0100)] 
smpt: fix starttls

In cases where the connection was fast, curl sometimes failed to open a
connection. This fixes a regression of c2d973627bab12abc5486a3f3.

The regression triggered in these steps:

1. Create an smtp connection
2. Use STARTTLS
3. Receive the response
4. We are inside the loop in `smtp_statemachine`, calling
   `smtp_state_starttls_resp`
5. In the good flow, we exit the loop, re-enter `smtp_statemachine` and
   run `smtp_perform_upgrade_tls` at the start of the function.

   In the bad flow, we stay in the while loop, calling
   `Curl_pp_readresp`, which reads part of the TLS handshake and things
   go wrong.

The reason is that `Curl_pp_moredata` changed behavior and always
returns `true`, so we stay in the loop in `smtp_statemachine`. With a
slow connection `Curl_pp_readresp` cannot read new data and returns
`CURL_AGAIN`, so we leave the loop and re-enter `smtp_statemachine`.

With a fast connection, `Curl_pp_readresp` reads new data from the tcp
connection, which is part of the TLS handshake.

The fix is in `Curl_pp_moredata`, which needs to take the final line
into account and return `false` if only the final line is stored.

Closes #13048

3 months agolib: enhance client reader resume + rewind
Stefan Eissing [Thu, 29 Feb 2024 09:12:39 +0000 (10:12 +0100)] 
lib: enhance client reader resume + rewind

- update client reader documentation
- client reader, add rewind capabilities
    - tell creader to rewind on next start
    - Curl_client_reset() will keep reader for future rewind if requested
    - add Curl_client_cleanup() for freeing all resources independent of
      rewinds
    - add Curl_client_start() to trigger rewinds
    - move rewind code from multi.c to sendf.c and make part of
      "cr-in"'s implementation
- http, move the "resume_from" handling into the client readers
    - the setup of a HTTP request is reshuffled to follow:
      * determine method, target, auth negotiation
      * install the client reader(s) for the request, including crlf
        conversions and "chunked" encoding
      * apply ranges to client reader
      * concat request headers, upgrades, cookies, etc.
      * complete request by determining Content-Length of installed
        readers in combination with method
      * send
    - add methods for client readers to
      * return the overall length they will generate (or -1 when unknown)
      * return the amount of data on the CLIENT level, so that
        expect-100 can decide if it want to apply itself
      * set a "resume_from" offset or fail if unsupported
    - struct HTTP has become largely empty now
- rename `Client_reader_*` to `Curl_creader_*`

Closes #13026

3 months agoopenssl-quic: fix BIO leak and Windows warning
Viktor Szakats [Mon, 4 Mar 2024 17:07:40 +0000 (17:07 +0000)] 
openssl-quic: fix BIO leak and Windows warning

Caused by an accidentally duplicated line in
d6825df334def106f735ce7e0c1a2ea87bddffb0.

```
.../lib/vquic/curl_osslq.c:1095:30: warning: implicit conversion loses integer precision: 'curl_socket_t' (aka 'unsigned long long') to 'int' [-Wshorten-64-to-32]
 1095 |   bio = BIO_new_dgram(ctx->q.sockfd, BIO_NOCLOSE);
      |         ~~~~~~~~~~~~~ ~~~~~~~^~~~~~
1 warning and 2 errors generated.
```

Reviewed-by: Stefan Eissing
Closes #13043

3 months agoopenssl-quic: fix unity build, casing, indentation
Viktor Szakats [Mon, 4 Mar 2024 17:12:42 +0000 (17:12 +0000)] 
openssl-quic: fix unity build, casing, indentation

- rename static functions to avoid duplicate symbols in unity mode.
- windows -> Windows/window in error message and comment.
- fix indentation.

Reviewed-by: Stefan Eissing
Closes #13044

3 months agogen.pl: make the "manpageification" faster
Daniel Stenberg [Mon, 4 Mar 2024 14:16:11 +0000 (15:16 +0100)] 
gen.pl: make the "manpageification" faster

The function that replaces occurances of "--longoption" with "-Z,
--longoption" etc with the proper highlight applied, no longer loops
over the options.

Closes #13041

3 months agoCONTRIBUTE: update the section on documentation format
Daniel Stenberg [Mon, 4 Mar 2024 22:04:05 +0000 (23:04 +0100)] 
CONTRIBUTE: update the section on documentation format

... since most of it is markdown now.

Closes #13046

3 months agosmtp: free a temp resource
Daniel Stenberg [Mon, 4 Mar 2024 09:07:07 +0000 (10:07 +0100)] 
smtp: free a temp resource

The returned address needs to be freed.

Follow-up to e3905de8196d67b89df1602feb84c1f993211b20
Spotted by Coverity

Closes #13038

3 months ago_VARIABLES.md: improve the description
Daniel Stenberg [Mon, 4 Mar 2024 13:26:14 +0000 (14:26 +0100)] 
_VARIABLES.md: improve the description

Closes #13040

3 months agobuild(deps): bump fsfe/reuse-action from 2 to 3
dependabot[bot] [Mon, 4 Mar 2024 14:31:36 +0000 (14:31 +0000)] 
build(deps): bump fsfe/reuse-action from 2 to 3

Bumps [fsfe/reuse-action](https://github.com/fsfe/reuse-action) from 2 to 3.
- [Release notes](https://github.com/fsfe/reuse-action/releases)
- [Commits](https://github.com/fsfe/reuse-action/compare/v2...v3)

---
updated-dependencies:
- dependency-name: fsfe/reuse-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
3 months agopytest: adapt to API change
Stefan Eissing [Mon, 4 Mar 2024 09:05:52 +0000 (10:05 +0100)] 
pytest: adapt to API change

- pytest has changed the signature of the hook pytest_report_header()
  for some obscure reason and that change landed in our CI now

- remove the changed param that we never used anyway

Closes #13037

3 months agocookie: if psl fails, reject the cookie
Daniel Stenberg [Sun, 3 Mar 2024 17:41:52 +0000 (18:41 +0100)] 
cookie: if psl fails, reject the cookie

A libpsl install without data and no built-in database is now considered
bad enough to reject all cookies since they cannot be checked. It is
somewhat of a user error, but still.

Reported-by: Dan Fandrich
Closes #13033

3 months agolib: further send/upload handling polish
Stefan Eissing [Wed, 28 Feb 2024 13:51:53 +0000 (14:51 +0100)] 
lib: further send/upload handling polish

- Move all the "upload_done" handling to request.c

  - add possibility to abort sending of a request
  - add `Curl_req_done_sending()` for checks
  - transfer.c: readwrite_upload() now clean

- removing data->state.ulbuf and data->req.upload_fromhere

  - as well as data->req.upload_present
  - set data->req.upload_done on having read all from
    the client and completely flushed the send buffer

- tftp, remove setting of data->req.upload_fromhere

  - serves no purpose as `upload_present` is not set
    and the data itself is directly `sendto()` anyway

- smtp, make upload EOB conversion a client reader
- xfer_ulbuf addition

  - add xfer_ulbuf for borrowing, similar to xfer_buf
  - use in file upload
  - use in c-hyper body sending

- h1-proxy, remove init of data->state.uilbuf that is never used
- smb, add own send_buf instead of using data->state.ulbuf

Closes #13010

3 months agoRELEASE-NOTES: synced
Daniel Stenberg [Mon, 4 Mar 2024 07:19:12 +0000 (08:19 +0100)] 
RELEASE-NOTES: synced

3 months agorustls: fix two warnings related to number types
kpcyrd [Thu, 29 Feb 2024 15:01:03 +0000 (16:01 +0100)] 
rustls: fix two warnings related to number types

Reported-by: Gisle Vanem
Follow-up to #12989
Closes #13017

3 months agobufq: writing into a softlimit queue cannot be partial
Stefan Eissing [Fri, 1 Mar 2024 08:12:50 +0000 (09:12 +0100)] 
bufq: writing into a softlimit queue cannot be partial

- when unable to obtain a new chunk on a softlimit bufq,
  this is an allocation error and needs to be reported as
  such.
- writes into a soflimit bufq never must be partial success

Reported-by: Dan Fandrich
Fixes #13020
Closes #13023

3 months agoconfigure: Don't build shell completions when disabled
Dan Fandrich [Sat, 2 Mar 2024 02:27:35 +0000 (18:27 -0800)] 
configure: Don't build shell completions when disabled

With the recent changes to completion file building, the files were
built always and only installation was selectively disabled.  Now, when
they are disabled they aren't even built, avoiding a build-time error in
environments where it's not possible to run the curl binary that was
just created (e.g. if library paths were not set up correctly).

Follow-up to 0f7aba83c

Reported-by: av223119 on github
Fixes #13027
Closes #13030

3 months agocmdline-opts/_EXITCODES: sync with libcurl-errors
Jay Satiro [Thu, 29 Feb 2024 06:36:31 +0000 (01:36 -0500)] 
cmdline-opts/_EXITCODES: sync with libcurl-errors

- Add error code 100 (CURLE_TOO_LARGE) to the list of error codes that
  can be returned by the curl tool.

Closes https://github.com/curl/curl/pull/13015

3 months agohyper: disable test1598 due to lack of trailer support
Stefan Eissing [Thu, 29 Feb 2024 10:31:55 +0000 (11:31 +0100)] 
hyper: disable test1598 due to lack of trailer support

Follow-up to 50838095

Closes #13016

3 months agoftp: Mark a const buffer as const
Dan Fandrich [Fri, 1 Mar 2024 09:07:02 +0000 (01:07 -0800)] 
ftp: Mark a const buffer as const

3 months agoappveyor: Properly skip if only CircleCI is changed
Dan Fandrich [Fri, 1 Mar 2024 09:06:28 +0000 (01:06 -0800)] 
appveyor: Properly skip if only CircleCI is changed

3 months agodocs: Update minimal binary size in INSTALL.md
Dan Fandrich [Fri, 10 Nov 2023 09:41:13 +0000 (01:41 -0800)] 
docs: Update minimal binary size in INSTALL.md

Include more options to reduce binary size.

3 months agoconfigure: Don't make shell completions without perl
Dan Fandrich [Fri, 1 Mar 2024 07:38:22 +0000 (23:38 -0800)] 
configure: Don't make shell completions without perl

The code that attempted to skip building the shell completions didn't
work properly and tried to build them even if perl wasn't available.
This step, as well as the install step, is now properly skipped without
perl.

Follow-up to 89733e2dd

Closes #13022

3 months agomisc: Fix typos in docs and lib
RainRat [Thu, 29 Feb 2024 19:55:28 +0000 (11:55 -0800)] 
misc: Fix typos in docs and lib

This fixes miscellaneous typos and duplicated words in the docs, lib
and test comments and a few user facing errorstrings.

Author: RainRat on Github
Reviewed-by: Daniel Gustafsson <daniel@yesql.se>
Reviewed-by: Dan Fandrich <dan@coneharvesters.com>
Closes: #13019
3 months agoconfigure: build & install shell completions when enabled
Dan Fandrich [Thu, 8 Feb 2024 19:34:34 +0000 (11:34 -0800)] 
configure: build & install shell completions when enabled

The --with-fish-functions-dir and --with-zsh-functions-dir options
currently have no effect on a normal build because the scripts/ directory
where they're used is not built. Add scripts/ to a normal build and
change the completion options to default to off to preserve the existing
behaviour.

Closes: #12906
3 months agogithub/labeler: improve the match patterns
Dan Fandrich [Thu, 29 Feb 2024 21:04:23 +0000 (13:04 -0800)] 
github/labeler: improve the match patterns

3 months agotests: add test1598 for POST with trailers
Stefan Eissing [Wed, 28 Feb 2024 12:20:59 +0000 (13:20 +0100)] 
tests: add test1598 for POST with trailers

- test POST fields with trailers and chunked encoding

Ref: #12938
Closes #13009

3 months agocmdline-opts/_VERSION: provide %VERSION correctly
Daniel Stenberg [Wed, 28 Feb 2024 10:40:04 +0000 (11:40 +0100)] 
cmdline-opts/_VERSION: provide %VERSION correctly

... so that it does not get included verbatim in the output. Fixes a
regression shipped in 8.6.0.

Also fix a format mistake in form.md

Closes #13008

3 months agolib: Curl_read/Curl_write clarifications
Stefan Eissing [Thu, 15 Feb 2024 15:22:53 +0000 (16:22 +0100)] 
lib: Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
  clarify when and at what level they operate
- send/recv of transfer related data is now done via
  `Curl_xfer_send()/Curl_xfer_recv()` which no longer has
  socket/socketindex as parameter. It decides on the transfer
  setup of `conn->sockfd` and `conn->writesockfd` on which
  connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
  `Curl_conn_send()/Curl_conn_recv()` which get the socket index
  as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
  naming consistency
- clarify that the special CURLE_AGAIN hangling to return
  `CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
  and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
  when it arrived in more than a single chunk (to be made
  into a sperate PR, also)

Added as documented [in
CLIENT-READER.md](https://github.com/curl/curl/blob/5b1f31dfbab8aef467c419c68aa06dc738cb75d4/docs/CLIENT-READERS.md).

- old `Curl_buffer_send()` completely replaced by new `Curl_req_send()`
- old `Curl_fillreadbuffer()` replaced with `Curl_client_read()`
- HTTP chunked uploads are now formatted in a client reader added when
  needed.
- FTP line-end conversions are done in a client reader added when
  needed.
- when sending requests headers, remaining buffer space is filled with
  body data for sending in "one go". This is independent of the request
  body size. Resolves #12938 as now small and large requests have the
  same code path.

Changes done to test cases:

- test513: now fails before sending request headers as this initial
  "client read" triggers the setup fault. Behaves now the same as in
  hyper build
- test547, test555, test1620: fix the length check in the lib code to
  only fail for reads *smaller* than expected. This was a bug in the
  test code that never triggered in the old implementation.

Closes #12969

3 months agocurldown: Fix email address in Copyright
Daniel Gustafsson [Wed, 28 Feb 2024 10:28:10 +0000 (11:28 +0100)] 
curldown: Fix email address in Copyright

The curldown conversion accidentally replaced daniel@haxx.se with
just daniel.se.  This reverts back to the proper email address in
the curldown docs as well as in a few other stray places where it
was incorrect (while unrelated to curldown).

Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Closes: #12997
3 months agogetparam: make --ftp-ssl work again
Daniel Stenberg [Wed, 28 Feb 2024 07:02:49 +0000 (08:02 +0100)] 
getparam: make --ftp-ssl work again

Follow-up to 9e4e527 which accidentally broke it

Reported-by: Jordan Brown
Fixes #13006
Closes #13007

3 months agoKNOWN_BUGS: IMAPS connection fails with rustls error 13005/head
Daniel Stenberg [Tue, 27 Feb 2024 16:50:25 +0000 (17:50 +0100)] 
KNOWN_BUGS: IMAPS connection fails with rustls error

Closes #10457

3 months agoKNOWN_BUGS: FTPS upload, FileZilla, GnuTLS and close_notify
Daniel Stenberg [Tue, 27 Feb 2024 16:48:18 +0000 (17:48 +0100)] 
KNOWN_BUGS: FTPS upload, FileZilla, GnuTLS and close_notify

Closes #11383

3 months agoKNOWN_BUGS: Implicit FTPS upload timeout
Daniel Stenberg [Tue, 27 Feb 2024 16:46:19 +0000 (17:46 +0100)] 
KNOWN_BUGS: Implicit FTPS upload timeout

Closes #11720

3 months agoKNOWN_BUGS: HTTP/2 prior knowledge over proxy
Daniel Stenberg [Tue, 27 Feb 2024 16:44:55 +0000 (17:44 +0100)] 
KNOWN_BUGS: HTTP/2 prior knowledge over proxy

Closes #12641

3 months agoTODO: build HTTP/3 with OpenSSL and nghttp3 using cmake
Daniel Stenberg [Tue, 27 Feb 2024 16:43:22 +0000 (17:43 +0100)] 
TODO: build HTTP/3 with OpenSSL and nghttp3 using cmake

Closes #12988

3 months agoTODO: Select signature algorithms
Daniel Stenberg [Tue, 27 Feb 2024 16:41:40 +0000 (17:41 +0100)] 
TODO: Select signature algorithms

Closes #12982

3 months agoexamples: use present tense in comments
Daniel Stenberg [Tue, 27 Feb 2024 11:29:27 +0000 (12:29 +0100)] 
examples: use present tense in comments

remove "will" and some other word fixes

Closes #13003

3 months agodocs: more language cleanups
Daniel Stenberg [Tue, 27 Feb 2024 09:35:28 +0000 (10:35 +0100)] 
docs: more language cleanups

- present tense
- avoid bad words

Closes #13003

3 months agosetopt: Fix disabling all protocols
Daniel Gustafsson [Tue, 27 Feb 2024 14:43:56 +0000 (15:43 +0100)] 
setopt: Fix disabling all protocols

When disabling all protocols without enabling any, the resulting
set of allowed protocols remained the default set.  Clearing the
allowed set before inspecting the passed value from --proto make
the set empty even in the errorpath of no protocols enabled.

Co-authored-by: Dan Fandrich <dan@telarity.com>
Reported-by: Dan Fandrich <dan@telarity.com>
Reviewed-by: Daniel Stenberg <daniel@haxx.se>
Closes: #13004
3 months agofopen: fix narrowing conversion warning on 32-bit Android
Andreas Kiefer [Mon, 26 Feb 2024 20:08:10 +0000 (21:08 +0100)] 
fopen: fix narrowing conversion warning on 32-bit Android

This was fixed in commit 06dc599405f, but came back in commit
03cb1ff4d62.

When building for 32-bit ARM or x86 Android, `st_mode` is defined as
`unsigned int` instead of `mode_t`, resulting in a
`-Wimplicit-int-conversion` clang warning because `mode_t` is
`unsigned short`. Add a cast to silence the warning, but only for
32-bit Android builds, because other architectures and platforms are
not affected.

Ref: https://android.googlesource.com/platform/bionic/+/refs/tags/ndk-r25c/libc/include/sys/stat.h#86
Closes https://github.com/curl/curl/pull/12998

3 months agolib: Curl_read/Curl_write clarifications
Stefan Eissing [Thu, 15 Feb 2024 15:22:53 +0000 (16:22 +0100)] 
lib: Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to
  clarify when and at what level they operate
- send/recv of transfer related data is now done via
  `Curl_xfer_send()/Curl_xfer_recv()` which no longer has
  socket/socketindex as parameter. It decides on the transfer
  setup of `conn->sockfd` and `conn->writesockfd` on which
  connection filter chain to operate.
- send/recv on a specific connection filter chain is done via
  `Curl_conn_send()/Curl_conn_recv()` which get the socket index
  as parameter.
- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for
  naming consistency
- clarify that the special CURLE_AGAIN hangling to return
  `CURLE_OK` with length 0 only applies to `Curl_xfer_send()`
  and CURLE_AGAIN is returned by all other send() variants.
- fix a bug in websocket `curl_ws_recv()` that mixed up data
  when it arrived in more than a single chunk

The method for sending not just raw bytes, but bytes that are either
"headers" or "body". The send abstraction stack, to to bottom, now is:

* `Curl_req_send()`: has parameter to indicate amount of header bytes,
  buffers all data.
* `Curl_xfer_send()`: knows on which socket index to send, returns
  amount of bytes sent.
* `Curl_conn_send()`: called with socket index, returns amount of bytes
  sent.

In addition there is `Curl_req_flush()` for writing out all buffered
bytes.

`Curl_req_send()` is active for requests without body,
`Curl_buffer_send()` still being used for others. This is because the
special quirks need to be addressed in future parts:

* `expect-100` handling
* `Curl_fillreadbuffer()` needs to add directly to the new
  `data->req.sendbuf`
* special body handlings, like `chunked` encodings and line end
  conversions will be moved into something like a Client Reader.

In functions of the pattern `CURLcode xxx_send(..., ssize_t *written)`,
replace the `ssize_t` with a `size_t`. It makes no sense to allow for negative
values as the returned `CURLcode` already specifies error conditions. This
allows easier handling of lengths without casting.

Closes #12964

3 months agomulti: make add_handle free any multi_easy
Daniel Stenberg [Mon, 26 Feb 2024 08:50:49 +0000 (09:50 +0100)] 
multi: make add_handle free any multi_easy

If the easy handle that is being added to a multi handle has previously
been used for curl_easy_perform(), there is a private multi handle here
that we can kill off. While it flushes some caches etc for the easy
handle would it be used for an easy interface transfer again after being
used in the multi stack, this cleanup simplifies behavior and uses less
memory.

Closes #12992

3 months agodocs: use present tense
Daniel Stenberg [Tue, 27 Feb 2024 06:48:10 +0000 (07:48 +0100)] 
docs: use present tense

avoid "will", detect "will" as a bad word in the CI

Also line wrapped a bunch of paragraphs

Closes #13001

3 months agoCURLOPT_SSL_CTX_FUNCTION.md: no promises of lifetime after return
Daniel Stenberg [Mon, 26 Feb 2024 21:27:07 +0000 (22:27 +0100)] 
CURLOPT_SSL_CTX_FUNCTION.md: no promises of lifetime after return

... and cleanup other language.

Closes #12999

3 months agolib: send rework
Stefan Eissing [Wed, 14 Feb 2024 11:09:32 +0000 (12:09 +0100)] 
lib: send rework

Curl_read/Curl_write clarifications

- replace `Curl_read()`, `Curl_write()` and `Curl_nwrite()` to 1clarify
  when and at what level they operate

- send/recv of transfer related data is now done via
  `Curl_xfer_send()/Curl_xfer_recv()` which no longer has
  socket/socketindex as parameter. It decides on the transfer setup of
  `conn->sockfd` and `conn->writesockfd` on which connection filter
  chain to operate.

- send/recv on a specific connection filter chain is done via
  `Curl_conn_send()/Curl_conn_recv()` which get the socket index as
  parameter.

- rename `Curl_setup_transfer()` to `Curl_xfer_setup()` for naming
  consistency

- clarify that the special CURLE_AGAIN handling to return `CURLE_OK`
  with length 0 only applies to `Curl_xfer_send()` and CURLE_AGAIN is
  returned by all other send() variants.

SingleRequest reshuffling

- move functions into request.[ch]
- differentiate between reset and free
- add Curl_req_done() to perform last actions
- add a send `bufq` to SingleRequest for future use in keeping upload data

Closes #12963

3 months agoRELEASE-NOTES: synced
Daniel Stenberg [Mon, 26 Feb 2024 22:05:51 +0000 (23:05 +0100)] 
RELEASE-NOTES: synced

3 months agohttp_chunks: remove unused 'endptr' variable
Daniel Stenberg [Mon, 26 Feb 2024 11:27:07 +0000 (12:27 +0100)] 
http_chunks: remove unused 'endptr' variable

Closes #12996

3 months agolib: initialize output pointers to NULL before calling strto[ff,l,ul]
Louis Solofrizzo [Mon, 26 Feb 2024 11:07:22 +0000 (12:07 +0100)] 
lib: initialize output pointers to NULL before calling strto[ff,l,ul]

In order to make MSAN happy:

    ==2200945==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x596f3b3ed246 in curlx_strtoofft [...]/libcurl/src/lib/strtoofft.c:239:11
    #1 0x596f3b402156 in Curl_httpchunk_read [...]/libcurl/src/lib/http_chunks.c:149:12
    #2 0x596f3b348550 in readwrite_data [...]/libcurl/src/lib/transfer.c:607:11
    [...]

    ==2202041==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x5a3fab66a72a in Curl_parse_port [...]/libcurl/src/lib/urlapi.c:547:8
    #1 0x5a3fab650645 in parse_authority [...]/libcurl/src/lib/urlapi.c:796:12
    #2 0x5a3fab6740f6 in parseurl [...]/libcurl/src/lib/urlapi.c:1176:16
    #3 0x5a3fab664fc5 in parseurl_and_replace [...]/libcurl/src/lib/urlapi.c:1342:12
    [...]

    ==2202320==WARNING: MemorySanitizer: use-of-uninitialized-value
    #0 0x569076a0d6b0 in ipv4_normalize [...]/libcurl/src/lib/urlapi.c:683:12
    #1 0x5690769f2820 in parse_authority [...]/libcurl/src/lib/urlapi.c:803:10
    #2 0x569076a160f6 in parseurl [...]/libcurl/src/lib/urlapi.c:1176:16
    #3 0x569076a06fc5 in parseurl_and_replace [...]/libcurl/src/lib/urlapi.c:1342:12
    [...]

Signed-off-by: Louis Solofrizzo <lsolofrizzo@scaleway.com>
Closes #12995

3 months agolib: move client writer into own source
Stefan Eissing [Wed, 7 Feb 2024 11:05:05 +0000 (12:05 +0100)] 
lib: move client writer into own source

Refactoring of the client writer that passes the data to the
client/application's callback functions.

- split out into own source cw-out.[ch] from sendf.c

- move tempwrite and tempcount from data->state into the context of the
  client writer

- redesign the 3 tempwrite dynbufs as a linked list of dynbufs. On
  paused transfers, this allows to "record" interleaved HEADER/BODY
  chunks to be "played back" in the same order on unpausing.

- keep the overall size limit of all buffered data to DYN_PAUSE_BUFFER.
  On exceeding that, return CURLE_TOO_LARGE instead of
  CURLE_OUT_OF_MEMORY as before.

- add method to be called when a transfer is DONE to allow writing of
  any data still buffered

- when paused, record HEADER writes exactly as they come for later
  playback. HEADERs are documented to be written one-by-one.

Closes #12898

3 months agourldata: move authneg bit from conn to Curl_easy
Stefan Eissing [Fri, 16 Feb 2024 11:15:10 +0000 (12:15 +0100)] 
urldata: move authneg bit from conn to Curl_easy

- from `conn->bits.authneg` to `data->req.authneg`
- this is a property of the request about to be made
  and not a property of the connection
- in multiuse connections, transfer could step on each others
  toes here potentially.

Closes #12949

3 months agoc-hyper: add header collection writer in hyper builds
Stefan Eissing [Tue, 6 Feb 2024 13:56:05 +0000 (14:56 +0100)] 
c-hyper: add header collection writer in hyper builds

Closes #12880

3 months agohttp: move headers collecting to writer
Stefan Eissing [Tue, 6 Feb 2024 12:55:07 +0000 (13:55 +0100)] 
http: move headers collecting to writer

- add a client writer that does "push" response
  headers written to the client if the headers api
  is enabled
- remove special handling in sendf.c
- needs to be installed very early on connection
  setup to catch CONNECT response headers

Closes #12880

3 months agosendf: Curl_client_write(), make passed in buf const
Stefan Eissing [Tue, 6 Feb 2024 11:10:19 +0000 (12:10 +0100)] 
sendf: Curl_client_write(), make passed in buf const

3 months agolib: remove curl_mimepart object when CURL_DISABLE_MIME
MAntoniak [Fri, 9 Feb 2024 17:20:47 +0000 (18:20 +0100)] 
lib: remove curl_mimepart object when CURL_DISABLE_MIME

Remove curl_mimepart object from UserDefined structure when
CURL_DISABLE_MIME flag is active. Reduce size of UserDefined structure.

Also remove unreachable code: when CURL_DISABLE_MIME is set, httpreq can
never have HTTPREQ_POST_MIME value and the same goes for the
CURL_DISABLE_FORM_API flag and the HTTPREQ_POST_FORM value

Closes #12948

3 months agorustls: make curl compile with 0.12.0
kpcyrd [Sun, 25 Feb 2024 20:50:18 +0000 (21:50 +0100)] 
rustls: make curl compile with 0.12.0

Closes #12989

3 months agostrtoofft: fix the overflow check
Daniel Stenberg [Sun, 25 Feb 2024 21:52:40 +0000 (22:52 +0100)] 
strtoofft: fix the overflow check

... to not rely on wrapping, since it is an undefined behavior that is
not what always might happen. This is in our private strtoff() parser
function, used only on platforms without a native version.

Reported-by: vulnerabilityspotter on hackerone
Closes #12990

3 months agolibssh/libssh2: return error on too big range
Daniel Stenberg [Sat, 24 Feb 2024 21:38:22 +0000 (22:38 +0100)] 
libssh/libssh2: return error on too big range

If trying to get the range 0 - 2^63 and the remote file is 2^63 bytes or
larger.

Fixes #12983
Closes #12984

3 months agosetopt: fix check for CURLOPT_PROXY_TLSAUTH_TYPE value
Scott Talbert [Sat, 24 Feb 2024 03:02:09 +0000 (22:02 -0500)] 
setopt: fix check for CURLOPT_PROXY_TLSAUTH_TYPE value

Prior to this change CURLOPT_PROXY_TLSAUTH_TYPE would return
CURLE_BAD_FUNCTION_ARGUMENT on any type other than NULL. Since there is
only one type of TLS auth and it is also the default (SRP) the TLS auth
would work anyway.

Closes https://github.com/curl/curl/pull/12981