]> git.ipfire.org Git - thirdparty/curl.git/log
thirdparty/curl.git
10 months agoci: tidy-ups
Viktor Szakats [Sat, 28 Sep 2024 10:23:48 +0000 (12:23 +0200)] 
ci: tidy-ups

- http3-linux: add newlines for readability.
- http3-linux: use `make pytest` to run pytest.
- checksrc: use `|` multiline to sync with other uses.
- checksrc: prefer `$()`.
- prefer `>-` over `>` for folded option lists.
- cygwin: drop a `grep` no longer necessary.

Closes #15081

10 months agocmake: websockets tidy-ups
Viktor Szakats [Fri, 27 Sep 2024 16:16:57 +0000 (18:16 +0200)] 
cmake: websockets tidy-ups

- restore change lost after websockets-default update.
  Ref: 6a1dcdc5d2f1b450de4d10739660b32d081c51a1 #14998
- delete unused line after websockets is on by default.
  Follow-up to d78e129d50b2d190f1c1bde2ad1f62f02f152db0 #14936

Closes #15078

10 months agoGHA linux: restore `apt-get update`
Viktor Szakats [Sat, 28 Sep 2024 18:42:38 +0000 (20:42 +0200)] 
GHA linux: restore `apt-get update`

It seems like the Ubuntu apt mirror list or other preloaded data on the
Linux runner can go stale (?) throughout a single week causing failures
like this, in some of the jobs:
```
Ign:16 http://security.ubuntu.com/ubuntu jammy-updates/main amd64 libapr1-dev amd64 1.7.0-8ubuntu0.22.04.1
Err:16 mirror+file:/etc/apt/apt-mirrors.txt jammy-updates/main amd64 libapr1-dev amd64 1.7.0-8ubuntu0.22.04.1
  404  Not Found [IP: 40.81.13.82 80]
Get:24 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 libnghttp2-dev amd64 1.43.0-1ubuntu0.2 [117 kB]
Get:25 http://azure.archive.ubuntu.com/ubuntu jammy-updates/main amd64 vsftpd amd64 3.0.5-0ubuntu1.1 [123 kB]
Fetched 4029 kB in 2s (1682 kB/s)
E: Failed to fetch mirror+file:/etc/apt/apt-mirrors.txt/pool/main/a/apr/libapr1-dev_1.7.0-8ubuntu0.22.04.1_amd64.deb  404  Not Found [IP: 40.81.13.82 80]
E: Unable to fetch some archives, maybe run apt-get update or try with --fix-missing?
Error: Process completed with exit code 100.
```
https://github.com/curl/curl/actions/runs/11074211380/job/30772496037#step:32:63

Restore `apt update` to fix it.

Follow-up to 1b8449674adb57ee0f60e761d654c69b20ee8fcf #14992

Closes #15082

10 months agodocs/cmdline-opts: GnuTLS supports PKCS#11 URI in --cert option
Tatsuhiro Tsujikawa [Fri, 27 Sep 2024 08:30:06 +0000 (17:30 +0900)] 
docs/cmdline-opts: GnuTLS supports PKCS#11 URI in --cert option

Closes #15075

10 months agosingleuse: limit checks to non-unity jobs
Viktor Szakats [Thu, 26 Sep 2024 20:42:48 +0000 (22:42 +0200)] 
singleuse: limit checks to non-unity jobs

`singleuse.pl` requires non-unity builds. With unity builds, it remains
silent.

- make some jobs non-unity to allow singleuse to do its job.
- since all jobs used the `--unit` singleuse checker option, make this
  option permanent.

Closes #15065

10 months agoGHA/windows: formatting, adjust timeouts, tidy-ups
Viktor Szakats [Mon, 23 Sep 2024 11:58:51 +0000 (13:58 +0200)] 
GHA/windows: formatting, adjust timeouts, tidy-ups

- reduce job timeouts for msys2 and old-mingw-w64 jobs.
- fold long lines and reorder options in MSVC jobs.
- make some options default in MSVC jobs.

Closes #15017

10 months agoCI: update 32-bit CI to Ubuntu 24.04 and enable more
Dan Fandrich [Thu, 26 Sep 2024 20:42:09 +0000 (13:42 -0700)] 
CI: update 32-bit CI to Ubuntu 24.04 and enable more

Enable librtmp, libssh, libidn2 and c-ares support for broader 32-bit
test coverage. Bump the gcc version to 14.

Closes #15068

10 months agoCI: improvements in test reliability and performance
Stefan Eissing [Wed, 25 Sep 2024 13:14:50 +0000 (15:14 +0200)] 
CI: improvements in test reliability and performance

- CI default: raise parallelism to 20
- CI valgind: set parallelism to 6
- CI non-native: adapt parallelism for OS builds
- CI Windows: no longer ignore FTP, TFTP, MQTT and SMTP
- CI Windows: restrict test timeout to 10 minutes
- CI Windows: do not run tests for msh3 build
- tests, various: restrict curl invocation to ipv4 to avoid talking to
  any ipv6 test server running in parallel
- tests: requiring http/2 server no longer needs to mention http server
- test 190: use a fixed timeout of 10 seconds instead of %FTPTIME2, as
  that value rises under parallel load beyond what the server is waiting
  and then produces different results
- test 1540: add debug logging
- testrunner: add verify checks for http/2 and http/3 server

Closes #15040

10 months agolib: fix disabled-verbose-strings + enable-debug build warnings
Daniel Stenberg [Fri, 27 Sep 2024 11:19:55 +0000 (13:19 +0200)] 
lib: fix disabled-verbose-strings + enable-debug build warnings

10 months agoWebSockets: make support official (non-experimental)
Daniel Stenberg [Fri, 27 Sep 2024 11:19:55 +0000 (13:19 +0200)] 
WebSockets: make support official (non-experimental)

Inverts the configure/cmake options to instead provide options that
disable WebSockets and have them (ws + wss) enabled by default.

Closes #14936

10 months agocodespell: extend checks to more subdirs
Viktor Szakats [Thu, 26 Sep 2024 23:16:38 +0000 (01:16 +0200)] 
codespell: extend checks to more subdirs

- fix issues found.
- fix a few more found locally.

Closes #15072

10 months agoGHA/torture: prefer pip `--break-system-packages` for speed
Viktor Szakats [Thu, 26 Sep 2024 22:46:26 +0000 (00:46 +0200)] 
GHA/torture: prefer pip `--break-system-packages` for speed

Follow-up to c5e3d8ba94791dbc51f98a56ce07b3ba6b0aa446 #14972

Closes #15071

10 months agosingleuse: make `git grep` faster, add Apple `nm` support
Viktor Szakats [Thu, 26 Sep 2024 19:16:21 +0000 (21:16 +0200)] 
singleuse: make `git grep` faster, add Apple `nm` support

- avoid regexp in grep to make it run faster.
- add support for parsing Apple `nm` output:
  - skip leading underscore from function names.
  - pick object name from output.

Closes #15070

10 months agoGHA/http3-linux: add name to align with other Linux workflows
Viktor Szakats [Thu, 26 Sep 2024 19:38:38 +0000 (21:38 +0200)] 
GHA/http3-linux: add name to align with other Linux workflows

Closes #15069

10 months agoRELEASE-NOTES: synced
Daniel Stenberg [Thu, 26 Sep 2024 21:52:47 +0000 (23:52 +0200)] 
RELEASE-NOTES: synced

10 months agoGHA/linux: improve cmake use, switch to Ninja
Viktor Szakats [Mon, 23 Sep 2024 22:14:55 +0000 (00:14 +0200)] 
GHA/linux: improve cmake use, switch to Ninja

- cmake: allow easy switching of generator (= make tool).
- merge autotools/cmake job steps.
- cmake: switch to Ninja.
  (build was already fast, Ninja doesn't make it noticeably faster)

Closes #15023

10 months agomulti: avoid reading whole struct pointer from pointer
Daniel Stenberg [Thu, 26 Sep 2024 14:05:20 +0000 (16:05 +0200)] 
multi: avoid reading whole struct pointer from pointer

The proper alignment is not guaranteed. This function now instead uses
only the first and last byte of the key since they are the ones likely
to change most (one of them, depending on CPU endian) and the hash is
tiny anyway.

Closes #15063

10 months agotests: use '-4' where needed
Stefan Eissing [Thu, 26 Sep 2024 13:20:43 +0000 (15:20 +0200)] 
tests: use '-4' where needed

Our test servers run either on ipv4 *or* on ipv6, as requested.
A test case using 'localhost' or '*.local' in the url needs to
run with the specific version of the server started.

Otherwise, curl's "happy eyeball"ing will connect to another
server that may be running due to parallel testing or for some
other reasons.

Note that port reuse here depends on the OS strategy and it
seems netbsd is especially likely to hit this.

Closes #15060

10 months agotests: improve mqtt server handling
Stefan Eissing [Thu, 26 Sep 2024 13:00:27 +0000 (15:00 +0200)] 
tests: improve mqtt server handling

Check that the mqtt server process actually is running.
Handle its port number similar to other servers.

Closes #15059

10 months agotests: check http/2 and http/3 server responsiveness
Stefan Eissing [Thu, 26 Sep 2024 12:55:56 +0000 (14:55 +0200)] 
tests: check http/2 and http/3 server responsiveness

Check responsiveness of http/3 server when running.

Also, a test case with http/2 or http/3 server requirement
now implicitly drags in a 'http' server and we need no longer
mention that in testdata.

Closes #15058

10 months agotest190: replace %FTPTIME2 with a fixed value
Stefan Eissing [Thu, 26 Sep 2024 12:48:42 +0000 (14:48 +0200)] 
test190: replace %FTPTIME2 with a fixed value

The variable FTPTIME2 may, on a loaded test server, become so
large that the timeout does not happen before the fixed 60 seconds
the server waits.

Closes #15056

10 months agotests: remove the %FTPTIME3 variable
Daniel Stenberg [Thu, 26 Sep 2024 14:46:49 +0000 (16:46 +0200)] 
tests: remove the %FTPTIME3 variable

Nothing uses it.

Ref: #15056
Closes #15064

10 months agosocks_gssapi: switch to dynbuf from buffer with strcpy
Daniel Stenberg [Thu, 26 Sep 2024 12:52:33 +0000 (14:52 +0200)] 
socks_gssapi: switch to dynbuf from buffer with strcpy

Closes #15057

10 months agotests: add file: tests with existing files
Dan Fandrich [Wed, 25 Sep 2024 20:30:29 +0000 (13:30 -0700)] 
tests: add file: tests with existing files

Windows sometimes has issues when opening the same file twice, so these
test two situations where that could potentially occur.

Reported-by: ralfjunker on github
Ref: #15043
Closes #15045

10 months agotest518: restore valgrind disable
Daniel Stenberg [Thu, 26 Sep 2024 14:08:12 +0000 (16:08 +0200)] 
test518: restore valgrind disable

Follow-up to c91c37b6e87c

The test does not work well when run with valgrind

Closes #15062

10 months agoopenssl: convert a memcpy to dynbuf use
Daniel Stenberg [Thu, 26 Sep 2024 11:22:57 +0000 (13:22 +0200)] 
openssl: convert a memcpy to dynbuf use

and avoid an alloc for CN handling unless necessary

Closes #15049

10 months agotest1540: add debug logging
Stefan Eissing [Thu, 26 Sep 2024 12:46:18 +0000 (14:46 +0200)] 
test1540: add debug logging

Closes #15055

10 months agotest504: fix handling on pending connect
Stefan Eissing [Thu, 26 Sep 2024 12:43:54 +0000 (14:43 +0200)] 
test504: fix handling on pending connect

Test expected a connect to a port no one is listening to immediately
fail. But Windows has its internal retry logic that may fail this.

As fix, multi_perform()/multi_wait() until transfer is done.

Closes #15054

10 months agotest2502: add libtest debug tracing
Stefan Eissing [Thu, 26 Sep 2024 12:40:48 +0000 (14:40 +0200)] 
test2502: add libtest debug tracing

Closes #15053

10 months agotestrun: explicitly set proper IP address for stunnel listen/connect
Stefan Eissing [Thu, 26 Sep 2024 08:28:38 +0000 (10:28 +0200)] 
testrun: explicitly set proper IP address for stunnel listen/connect

Closes #15051

10 months agolib/cw-out: initialize 'flush_all' directly
Gabriel Marin [Wed, 25 Sep 2024 20:00:36 +0000 (23:00 +0300)] 
lib/cw-out: initialize 'flush_all' directly

Closes #15044

10 months agotest1035: convert host name back to utf8 as should be
Stefan Eissing [Thu, 26 Sep 2024 12:34:20 +0000 (14:34 +0200)] 
test1035: convert host name back to utf8 as should be

Follow-up to 0b70b23ef4d00

Closes #15050

10 months agoopenssl: remove two strcpy() calls
Daniel Stenberg [Thu, 26 Sep 2024 12:39:04 +0000 (14:39 +0200)] 
openssl: remove two strcpy() calls

Closes #15052

10 months agotool_doswin: simplify; remove unused options and strncpy calls 15047/head
Daniel Stenberg [Thu, 26 Sep 2024 06:26:11 +0000 (08:26 +0200)] 
tool_doswin: simplify; remove unused options and strncpy calls

SANITIZE_ALLOW_TRUNCATE and SANITIZE_ALLOW_COLONS were never used by
code, thus only making the code complicated for no good use.

Since nothing should truncate, using strncpy() is wrong.

Two cases of malloc + copy replaced with proper strdup() calls.

Fixup unit test 1604 accordingly.

Closes #15047

10 months agotests: add codeset-utf8 as a feature
Daniel Stenberg [Wed, 25 Sep 2024 12:25:55 +0000 (14:25 +0200)] 
tests: add codeset-utf8 as a feature

To avoid having to use <precheck> for tests that require UTF-8 support.

Closes #15039

10 months agotests: introduce %CLIENT6IP-NB
Daniel Stenberg [Wed, 25 Sep 2024 12:11:01 +0000 (14:11 +0200)] 
tests: introduce %CLIENT6IP-NB

This is the %CLIENT6IP variable - but without outmost brackets since
some commmand lines need the address without the brackets. With this
variable we can run three more tests without prechecks.

Closes #15039

10 months agotests: make precheck for HTTP on 127.0.0.1 into a feature
Daniel Stenberg [Wed, 25 Sep 2024 11:53:17 +0000 (13:53 +0200)] 
tests: make precheck for HTTP on 127.0.0.1 into a feature

It can now be required easily in <features> instead of having perl code
in a <precheck>.

Closes #15039

10 months agotests: postcheck is now in verify
Daniel Stenberg [Wed, 25 Sep 2024 21:36:05 +0000 (23:36 +0200)] 
tests: postcheck is now in verify

Also introduce 'notexists' for verifying that directory entries do not
exist after a test. Now an explicit supported feature instead of using
"funny" perl in postcheck.

Closes #15046

10 months agobuild: fix cross-compile check for poll with bionic
Daniel Stenberg [Wed, 25 Sep 2024 07:38:07 +0000 (09:38 +0200)] 
build: fix cross-compile check for poll with bionic

Since it seems the _POSIX_C_SOURCE "trick" does not work there, the
check does not find poll().

Fixes #15013
Reported-by: vvb2060 on github
Closes #15037

10 months agoTHANKS: cleanup duplicates
Daniel Stenberg [Wed, 25 Sep 2024 07:54:09 +0000 (09:54 +0200)] 
THANKS: cleanup duplicates

10 months agobuild: add pytest targets
Viktor Szakats [Mon, 23 Sep 2024 23:53:27 +0000 (01:53 +0200)] 
build: add pytest targets

It enables running pytests in cmake jobs, regardless of underlying build
tool choice (= makes it work with ninja.)

Also:

- drop pytest logic launching `make` and exiting in case of failure.
  Maybe there is a better way and keep this functionality somehow, bind
  it to a command-line option? make it fail softly?

- GHA/linux: invoke pytest via the build, not directly.

- autotools: add missing dummy runtests targets when cross-compiling.

Closes #15034

10 months agoGHA/linux: tidy up msh3 build step
Viktor Szakats [Mon, 23 Sep 2024 20:26:45 +0000 (22:26 +0200)] 
GHA/linux: tidy up msh3 build step

Ninja does not improve msh3 build speed on GHA/linux:
https://github.com/curl/curl/actions/runs/11020206432/job/30604509300

Cherry-picked from #15023

10 months agobuild: clarify CA embed is for curl tool, mark default, improve summary
Viktor Szakats [Tue, 24 Sep 2024 19:07:30 +0000 (21:07 +0200)] 
build: clarify CA embed is for curl tool, mark default, improve summary

- say that CA embed is for the curl tool.
- show "no" in summary when there is no CA embed.
- cmake: sync wording.

Closes #15035

10 months agoGHA/linux: review and prune valgrind use
Viktor Szakats [Mon, 23 Sep 2024 19:05:37 +0000 (21:05 +0200)] 
GHA/linux: review and prune valgrind use

Valgrind jobs are slow, drop it from jobs where its use is redundant
and/or has limited impact:

- BearSSL: deprecated.
- LibreSSL heimdal with autotools.
  Keep valgrind for the same job with cmake.
- msh3.
- IntelC no-SSL.
  Keep valgrind for IntelC OpenSSL.
- OpenSSL 3.
  All OpenSSL jobs are v3 now, keep valgrind for the `-O3`, and
  libssh2 + sync-resolver variants.

Closes #15020

10 months agotidy-up: indentation in autotools sources
Viktor Szakats [Wed, 28 Aug 2024 16:01:27 +0000 (18:01 +0200)] 
tidy-up: indentation in autotools sources

Indentation in `configure.ac` and `.m4` files.

Closes #14692

10 months agocleanup: added space around ternary expressions
Gabriel Marin [Sun, 22 Sep 2024 21:32:44 +0000 (00:32 +0300)] 
cleanup: added space around ternary expressions

Closes #14990

10 months agotest1185: Updated test data with NOSPACEC rule
Gabriel Marin [Sun, 22 Sep 2024 21:23:37 +0000 (00:23 +0300)] 
test1185: Updated test data with NOSPACEC rule

Closes #14990

10 months agochecksrc: Added checks for colon operator in ternary expressions
Gabriel Marin [Fri, 20 Sep 2024 22:53:34 +0000 (01:53 +0300)] 
checksrc: Added checks for colon operator in ternary expressions

Closes #14990

10 months agoconfigure: improve help string for some options
Ian Spence [Tue, 24 Sep 2024 17:48:59 +0000 (10:48 -0700)] 
configure: improve help string for some options

Closes #15033

10 months agocurl_trc: fix build with verbose messages disabled
Baruch Siach [Tue, 24 Sep 2024 04:52:13 +0000 (07:52 +0300)] 
curl_trc: fix build with verbose messages disabled

Add empty definition of Curl_trc_ws() to fix this following build error:

In file included from sendf.h:29,
                 from ws.c:35:
ws.c: In function 'Curl_ws_accept':
curl_trc.h:100:10: error: implicit declaration of function 'Curl_trc_ws'; did you mean 'Curl_trc_ftp'? [-Wimplicit-function-declaration]
  100 |          Curl_trc_ws(data, __VA_ARGS__); } while(0)
      |          ^~~~~~~~~~~
ws.c:779:5: note: in expansion of macro 'CURL_TRC_WS'
  779 |     CURL_TRC_WS(data, "WS, using chunk size %zu", chunk_size);
      |     ^~~~~~~~~~~

Closes #15026

10 months agomulti.c: warn/assert on stall only without timer
Stefan Eissing [Tue, 24 Sep 2024 11:06:03 +0000 (13:06 +0200)] 
multi.c: warn/assert on stall only without timer

Warn/assert about a possibly stalling transfer only when it
has no timeout set.

The assertion triggered in test 1540 on loaded CI sometimes.

Closes #15028

10 months agoGHA/linux: merge AWS-LC workflow
Viktor Szakats [Tue, 24 Sep 2024 13:01:26 +0000 (15:01 +0200)] 
GHA/linux: merge AWS-LC workflow

Closes #15031

10 months agoGHA/linux: merge wolfSSL workflow
Viktor Szakats [Tue, 24 Sep 2024 12:26:20 +0000 (14:26 +0200)] 
GHA/linux: merge wolfSSL workflow

Add wolfSSL builds to cache.

Also apply -j4 for the valgrind job (was: -j2).

Closes #15030

10 months agobuild: show if CA bundle to embed was found
Viktor Szakats [Tue, 24 Sep 2024 09:51:02 +0000 (11:51 +0200)] 
build: show if CA bundle to embed was found

Follow-up to 8a3740bc8e558b9a9d4a652b74cf27a0961d7010 #14059
Closes #15027

10 months agoGHA/linux: enable test bundles for cmake jobs
Viktor Szakats [Mon, 23 Sep 2024 22:02:38 +0000 (00:02 +0200)] 
GHA/linux: enable test bundles for cmake jobs

Test build step speed-up (3x): 18s -> 6s

Follow-up to 71cf0d1fca9e1f53524e1545ef0c08d174458d80 #14772
Closes #15022

10 months agobuild: fix possible `-Wformat-overflow` in lib557 with test bundle builds
Viktor Szakats [Sun, 22 Sep 2024 23:22:36 +0000 (01:22 +0200)] 
build: fix possible `-Wformat-overflow` in lib557 with test bundle builds

- lib557: suppress `-Wformat-overflow` warning in source.
  Fixes:
  ```
  lib557.c: In function â€˜test_float_formatting’:
  lib557.c:1408:37: error: â€˜%*f’ directive output of 2147483648 bytes exceeds â€˜INT_MAX’ [-Werror=format-overflow=]
   1408 |   curl_msnprintf(buf, sizeof(buf), "%*f", INT_MIN, 9.1);
        |                                     ^~~
  lib557.c:1408:3: note: â€˜curl_msnprintf’ output 2147483649 bytes
   1408 |   curl_msnprintf(buf, sizeof(buf), "%*f", INT_MIN, 9.1);
        |   ^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
  ```
  Ref: https://app.circleci.com/pipelines/github/curl/curl/10226/workflows/87642ee9-cda6-4916-8206-c82aac5f595e/jobs/107669?invite=true#step-106-40996_46

  The root cause of why this option gets enabled remains undiscovered.

Reported-by: Daniel Stenberg
  Fixes #15008
  Follow-up to 71cf0d1fca9e1f53524e1545ef0c08d174458d80 #14772

- build: drop `-Wno-format-overflow` from picky warning list.
  These options only get used with picky warnings enabled.
  Follow-up to 145f87b9e89f3a5e287233fe7d3cf57aca23dd8c #14598

- unit1652: suppress in source (and not rely on picky warnings anymore.)

Closes #15012

10 months agoCI: improve readability of Circle CI config
Viktor Szakats [Sun, 22 Sep 2024 23:10:37 +0000 (01:10 +0200)] 
CI: improve readability of Circle CI config

- move curl configure commands next to each other.
- re-order configure options to improve readability.
- fold long configure commands.

Closes #15011

10 months agoGHA/windows: mark 3023, 3024 flaky for mingw-w64 7.3.0 job
Viktor Szakats [Sun, 22 Sep 2024 18:15:51 +0000 (20:15 +0200)] 
GHA/windows: mark 3023, 3024 flaky for mingw-w64 7.3.0 job

They became flaky recently as reported by Testclutch and firsthand
observations. It's curious because it only seems to happen in 7.3.0,
but not in the very similar (except non-Unicode) 9.5.0 job.

We've found no explanation or reason for them so far.

```
test 3023...[HTTPS GET to localhost, first subject alt name matches, CN does not match (Schannel variant)]

 3023: protocol FAILED!
 There was no content at all in the file log/14/server.input.
 Server glitch? Total curl failure? Returned: 7
```
Ref: https://github.com/curl/curl/actions/runs/10871191391/job/30164710777?pr=14918#step:13:4849

```
test 3024...[HTTPS GET to localhost, last subject alt name matches, CN does not match (Schannel variant)]

 3024: protocol FAILED!
 There was no content at all in the file log/8/server.input.
 Server glitch? Total curl failure? Returned: 7
```
Ref: https://github.com/curl/curl/actions/runs/10871191391/job/30164710777?pr=14918#step:13:4713

```
FAIL 3023: 'HTTPS GET to localhost, first subject alt name matches, CN does not match (Schannel variant)' HTTPS, HTTP GET, PEM certificate
FAIL 3024: 'HTTPS GET to localhost, last subject alt name matches, CN does not match (Schannel variant)' HTTPS, HTTP GET, PEM certificate
```
https://github.com/curl/curl/actions/runs/10982991876/job/30491895264?pr=14930#step:14:4914
https://github.com/curl/curl/actions/runs/10871191391/job/30164710777?pr=14918
https://github.com/curl/curl/actions/runs/10841065505/job/30084573629?pr=14859#step:13:5484
https://github.com/curl/curl/actions/runs/10858001821/job/30135376138?pr=14906#step:13:4841

Reported-by: Testclutch
Fixes https://github.com/curl/curl/pull/14905#issuecomment-2350772804
Ref: https://github.com/curl/curl/discussions/14854#discussioncomment-10652044

Closes #15006

10 months agocmake: make `test-ci` target skip building dependencies
Viktor Szakats [Sun, 22 Sep 2024 10:01:36 +0000 (12:01 +0200)] 
cmake: make `test-ci` target skip building dependencies

Make `test-ci` not depend on the `testdeps` target.

`test-ci` is designed to run curl tests in CI. In CI we build all
necessary dependencies explicitly beforehand, and they are always ready
when calling the `test-ci` step. Thus, it isn't necessary to enforce
them via a dependency rule. Dropping it saves redundant work and delay
in CI jobs.

The `testdeps` dependency should not normally be a problem. It's
supposed to be a no-op if those targets are already built. In practice
however, it causes a delay and/or an actual rebuild of those
dependencies depending on generator (= build tool) used and other
factors.

As observed in the GHA/windows workflow, the `testdeps` dependency:

- with Ninja, causes no delay, and no extra work:
  https://github.com/curl/curl/actions/runs/10980099984/job/30485440389#step:25:18

- with GNU Make, caused a re-evaluation of `testdeps` targets,
  but did not actually rebuild them. This re-evaluation took a
  noticeable time (esp. with non-bundled tests):
  https://github.com/curl/curl/actions/runs/10980099984/job/30485440155#step:14:11 (with bundles)
  https://github.com/curl/curl/actions/runs/10973851013/job/30471690331#step:14:11 (w/o bundles)
  verbose: https://github.com/curl/curl/actions/runs/10980506956/job/30486434629#step:14:13

- with MSBuild, caused a re-evaluation of `testdeps` targets, and
  triggered a _rebuild_:
  https://github.com/curl/curl/actions/runs/10980099984/job/30485435968#step:14:19 (with bundles)
  https://github.com/curl/curl/actions/runs/10973851013/job/30471689714#step:14:19 (w/o bundles)
  verbose: https://github.com/curl/curl/actions/runs/10980506956/job/30486436368#step:14:48
  It's suspected that our use of
  `-DCMAKE_VS_GLOBALS=TrackFileAccess=false` in CI is contributing to
  this. This option is supposed to affect incremental builds only. For
  some reason it affects CI builds too, even though they are not
  incremental, and no sources are changed between build steps.
Reported-by: Aki Sakurai
  Ref: #14999

Notice that `test-*` targets depending on `testdeps` is NOT sufficient
to build everything to run tests, e.g. it misses to build the curl tool,
which is essential. This is also true for autotools' `test-ci` target
which misses to build libcurl and the curl tool.

Perhaps it'd be best to drop `testdeps` as a dependency for _all_
`test-*` targets and make it official to require building dependencies
manually. Alternatively these targets could be fixed to rebuild
everything necessary to run tests.

Closes #15001

10 months agoCI: add missed updates to the `configure-libssh` job
Viktor Szakats [Sun, 22 Sep 2024 23:02:05 +0000 (01:02 +0200)] 
CI: add missed updates to the `configure-libssh` job

Disable dependency tracking and enable unity + test bundles for
the `configure-libssh` job that was missed in earlier commits.

Follow-up to 71cf0d1fca9e1f53524e1545ef0c08d174458d80 #14772
Follow-up to dff66196d0bc52b53a91f59d3972769412f9639b #14975

Closes #15010

10 months agoRELEASE-NOTES: synced
Daniel Stenberg [Sun, 22 Sep 2024 19:35:44 +0000 (21:35 +0200)] 
RELEASE-NOTES: synced

bump the pending curl version to 8.11.0

10 months agotool_operate: let --create-dirs work for --dump-header as well
Daniel Stenberg [Thu, 19 Sep 2024 08:43:58 +0000 (10:43 +0200)] 
tool_operate: let --create-dirs work for --dump-header as well

Add test 3011 to verify

Proposed-by: Montg0mery on github
Fixes #14941
Closes #14965

10 months agotests: libtests and unit tests need explicit #include memdebug
Daniel Stenberg [Sun, 22 Sep 2024 18:36:04 +0000 (20:36 +0200)] 
tests: libtests and unit tests need explicit #include memdebug

... as otherwise they do not monitor the resource use within the test
files - until they are unity built.

Closes #15007

10 months agogtls: Add P12 format support
Tatsuhiro Tsujikawa [Fri, 20 Sep 2024 09:04:46 +0000 (18:04 +0900)] 
gtls: Add P12 format support

This change adds P12 format support for GnuTLS backend.

Closes #14991

10 months agochecksrc: fixed typo
Gabriel Marin [Thu, 19 Sep 2024 14:47:22 +0000 (17:47 +0300)] 
checksrc: fixed typo

Closes #14968

10 months agocmake: require quictls (or fork) when using msh3 on non-Windows
Viktor Szakats [Sun, 22 Sep 2024 13:59:14 +0000 (15:59 +0200)] 
cmake: require quictls (or fork) when using msh3 on non-Windows

Syncing behavior with `./configure`.

Closes #15003

10 months agoci: update rojopolis/spellcheck-github-actions digest to b83ca7c
renovate[bot] [Sun, 22 Sep 2024 16:18:06 +0000 (16:18 +0000)] 
ci: update rojopolis/spellcheck-github-actions digest to b83ca7c

Closes #15004

10 months agocurl: add options for safe/no CA bundle search (Windows)
Viktor Szakats [Sun, 18 Aug 2024 07:51:49 +0000 (09:51 +0200)] 
curl: add options for safe/no CA bundle search (Windows)

Add `CURL_CA_SEARCH_SAFE` build-time option to enable CA bundle search
in the `curl` tool directory. The lookup method was already used to find
`.curlrc` and `_curlrc` (on Windows). On Windows it overrides the unsafe
default `SearchPath()` method.

Enable with:
- cmake: `-DCURL_CA_SEARCH_SAFE=ON`
- autotools: `--enable-ca-search-safe`
- raw: `CPPFLAGS=-DCURL_CA_SEARCH_SAFE`

On Windows, before this patch the whole `PATH` was searched for
a CA bundle. `PATH` may contain unwanted or world-writable locations,
including the current directory. Searching them all is convenient to
pick up any CA bundle, but not secure.

The Muldersoft curl distro implements such CA search via a custom
patch for Windows:
https://github.com/lordmulder/cURL-build-win32/blob/cd652d4792c177c98b08b4309d3cac2b8dbbf9b0/patch/curl_tool_doswin.diff#L50

MSYS2/mingw-w64 distro has also been rolling a patch solving this:
https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-curl/0001-Make-cURL-relocatable.patch
https://github.com/msys2/MINGW-packages/blob/master/mingw-w64-curl/pathtools.c

Also add option to fully disable Windows CA search:
- cmake: `-DCURL_DISABLE_CA_SEARCH=ON`
- autotools: `--disable-ca-search`
- raw: `CPPFLAGS=-DCURL_DISABLE_CA_SEARCH`.

Both options are considered EXPERIMENTAL, with possible incompatible
changes or even (partial) removal in the future, depending on feedback.

An alternative, secure option is to embed the CA bundle into the binary.

Safe search can be extended to other platforms if necessary or useful,
by using `_NSGetExecutablePath()` (macOS),
`/proc/self/exe` (Linux/Cygwin), or `argv[0]`.

Closes #14582

10 months agobuild: drop exclamation marks, use "msh3/msquic" where incomplete
Viktor Szakats [Sun, 22 Sep 2024 13:21:02 +0000 (15:21 +0200)] 
build: drop exclamation marks, use "msh3/msquic" where incomplete

Closes #15002

10 months agoGHA/windows: formatting
Viktor Szakats [Sun, 22 Sep 2024 09:37:42 +0000 (11:37 +0200)] 
GHA/windows: formatting

Follow-up to 1b8449674adb57ee0f60e761d654c69b20ee8fcf #14992

10 months agoGHA: use more ninja, build examples in the last step, and more
Viktor Szakats [Sat, 21 Sep 2024 01:37:54 +0000 (03:37 +0200)] 
GHA: use more ninja, build examples in the last step, and more

- linux: bump up test parallelism for valgrind tests to `-j4`
  (from `-j2`). (EXPERIMENTAL)
- linux: drop `apt-get update` for the default architecture on the GHA
  native runner. It makes prereq install steps complete faster.
  The runner image gets weekly updates, and that should be enough to
  guarantee fresh packages in most cases:
  https://github.com/actions/runner-images/commits/main/images/ubuntu/Ubuntu2204-Readme.md
- aws-lc: use ninja with cmake.
- aws-lc: build examples with cmake.
- aws-lc: drop `apt update`.
- aws-lc, wolfssl, linux32, http3-linux: move building examples to
  the last step.
  Follow-up to 45202cbba4bb3d12b4469063864b57d2f8765d9c #14906
- windows: formatting.

Closes #14992

10 months agoGHA: revert some build test steps added by #14772
Viktor Szakats [Sat, 21 Sep 2024 14:01:44 +0000 (16:01 +0200)] 
GHA: revert some build test steps added by #14772

They are still slow in these jobs/combinations.

- non-native/FreeBSD/arm64 autotools +36s
- non-native/FreeBSD/arm64 cmake +1m
- windows/linux-cross-mingw-w64 autotools +33s

These ones remain:
- linux/aws-lc cmake +6s
- windows/linux-cross-mingw-w64 cmake +12s

Follow-up to 71cf0d1fca9e1f53524e1545ef0c08d174458d80 #14772

10 months agotests: speed up builds with single-binary test bundles
Viktor Szakats [Mon, 2 Sep 2024 12:46:00 +0000 (14:46 +0200)] 
tests: speed up builds with single-binary test bundles

Add support for single-block binaries that contain all libtests and
unit tests respectively.

Enable with:
- autotools: `--enable-test-bundles`
- cmake: `-DCURL_TEST_BUNDLES=ON`

(They are compatible with `--enable-unity` and `-DCMAKE_UNITY_BUILD=ON`
options, for further speed-up.)

Makes libtests and unit tests build _fast_, needing little disk space
even in static mode. Similar to CMake unity mode, but with a custom
script, also supporting autotools builds.

The price is having to deal with symbols/macros colliding between
`lib*.c` and `unit*.c` sources. Maybe with naming conventions or other
solutions this can be improved gradually and reduce the need for manual
intervention by `mk-bundle.mk`. I've included a script that does the bulk
of detecting name collisions.

Also:
- CI: enable test bundles.
- CI: build tests in more jobs.
- lib2305: fix FILE handle leak.
- unit1661: fix memleak found by torture test by releasing the `bufref`
  structure in `unit_stop()` that was allocated in `unit_setup()`.
  ```
  test 1661...[bufref unit tests]
  Leak detected: memory still allocated: 13 bytes
   allocated by /home/runner/work/curl/curl/tests/unit/unit1661.c:70
   1661: torture FAILED: function number 1 in test.
  ```
  Ref: https://github.com/curl/curl/actions/runs/10967279334/job/30456745290?pr=14772#step:8:41

Similar test suite builds with autotools default and cmake+bundle+unity:
- GHA/Linux: 33s vs 7s
  https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
- GHA/macOS 34s vs 2s
  https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
- GHA/FreeBSD: 15m25 vs 6m21 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
- GHA/Cygwin: 9m52 vs 32s
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 3m52 vs 14s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 5m45 vs 30s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681628787

Autotools test suite builds compared between master -> `--enable-test-bundles`:
- GHA/Linux: 33s -> 9s (run tests: 22m23 -> 20m44)
  https://github.com/curl/curl/actions/runs/10710030193/job/29695932185
  https://github.com/curl/curl/actions/runs/10967831456/job/30458220344
- GHA/macOS: 25s -> 4s (run tests: 2m58 -> 2m24)
  https://github.com/curl/curl/actions/runs/10710030195/job/29695938444
  https://github.com/curl/curl/actions/runs/10967831452/job/30458225762
- GHA/non-native (FreeBSD): 4m8 -> 3m12 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10710030198/job/29695928401
  https://github.com/curl/curl/actions/runs/10967831458/job/30458212692
- GHA/Cygwin: 9m25 -> 1m9 (run tests: 9m19 -> 3m28)
  https://github.com/curl/curl/actions/runs/10710030212/job/29695928213
  https://github.com/curl/curl/actions/runs/10967831453/job/30458213268
- GHA/MSYS2: 3m54 -> 32s (run tests: 6m3 -> 3m59)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704850591
  https://github.com/curl/curl/actions/runs/10967831449/job/30459280005
- GHA/mingw-w64: 5m42 -> 1m5 (run tests: 7m41 -> 5m36)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704852058
  https://github.com/curl/curl/actions/runs/10967831449/job/30459280862
- Azure MSYS2 mingw64 openssl: 38m55 -> 11m58
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=b58b8c59-0f61-52e9-0f9e-fad562a1e77f&t=0f9230a7-3b10-53ca-9938-700ece377c5e
- Azure Ubuntu default: 2m15 -> 55s (all build)
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25546&view=logs&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7
  https://dev.azure.com/daniel0244/curl/_build/results?buildId=25547&view=logs&jobId=39473db1-3945-55d5-deb5-c218fad88dce&j=9d58b9ac-e1e6-53b6-f83a-1f9f1d912522&t=a6b38d83-e7cf-5a9b-c762-a178412717b7

Cmake test suite builds compared between master -> `-DCURL_TEST_BUNDLES=ON` + unity:
- GHA/Linux: 29s -> 7s (run tests: 4m50 -> 4m57, 20m43 -> 20m45)
  https://github.com/curl/curl/actions/runs/10710030193/job/29695941814
  https://github.com/curl/curl/actions/runs/10705668823/job/29681622201
- GHA/Linux old: 44s -> 13s (bundle+no unity) (run tests: 5m5 -> 5m6)
  https://github.com/curl/curl/actions/runs/10718264094/job/29719794727
  https://github.com/curl/curl/actions/runs/10718653175/job/29721009613
- GHA/macOS: 32s -> 2s (run tests: 2m43 -> 2m40)
  https://github.com/curl/curl/actions/runs/10710030195/job/29695931956
  https://github.com/curl/curl/actions/runs/10705668813/job/29681638937
- GHA/non-native (*BSD): inconclusive (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10710030198
  https://github.com/curl/curl/actions/runs/10705668811
- GHA/Cygwin: 3m9 -> 32s
  https://github.com/curl/curl/actions/runs/10710030212/job/29695929075
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
- GHA/MSYS2: 2m24 -> 14s
  https://github.com/curl/curl/actions/runs/10710030190/job/29704850996
  https://github.com/curl/curl/actions/runs/10705668808/job/29681624295
- GHA/mingw-w64: 3m56 -> 30s (run tests: 4m2 -> 3m52)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704852219
  https://github.com/curl/curl/actions/runs/10705668808/job/29681631393
- GHA/mingw-w64-old: 7m19 -> 1m44 (run tests: 3m30 -> 2m53)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704849763
  https://github.com/curl/curl/actions/runs/10705668808/job/29681622329
- GHA/MSVC: 3m22 -> 13s (run tests: 9m43 -> 4m22)
  https://github.com/curl/curl/actions/runs/10710030190/job/29704850411
  https://github.com/curl/curl/actions/runs/10705668808/job/29681623313
- AppVeyor CI MSVC 2008: 4m3 -> 45s (full build)
- AppVeyor CI MSVC 2010: 2m56 -> 1m8 (full build)
- AppVeyor CI MSVC 2022: 10m19 -> 2m23 (full build)
  https://ci.appveyor.com/project/curlorg/curl/builds/50538455
  https://ci.appveyor.com/project/curlorg/curl/builds/50536558
- AppVeyor CI total build time: 10m30 (master) -> 6m48 (unity) -> 4m5 (bundle) -> 3m24 (bundle+unity) -> 5m7 (bundle+unity+all jobs building tests)

Closes #14772

10 months agocmake: tidy up
Viktor Szakats [Mon, 26 Aug 2024 09:38:17 +0000 (11:38 +0200)] 
cmake: tidy up

- unroll two short loops.
- unfold lines.
- merge lines with their comments.
- add missing quotes.
- tidy up grammar in error/warning messages.

Cherry-picked from #14692
Closes #14998

10 months agotidy-up: indent, whitespace, `#error` in make files
Viktor Szakats [Mon, 26 Aug 2024 11:32:18 +0000 (13:32 +0200)] 
tidy-up: indent, whitespace, `#error` in make files

Replace invalid C with `#error`.

Cherry-picked from #14692
Closes #14997

10 months agotidy-up: spelling
Viktor Szakats [Tue, 10 Sep 2024 08:20:47 +0000 (10:20 +0200)] 
tidy-up: spelling

C89, Schannel, Secure Transport, contractions.

Cherry-picked from #14692
Closes #14996

10 months agotidy-up: indent, whitespace, comment in sources
Viktor Szakats [Mon, 26 Aug 2024 11:25:34 +0000 (13:25 +0200)] 
tidy-up: indent, whitespace, comment in sources

Cherry-picked from #14692
Closes #14995

10 months agoRELEASE-NOTES: synced
Daniel Stenberg [Sat, 21 Sep 2024 15:57:39 +0000 (17:57 +0200)] 
RELEASE-NOTES: synced

10 months agobuild: use `configurehelp.pm.in` with autotools and cmake
Viktor Szakats [Fri, 6 Sep 2024 23:49:40 +0000 (01:49 +0200)] 
build: use `configurehelp.pm.in` with autotools and cmake

Before this patch, each build tool generated `tests/configurehelp.pm`
manually.

Ref: https://github.com/curl/curl/pull/14802#issuecomment-2332734326
Closes #14819

10 months agobuild: tidy up and improve versioned-symbols options
Viktor Szakats [Fri, 6 Sep 2024 22:32:01 +0000 (00:32 +0200)] 
build: tidy up and improve versioned-symbols options

- autotools: add support for custom prefix.
  Usage: `--enable-versioned-symbols=MYPREFIX_`
  Catching up with cmake.

- add default versioned prefix for Rustls builds.

- delete `HIDDEN` entry from version script `lib/libcurl.vers.in`.
  It referred to symbol names that never existed in libcurl.

- cmake: use `lib/libcurl.vers.in` as a template instead of generating
  it from scratch. Adapt `./configure` accordingly, and add comments
  to keep hard-wired soname in sync with `lib/Makefile.soname`.

- autotools: delete Schannel and Secure Transport version prefixes.
  Windows and macOS don't support the versioned symbols option.

Follow-up to 7b1444979094a365c82c665cce0e2ebc6b69467b #14378
Closes #14818

10 months agoconfigure: catch Apple in more target triplets
Viktor Szakats [Thu, 29 Aug 2024 15:05:43 +0000 (17:05 +0200)] 
configure: catch Apple in more target triplets

Before this patch, only these triplets were considered Apple:
`<cpu>-apple-darwin`

After this patch, these are also considered Apple:
`<cpu>-apple-(ios*|tvos*|visionos*|watchos*|<ETC>)`

`$host_os` (the last third of the triplet) still has a valid use
to differentiate between OS flavours, though for now this isn't
used, aligning with CMake.

Closes #14728

10 months agoGHA/configure-vs-cmake: check `libcurl.pc`/`curl-config`, fix issues
Viktor Szakats [Sun, 25 Aug 2024 19:40:13 +0000 (21:40 +0200)] 
GHA/configure-vs-cmake: check `libcurl.pc`/`curl-config`, fix issues

Add CI checker to compare `libcurl.pc` and `curl-config` files
generated by autotools and cmake builds.

Fix differences and apply tiny cleanups:
- curl-config: use single-quotes for literals.
- curl-config: quote all variables.
- curl-config: replace double with single quotes in a substituted value
  that's always literal (`@prefix@`).
- libcurl.pc: spelling in `Description:`.
- libcurl.pc: avoid substitution in a comment.
- cmake: fill `libdir` with `${exec_prefix}` instead of a literal.
  To sync with './configure'.
- configure: fix `CURL_CA_BUNDLE` value to not generate nested quotes
  in `curl-config`.
- configure: add missing `LDFLAGS` to `Libs.private` in `libcurl.pc`.
  To sync with CMake.
- cmake: skip adding `CMAKE_C_IMPLICIT_LINK_LIBRARIES` for MINGW and
  UNIX. They added these values as seen in CI:
  MINGW: `-lmingw32 -lgcc -lmoldname -lmingwex -ladvapi32 -lshell32 -luser32 -lkernel32 -lmingw32 -lgcc -lmoldname -lmingwex`
  Linux: `-lgcc -lgcc_s -lc -lgcc -lgcc_s`
- cmake: delete FIXME about enabling libssh2 by default.
  `./configure` has special defaults for these packages (called: "off"):
  brotli, zstd, libpsl, libssh2, libssh, wolfssl, librtmp
  It looks for them, but only at system locations, which makes them
  never detected e.g. on macOS. CMake doesn't offer such default mode
  for now.
- GHA/macos: drop now redundant `-DCURL_DISABLE_LDAPS=ON`.
- cmake: use `CMAKE_INSTALL_INCLUDEDIR` and `CMAKE_INSTALL_LIBDIR`
  instead of hardcoded `include`/`lib` when generating `libcurl.pc`.

Updates to the GHA workflow:
- move autotools out-of-tree and rename cmake out-of-tree directory
  to `bld_cm` to tell it's cmake.
- disable static libcurl for `./configure` to match cmake.
- enable `pkg-config` debug output with `./configure`.
- dump list of Homebrew packages on macOS.
- dump `./configure` detailed logs.
- disable zstd and brotli for Linux, to match cmake.

There remain differences, mostly due to detection order and method. Also
some values are inherently different when using CMake and autotools,
such as `--cc`, `--configure`. autotools also generates duplicates for
`-lssl` and `-lcrypto`. macOS LDAP wants to link `-lber` while autotools
doesn't. Some build defaults are also different in autotools and cmake.
These differences are smoothened out for now by the checker script, or
via build options. Notice that lib order (a dupes) _can_ be significant
in some cases. E.g. the binutils linker is infamous for that on Windows.

Closes #14681

10 months agoipfs: add options to disable
Viktor Szakats [Sun, 8 Sep 2024 10:26:45 +0000 (12:26 +0200)] 
ipfs: add options to disable

- CPPFLAGS: `-DCURL_DISABLE_IPFS`
- configure: `--disable-ipfs`
- cmake: `-DCURL_DISABLE_IPFS=ON`

Fixes #14824
Closes #14827

10 months agosrc: tidy-up conditions for CA bundle search
Viktor Szakats [Tue, 10 Sep 2024 01:23:33 +0000 (03:23 +0200)] 
src: tidy-up conditions for CA bundle search

- delete redundant Schannel check.

- move `feature_ssl` check one level up from `FindWin32CACert()`.

- check `feature_ssl` early to skip a bunch of CA bundle search logic
  for no-ssl configurations.

Reviewed-by: Jay Satiro
Closes #14841

10 months agoCI: enable RTMP and WebSockets in old Linux build
Dan Fandrich [Sat, 21 Sep 2024 07:27:20 +0000 (00:27 -0700)] 
CI: enable RTMP and WebSockets in old Linux build

Try to enable as much as possible to check for compatibility with old
dependencies.

10 months agolib: memdebug comment fixup [ci skip]
Viktor Szakats [Fri, 20 Sep 2024 23:27:22 +0000 (01:27 +0200)] 
lib: memdebug comment fixup [ci skip]

Follow-up to 3efba94f773db5d8ae19e33aa749ab7914cafeea #14765

10 months agoGHA/linux: disable unity build for fix scanbuild job
Viktor Szakats [Fri, 20 Sep 2024 22:31:04 +0000 (00:31 +0200)] 
GHA/linux: disable unity build for fix scanbuild job

Unity mode seems to defeat the scanner and miss issues.

before, miss: https://github.com/curl/curl/actions/runs/10967056702/job/30456136390
after, OK: https://github.com/curl/curl/actions/runs/10967128744/job/30456330732#step:35:1232

Tested with PR commit:
https://github.com/curl/curl/pull/14880/commits/32854bb30861e757d4f8d61d303c1b1f0e55bd26

Follow-up to 60c3d0446546332e1645541f2dc2c072cd019fe9 #14815
Closes #14987

10 months agocmake: fix broken dependency chain for cmdline-opts, tidy-ups
Viktor Szakats [Thu, 12 Sep 2024 13:16:51 +0000 (15:16 +0200)] 
cmake: fix broken dependency chain for cmdline-opts, tidy-ups

- make `curl.1` and `curl.txt` depend on `DPAGES`.
  To trigger a rebuild when an individual manpage is updated.

- tell CMake that the cmdline-opts command also creates `curl.txt`.

- make `tool_hugehelp.c` depend on `curl.txt` (was: `curl.1`), to match
  what it actually uses for input.

- stop using `generate-curl.1` as an indirect way to create `curl.txt`
  in time for `tool_hugehelp.c`. After the fixes above there is a direct
  depedency chain between them.

- move `ASCIIPAGE` and `MANPAGE` variables to top-level, re-use them in
  `src` and prefix them with `CURL_` to avoid clashing with other
  projects.

- drop double quotes from `generate-curl.1` as a hint that it is not
  a filename, but a target name.

- src: tidy up order of dependency lists.

Closes #14883

10 months agobuild: tidy up deprecation suppression, enable warnings for clang
Viktor Szakats [Thu, 5 Sep 2024 00:11:35 +0000 (02:11 +0200)] 
build: tidy up deprecation suppression, enable warnings for clang

Suppress deprecation warnings the closest to the deprecated code, using
`CURL_IGNORE_DEPRECATION()`. Then drop build-specific suppressions, and
file-wide ones. The latter is not compatible with Unity mode. Also
replace manual suppressions with a macro to apply to all compilers with
deprecation warning support. Also enable deprecation warnings for clang.

- curl/curl.h: enable deprecation warnings for clang.

- docs/examples: stop setting `CURL_DISABLE_DEPRECATION` with autotools.
  Suppression moved to C-level earlier. Syncs with cmake.
  Follow-up to 5fc61a37c1b177cefbc11dc9f5eef7b2a14538da #14123

- tests/http/clients: stop setting `CURL_DISABLE_DEPRECATION` in
  autotools. If it becomes necessary in the future, it can be done in
  C via the macro. Syncs with cmake.

- lib1545: stop setting `CURL_DISABLE_DEPRECATION` in autotools.
  Drop guard from test source.
  Follow-up to 0f103600731f7e08637017ef4df67a194e0d6711 #12444

- libtest, unit: replace `CURL_DISABLE_DEPRECATION` with
  `CURL_IGNORE_DEPRECATION()`.

- docs/examples: replace pragmas with `CURL_IGNORE_DEPRECATION()`.

Closes #14789

10 months agocmake: expand `CURL_USE_PKGCONFIG` to non-cross `MINGW`
Viktor Szakats [Tue, 20 Aug 2024 09:13:19 +0000 (11:13 +0200)] 
cmake: expand `CURL_USE_PKGCONFIG` to non-cross `MINGW`

Enable `CURL_USE_PKGCONFIG` by default for more environments:

- for `MINGW` targets when not using cross-compilation.
- stop restricting vcpkg to MSVC. (this currently unlocks mingw,
  also unlocked by the update above.)

Also:
- cache `CURL_USE_PKGCONFIG` in `CURLConfig.cmake`.
Suggested-by: Kai Pastor
Follow-up to c555ab469d74756b0e3a21f797237d53f9334ce3 #14575
Closes #14658

10 months agoGHA: speed up builds in torture jobs, tidy up
Viktor Szakats [Thu, 19 Sep 2024 20:38:04 +0000 (22:38 +0200)] 
GHA: speed up builds in torture jobs, tidy up

- use cmake with ninja.
- drop unnecessary packages.

Also:
- Linux: switch to Ubuntu 20.24 runner. Initially for newer packages,
  then left it there for variation and as canary.
- Linux: make it easier to enable valgrind by dropping `-n` tflags.
- Linux: show `curl -V` after build.
- Linux: sync job names with macOS.
- Linux, macOS: review and adjust job timeouts.
- Linux, macOS: sync job configs more:
  Linux: enable libssh2, macOS: enable brotli, zstd, libssh2.
- macOS: prefer `source` (over manual `PATH` editing) for `venv` setup.
- macOS: drop redundant `-DENABLE_CURLDEBUG=ON`.
- macOS: drop redundant `-n` tflags. There is no valgrind on macOS.
- macOS: allow overriding test parallelism per job.
- macOS: unlock all disabled tests for torture jobs.

Speed-ups (configure/build stage):
- Linux !FTP: 83s -> 36s
  before: https://github.com/curl/curl/actions/runs/10948030198/job/30398134647
  after: https://github.com/curl/curl/actions/runs/10961987675/job/30440179978?pr=14972
- Linux FTP: 79s -> 32s
  before: https://github.com/curl/curl/actions/runs/10948030198/job/30398137070
  after: https://github.com/curl/curl/actions/runs/10961987675/job/30440182406?pr=14972
- macOS !FTP: 98s -> 29s
  before: https://github.com/curl/curl/actions/runs/10960141275/job/30434081412?pr=14972
  after: https://github.com/curl/curl/actions/runs/10960141275/job/30434082357?pr=14972
- macOS FTP: 129s -> 29s
  before: https://github.com/curl/curl/actions/runs/10960141275/job/30434082041?pr=14972
  after: https://github.com/curl/curl/actions/runs/10960141275/job/30434083050?pr=14972

Closes #14972

10 months agoautotools: add support for 'unity' builds, enable in CI
Viktor Szakats [Fri, 6 Sep 2024 08:26:06 +0000 (10:26 +0200)] 
autotools: add support for 'unity' builds, enable in CI

Implement the "unity" builds as known from CMake, but for autotools.
It's limited to `lib` and `src` (CMake also supports it in `tests`).

Enable with: `--enable-unity` (disabled by default)

Unity builds speed up builds significantly. Cygwin and Windows builds in
particular, but the effect is noticeable on most systems. It also allows
discovering unity issues with autotools, benefitting also CMake when
building the same combination. In CI it makes turnaround times quicker.

This closes build performance with CMake. autotools still lags behind
because it builds shared and static libcurl in two, separate passes.
CMake does it in one. Manpage compilation isn't batched, it is in CMake.
After unity and test bundle support the slowest parts of the build are
the configuration phase (which is effectively a tedious, non-parallel,
compilation and/or linking of 300+ tiny programs. The next bottleneck
is compiling individual examples and finally test servers (only slow
with autotools).

The autotools implementation is slightly less efficient than CMake,
because 3 sources are permanently excluded while in CMake this isn't
necessary and solved more efficiently while building libtests. There is
also no 'unity' support for tests, making them a less efficient also.

Enable it in CI for most `configure` jobs. Except in GHA/dist (though
it works fine there too), to use the default config there. Also skip for
the Linux AWC-LC job where it made builds time a few seconds longer
(reason undiscovered.)

Autotools test suite builds compared between master -> `--enable-unity`:
- GHA/Linux: 32s -> 12s
  https://github.com/curl/curl/actions/runs/10705668823/job/29681617374
  https://github.com/curl/curl/actions/runs/10742978889/job/29796766297
- GHA/macOS: 37s -> 10s
  https://github.com/curl/curl/actions/runs/10705668813/job/29681632885
  https://github.com/curl/curl/actions/runs/10742978699/job/29796768875
- GHA/FreeBSD: 15m25 -> 10m58 (full workflow time, ~qemu)
  https://github.com/curl/curl/actions/runs/10705668811/job/29681607915
  https://github.com/curl/curl/actions/runs/10742978937/job/29796766115
- GHA/Cygwin: 3m32 -> 1m21
  https://github.com/curl/curl/actions/runs/10705668809/job/29681609965
  https://github.com/curl/curl/actions/runs/10742978645/job/29796756933
- GHA/MSYS2: 2m42 -> 50s
  https://github.com/curl/curl/actions/runs/10705668808/job/29681621166
  https://github.com/curl/curl/actions/runs/10742978662/job/29799739289
- GHA/mingw-w64: 5m32 -> 1m23
  https://github.com/curl/curl/actions/runs/10705668808/job/29681628787
  https://github.com/curl/curl/actions/runs/10742978662/job/29799741568

Closes #14815

10 months agocmake: separate target for examples, optimize CI, fix fallouts
Viktor Szakats [Sat, 14 Sep 2024 00:21:33 +0000 (02:21 +0200)] 
cmake: separate target for examples, optimize CI, fix fallouts

- Move `docs/examples` builds under a separate target.

- Make `BUILD_EXAMPLES` default to `ON`. It means to generate the rules
  for `docs/examples` by default, but not build them. To build them,
  an explicit `make curl-examples` (or ninja, etc) command is necessary.
  This syncs behaviour with autotools, and also how both cmake and
  autotools are building tests.

- GHA: update cmake jobs to use the new way of building examples.

- GHA: move examples build step at the end of the job, after building
  and running tests. This allows to have build and test run results
  faster, and leave the seldom-changing examples build to the end.
  Building examples is the slowest build step with no practical way to
  make them fast.

- appveyor: enable building examples in two old-MSVC jobs.

- examples: fix examples to build cleanly with old MSVC versions.

- GHA/non-native: move example build log under a GHA foldable section.

- GHA/windows: move building examples into separate step for Linux cross
  jobs.

Follow-up to dfdd978f7c60224dffe2aac25b436dc0a5cd0186 #13491
Closes #14906

10 months agoruntests: log output improvements
Viktor Szakats [Fri, 20 Sep 2024 00:54:38 +0000 (02:54 +0200)] 
runtests: log output improvements

- show invocation arguments (tflags) in the log.
- show `curl -V` protocols in the log.

Closes #14985

10 months agorequest: on shutdown send, proceed normally on timeout
Stefan Eissing [Tue, 10 Sep 2024 12:08:17 +0000 (14:08 +0200)] 
request: on shutdown send, proceed normally on timeout

When ending an FTP upload, we shut down the connection gracefully, since
the server should be notified we had send all bytes. Mostly, this is a
NOP without TLS involved. With TLS, close-notify messages should be
exchanged.

As reported in #14843, not all servers seem to do that. Since it is the
server's responsiblity to check it has received everything, we just log
the timeout and proceed as if everything is fine.

In the receive direction, we still fail the transfer if the server does
not shut down its direction properly.

Fixes #14843
Reported-by: Rasmus Melchior Jacobsen
Closes #14848

10 months agoalt-svc: honor data->state.httpwant
Stefan Eissing [Thu, 19 Sep 2024 10:23:55 +0000 (12:23 +0200)] 
alt-svc: honor data->state.httpwant

When a transfer is set for a speficif HTTP version, only lookup that
protocol in the alt-svc mappings. When no speicific version is
requested, scan all entries as before.

Closes #14966

10 months agourl: connection reuse on h3 connections
Stefan Eissing [Thu, 19 Sep 2024 09:47:29 +0000 (11:47 +0200)] 
url: connection reuse on h3 connections

- When searching for existing connections, interpret the
  default CURL_HTTP_VERSION_2TLS as "anything goes". This
  will allow us to reuse HTTP/3 connections better
- add 'http/1.1' as allowed protocol identifier in Alt-Svc
  files
- add test_02_0[345] for testing protocol selection on
  provided alt-svc files

Fixes #14890
Reported-by: MacKenzie
Closes #14966

10 months agotests: remove all valgrind disble instructions
Daniel Stenberg [Fri, 20 Sep 2024 14:28:27 +0000 (16:28 +0200)] 
tests: remove all valgrind disble instructions

Closes #14983

10 months agolibssh2: use the Curl_* memory functions to avoid memdebug
Daniel Stenberg [Fri, 20 Sep 2024 15:27:26 +0000 (17:27 +0200)] 
libssh2: use the Curl_* memory functions to avoid memdebug

This prevents our torture tests from detecting and getting trapped by
memory leaks in libssh2.

Closes #14984

10 months agolibssh.c: handle EGAINS during proto-connect correctly
Stefan Eissing [Fri, 20 Sep 2024 14:06:25 +0000 (16:06 +0200)] 
libssh.c: handle EGAINS during proto-connect correctly

The implementation did not check the poll flags when a ssh_connect()
EAGAINed. Also the poll check did not allow for both directions
to be signalled by libssh, which it does.

This appeared during CI testing with higher paralellism.

Closes #14982