Daniel Stenberg [Wed, 5 Oct 2022 22:49:10 +0000 (00:49 +0200)]
strcase: add and use Curl_timestrcmp
This is a strcmp() alternative function for comparing "secrets",
designed to take the same time no matter the content to not leak
match/non-match info to observers based on how fast it is.
The time this function takes is only a function of the shortest input
string.
Daniel Stenberg [Mon, 3 Oct 2022 15:40:02 +0000 (17:40 +0200)]
websockets: remodeled API to support 63 bit frame sizes
curl_ws_recv() now receives data to fill up the provided buffer, but can
return a partial fragment. The function now also get a pointer to a
curl_ws_frame struct with metadata that also mentions the offset and
total size of the fragment (of which you might be receiving a smaller
piece). This way, large incoming fragments will be "streamed" to the
application. When the curl_ws_frame struct field 'bytesleft' is 0, the
final fragment piece has been delivered.
curl_ws_recv() was also adjusted to work with a buffer size smaller than
the fragment size. (Possibly needless to say as the fragment size can
now be 63 bit large).
curl_ws_send() now supports sending a piece of a fragment, in a
streaming manner, in addition to sending the entire fragment in a single
call if it is small enough. To send a huge fragment, curl_ws_send() can
be used to send it in many small calls by first telling libcurl about
the total expected fragment size, and then send the payload in N number
of separate invokes and libcurl will stream those over the wire.
The struct curl_ws_meta() returns is now called 'curl_ws_frame' and it
has been extended with two new fields: *offset* and *bytesleft*. To help
describe the passed on data chunk when a fragment is delivered in many
smaller pieces.
docs/examples: avoid deprecated options in examples where possible
Example programs targeting a deprecated feature/option are commented with
a warning about it.
Other examples are adapted to not use deprecated options.
Viktor Szakats [Thu, 6 Oct 2022 15:30:13 +0000 (15:30 +0000)]
tidy-up: delete parallel/unused feature flags
Detecting headers and lib separately makes sense when headers come in
variations or with extra ones, but this wasn't the case here. These were
duplicate/parallel macros that we had to keep in sync with each other
for a working build. This patch leaves a single macro for each of these
dependencies:
- Rely on `HAVE_LIBZ`, delete parallel `HAVE_ZLIB_H`.
Also delete CMake logic making sure these two were in sync, along with
a toggle to turn off that logic, called `CURL_SPECIAL_LIBZ`.
Also delete stray `HAVE_ZLIB` defines.
There is also a `USE_ZLIB` variant in `lib/config-dos.h`. This patch
retains it for compatibility and deprecates it.
- Rely on `USE_LIBSSH2`, delete parallel `HAVE_LIBSSH2_H`.
Also delete `LIBSSH2_WIN32`, `LIBSSH2_LIBRARY` from
`winbuild/MakefileBuild.vc`, these have a role when building libssh2
itself. And `CURL_USE_LIBSSH`, which had no use at all.
Also delete stray `HAVE_LIBSSH2` defines.
- Rely on `USE_LIBSSH`, delete parallel `HAVE_LIBSSH_LIBSSH_H`.
Also delete `LIBSSH_WIN32`, `LIBSSH_LIBRARY` and `HAVE_LIBSSH` from
`winbuild/MakefileBuild.vc`, these were the result of copy-pasting the
libssh2 line, and were not having any use.
- Delete unused `HAVE_LIBPSL_H` and `HAVE_LIBPSL`.
Daniel Stenberg [Wed, 5 Oct 2022 22:52:35 +0000 (00:52 +0200)]
CURLOPT_COOKIEFILE: insist on "" for enable-without-file
The former way that also suggested using a non-existing file to just
enable the cookie engine could lead to developers maybe a bit carelessly
guessing a file name that will not exist, and then in a future due to
circumstances, such a file could be made to exist and then accidentally
libcurl would read cookies not actually meant to.
Daniel Stenberg [Thu, 6 Oct 2022 07:07:03 +0000 (09:07 +0200)]
tests/Makefile: remove run time stats from ci-test
The ci-test is the normal makefile target invoked in CI jobs. This has
been using the -r option to runtests.pl since a long time, but I find
that it mostly just adds many lines to the test output report without
anyone caring much about those stats.
tool: avoid generating ambiguous escaped characters in --libcurl
C string hexadecimal-escaped characters may have more than 2 digits.
This results in a wrong C compiler interpretation of a 2-digit escaped
character when followed by an hex digit character.
The solution retained here is to represent such characters as 3-digit
octal escapes.
Exclude linker flags specifying depedency libs and libpaths, when
building against `libcurl.dll`. In such case these options are not
necessary (but may cause errors if not/wrongly configured.)
Also move and reword a comment on `CPPFLAGS` to not apply to
`UNICODE` options. These are necessary for all build targets.
Jay Satiro [Wed, 5 Oct 2022 07:33:39 +0000 (03:33 -0400)]
runtests: fix uninitialized value on ignored tests
- Don't show TESTFAIL message (ie tests failed which aren't ignored) if
only ignored tests failed.
Before:
IGNORED: failed tests: 571 612 1056
TESTDONE: 1214 tests out of 1217 reported OK: 99%
Use of uninitialized value $failed in concatenation (.) or string at
./runtests.pl line 6290.
TESTFAIL: These test cases failed:
After:
IGNORED: failed tests: 571 612 1056
TESTDONE: 1214 tests out of 1217 reported OK: 99%
Jay Satiro [Sun, 2 Oct 2022 21:21:28 +0000 (17:21 -0400)]
cirrus: use make LDFLAGS=-all-static instead of curl_LDFLAGS
- Correct the use of -all-static for static Windows CI builds.
curl_LDFLAGS was removed from the makefile when metalink support was
removed. LDFLAGS=-all-static is passed to make only, because it is not a
valid option for configure compilation tests.
Viktor Szakats [Tue, 4 Oct 2022 19:14:38 +0000 (19:14 +0000)]
Makefile.m32: fix regression with tool_hugehelp [ci skip]
In a recent commit I mistakenly deleted this logic, after seeing a
reference to a filename ending with `.cvs` and thinking it must have
been long gone. Turns out this is an existing file. Restore the rule
and the necessary `COPY` definitions with it.
The restored logic is required for a successful build on a bare source
tree (as opposed to a source release tarball).
Also shorten an existing condition similar to the one added in this
patch.
Viktor Szakats [Tue, 4 Oct 2022 17:09:51 +0000 (17:09 +0000)]
Makefile.m32: deduplicate build rules [ci skip]
After this patch, we reduce the three copies of most `Makefile.m32`
logic to one. This now resides in `lib/Makefile.m32`. It makes future
updates easier, the code shorter, with a small amount of added
complexity.
`Makefile.m32` reduction:
| | bytes | LOC total | blank | comment | code |
|-------------------|-------:|----------:|-------:|---------:|------:|
| 7.85.0 | 34772 | 1337 | 79 | 192 | 1066 |
| before this patch | 17601 | 625 | 62 | 106 | 457 |
| after this patch | 11680 | 392 | 52 | 104 | 236 |
Details:
- Change rules to create objects for the `v*` subdirs in the `lib` dir.
This allows to use a shared compile rule and assumes that filenames
are not (and will not be) colliding across these directories.
`Makefile.m32` now also stores a list of these subdirs. They are
changing rarely though.
- Sync as much as possible between the three `Makefile.m32` scripts'
rules and their source/target sections.
- After this patch `CPPFLAGS` are all applied to the `src` sources once
again. This matches the behaviour of cmake/autotools. Only zlib ones
are actually required there.
- Use `.rc` names from `Makefile.inc` instead of keeping a duplicate.
- Change examples to link `libcurl.dll` by default. This makes building
trivial, even as a cross-build:
`CC=x86_64-w64-mingw32-gcc make -f Makefile.m32`
To run them, you need to move/copy or add-to-path `libcurl.dll`.
You can select static mode via `CFG=-static`.
- List more of the `Makefile.m32` config variables.
- Drop `.rc` support from examples. It made it fragile without much
benefit.
- Include a necessary system lib for the `externalsocket.c` example.
- Exclude unnecessary systems libs when building in `-dyn` mode.
Viktor Szakats [Mon, 3 Oct 2022 19:46:56 +0000 (19:46 +0000)]
Makefile.m32: major rework [ci skip]
This patch overhauls `Makefile.m32` scripts, fixing a list of quirks,
making its behaviour and customization envvars align better with other
build systems, aiming for less code, that is easier to read, use and
maintain.
Details:
- Rename customization envvars:
`CURL_CC` -> `CC`
`CURL_RC` -> `RC`
`CURL_AR` -> `AR`
`CURL_LDFLAG_EXTRAS_DLL` -> `CURL_LDFLAGS_LIB`
`CURL_LDFLAG_EXTRAS_EXE` -> `CURL_LDFLAGS_BIN`
- Drop `CURL_STRIP` and `CURL_RANLIB`. These tools are no longer used.
- Accept `CFLAGS`, `CPPFLAGS`, `RCFLAGS`, `LDFLAGS` and `LIBS` envvars.
- Drop `CURL_CFLAG_EXTRAS`, `CURL_LDFLAG_EXTRAS`, `CURL_RCFLAG_EXTRAS` in
favor of the above.
- Do not automatically enable `zlib` with `libssh2`. `zlib` is optional
with `libssh2`.
- Omit unnecessary `CPPFLAGS` options when building `curl.exe` and
examples.
- Drop support for deprecated `-winssl` `CFG` option. Use `-schannel`
instead.
- Avoid late evaluation where not necessary (`=` -> `:=`).
- Drop support for `CURL_DLL_A_SUFFIX` to override the implib suffix.
Instead, use the standard naming scheme by default: `libcurl.dll.a`.
The toolchain recognizes the name, and selects it automatically when
asking for a `-shared` vs. `-static` build.
- Stop applying `strip` to `libcurl.a`. Follow-up from 16a58e9f93c7e89e1f87720199388bcfcfa148a4. There was no debug info to
strip since then.
- Stop setting `-O3`, `-W`, `-Wall` options. You can add these to
`CFLAGS` as desired.
- Always enable `-DCURL_DISABLE_OPENSSL_AUTO_LOAD_CONFIG` with OpenSSL,
to avoid that vulnerability on Windows.
- Add `-lbrotlicommon` to `LIBS` when using `brotli`.
- Do not enable `-nghttp3` without `-ngtcp2`.
- `-ssh2` and `-rtmp` options no longer try to auto-select a TLS-backend.
You need to set the backend explicitly. This scales better and avoids
issues with certain combinations (e.g. `libssh2` + `wolfssl` with no
`schannel`).
- Default to OpenSSL TLS-backend with `ngtcp2`. Possible to override via
`NGTCP2_LIBS`.
- Old, alternate method of enabling components (e.g. `SSH2=1`) no longer
supported.
- Delete `SPNEGO` references. They were no-ops.
- Drop support for Win9x environments.
- Allow setting `OPENSSL_LIBS` independently from `OPENSSL_LIBPATH`.
- Support autotools/CMake `libssh2` builds by default.
- Respect `CURL_DLL_SUFFIX` in `-dyn` mode when building `curl.exe` and
examples.
- Assume standard directory layout with `LIBCARES_PATH`. (Instead of the
long gone embedded one.)
- Stop static linking with c-ares by default. Add
`CPPFLAGS=-DCARES_STATICLIB` to enable it.
- Reorganize internal layout to avoid redundancy and emit clean diffs
between src/lib and example make files.
- Delete unused variables.
- Code cleanups/rework.
- Comment and indentation fixes.
- Drop auto-detection of OpenSSL 1.0.2 and earlier. Now always defaulting
to OpenSSL 1.1.0 and later, LibreSSL and BoringSSL.
- Drop `Invalid path to OpenSSL package` detection. OpenSSL has been
using a standard file layout since 1.1.0, so this seems unnecessary
now.
- Drop special logic to enable Novell LDAP SDK support.
- Drop special logic to enable OpenLDAP LDAP SDK support. This seems
to be distinct from native OpenLDAP, with support implemented inside
`lib/ldap.c` (vs. `lib/openldap.c`) back when the latter did not exist
yet in curl.
- Add `-lwldap32` only if there is no other LDAP library (either native
OpenLDAP, or SDKs above) present.
- Update `doc/INSTALL.md` accordingly.
After this patch, it's necessary to make configration changes when using
OpenSSL 1.0.2 or earlier, or the two LDAP SDKs.
I haven't tested these scenarios, and in general we recommend using
a recent OpenSSL release. Also, WinLDAP (the Windows default) and
OpenLDAP (via `-DUSE_OPENLDAP`) are the LDAP options actively worked on
in curl.
Viktor Szakats [Sat, 1 Oct 2022 18:23:09 +0000 (18:23 +0000)]
docs: update sourceforge project links [ci skip]
SourceForge projects can now choose between two hostnames, with .io and
.net ending. Both support HTTPS by default now. Opening the other variant
will perm-redirected to the one chosen by the project.
The .io -> .net redirection is done insecurely.
Let's update the URLs to point to the current canonical endpoints to
avoid any redirects.
Viktor Szakats [Sat, 1 Oct 2022 10:07:34 +0000 (10:07 +0000)]
Makefile.m32: cleanups and fixes [ci skip]
- Add `-lcrypt32` once, and add it always for simplicity.
- Delete broken link and reference to the pre-Vista WinIDN add-on.
MS no longer distribute it.
- Delete related `WINIDN_PATH` option. IDN is a system lib since Vista.
- Sync `LIBCARES_PATH` default with the rest of dependencies.
- Delete version numbers from dependency path defaults.
- `libgsasl` package is now called `gsasl`.
- Delete `libexpat` and `libxml2` references. No longer used by curl.
- Delete `Edit the path below...` comments. We recommend to predefine
those envvars instead.
- `libcares.a` is not an internal dependency anymore. Stop using it as
such.
- `windres` `--include-dir` -> `-I`, `-F` -> `--target=` for readability.
- Delete `STRIP`, `CURL_STRIP`, `AR` references from `src/Makefile.m32`.
They were never used.
- Stop to `clean` some objects twice in `src/Makefile.m32`.
- Delete cvs-specific leftovers.
- Finish resource support in examples make file.
- Delete `-I<root>/lib` from examples make file.
- Fix copyright start year in examples make file.
- Delete duplicate `ftpuploadresume` input in examples make file.
- Sync OpenSSL lib order, `SYNC` support, `PROOT` use, dependency path
defaults, variables names and other internal bits between the three
make files.
- `lib/Makefile.m32` accepted custom options via `DLL_LIBS` envvar. This
was lib-specific and possibly accidental. Use `CURL_LDFLAG_EXTRAS_DLL`
envvar for the same effect.
- Fix linking `curl.exe` and examples to wrong static libs with
auto-detected OpenSSL 1.0.2 or earlier.
- Add `-lgdi32` for OpenSSL 1.0.2 and earlier only.
- Add link to Novell LDAP SDK and use a relative default path. Latest
version is from 2016, linked to an outdated OpenSSL 1.0.1.
- Whitespace and comment cleanups.
TODO in a next commit:
Delete built-in detection/logic for OpenSSL 1.0.2 and earlier, the Novell
LDAP SDK and the other LDAP SDK (which is _not_ OpenLDAP). Write up the
necessary custom envvars to configure them.
Added in 68b215157fdf69612edebdb220b3804822277822, while adding openldap
support. This is also the single mention of this constant in the source
tree and also in that commit. Based on these, it seems like an accident.
Patrick Monnerat [Wed, 28 Sep 2022 16:12:15 +0000 (18:12 +0200)]
lib: sanitize conditional exclusion around MIME
The introduction of CURL_DISABLE_MIME came with some additional bugs:
- Disabled MIME is compiled-in anyway if SMTP and/or IMAP is enabled.
- CURLOPT_MIMEPOST, CURLOPT_MIME_OPTIONS and CURLOPT_HTTPHEADER are
conditioned on HTTP, although also needed for SMTP and IMAP MIME mail
uploads.
In addition, the CURLOPT_HTTPHEADER and --header documentation does not
mention their use for MIME mail.
Daniel Stenberg [Tue, 27 Sep 2022 15:20:23 +0000 (17:20 +0200)]
altsvc: reject bad port numbers
The existing code tried but did not properly reject alternative services
using negative or too large port numbers.
With this fix, the logic now also flushes the old entries immediately
before adding a new one, making a following header with an illegal entry
not flush the already stored entry.
Report from the ongoing source code audit by Trail of Bits.
Daniel Stenberg [Tue, 27 Sep 2022 09:13:21 +0000 (11:13 +0200)]
configure: deprecate builds with small curl_off_t
If curl_off_t turns out to be smaller than 8 bytes,
--with-n64-deprecated needs to be used to allow the build to
continue. This is to highlight the fact that support for such builds is
going away next year.
Daniel Stenberg [Mon, 26 Sep 2022 14:19:50 +0000 (16:19 +0200)]
openssl: make certinfo available for QUIC
Curl_ossl_certchain() is now an exported function in lib/vtls/openssl.c that
can also be used from quiche.c and ngtcp2.c to get the cert chain for QUIC
connections as well.
The *certchain function was moved to the top of the file for this reason.
Viktor Szakats [Mon, 26 Sep 2022 22:31:09 +0000 (22:31 +0000)]
windows: adjust name of two internal public functions
According to `docs/INTERNALS.md`, internal function names spanning source
files start with uppercase `Curl_`. Bring these two functions in
alignment with this.
This also stops exporting them from `libcurl.dll` in autotools builds.
Jay Satiro [Fri, 9 Sep 2022 19:33:47 +0000 (15:33 -0400)]
schannel: ban server ALPN change during recv renegotiation
By the time schannel_recv is renegotiating the connection, libcurl has
already decided on a protocol and it is too late for the server to
select a protocol via ALPN except for the originally selected protocol.
Viktor Szakats [Sun, 25 Sep 2022 22:19:13 +0000 (22:19 +0000)]
autotools: allow --enable-symbol-hiding with windows
This local autotools logic was put in place in 9e24b9c7afbcb81120af4cf3f6cdee49a06d8224 (in 2012) which disabled it for
Windows unconditionally. Testing reveals that it actually works with
tested toolchains (mingw-w64 and CI ones), so let's allow this build
feature on that platform. Bringing this in sync with CMake, which already
supported this.
Viktor Szakats [Sun, 25 Sep 2022 21:58:21 +0000 (21:58 +0000)]
autotools: reduce brute-force when detecting recv/send arg list
autotools uses brute-force to detect `recv`/`send`/`select` argument
lists, by interating through _all_ argument type combinations on each
`./configure` run. This logic exists since 01fa02d0b545e1433dced2430561f8c0c72b74a9 (from 2006) and was a bit later
extended with Windows support.
This results in a worst-case number of compile + link cycles as below:
- `recv`: 96
- `send`: 192
- `select`: 60
Total: 348 (the number of curl C source files is 195, for comparison)
Notice that e.g. curl-for-win autotools builds require two `./configure`
invocations, doubling these numbers.
`recv` on Windows was especially unlucky because `SOCKET` (the correct
choice there) was listed _last_ in one of the outer trial loops. This
resulted in lengthy waits while autotools was trying all invalid
combinations first, wasting cycles, disk writes and slowing down
iteration.
This patch reduces the amount of idle work by reordering the tests in
a way to succeed first on a well-known platform such as Windows, and
also on non-Windows by testing for POSIX prototypes first, on the
assumption that these are the most likely candidates these days. (We do
not touch `select`, where the order was already optimal for these
platforms.)
For non-Windows, this means to try a return value of `ssize_t` first,
then `int`, reordering the buffer argument type to try `void *` first,
then `byte *`, and prefer the `const` flavor with `send`. If we are
here, also stop testing for `SOCKET` type in non-Windows builds.
After the patch, detection on Windows is instantaneous. It should also be
faster on popular platforms such as Linux and BSD-based ones.
If there are known-good variations for other platforms, they can also be
fast-tracked like above, given a way to check for that platform inside
the autotools logic.
Daniel Stenberg [Thu, 22 Sep 2022 09:52:08 +0000 (11:52 +0200)]
mprintf: use snprintf if available
This is the single place in libcurl code where it uses the "native"
s(n)printf() function. Used for writing floats. The use has been
reviewed and vetted and uses a HUGE target buffer, but switching to
snprintf() still makes this safer and removes build-time warnings.
Reported-by: Philip Heiduck
Fixes #9569
Closes #9570
Patrick Monnerat [Wed, 21 Sep 2022 08:41:22 +0000 (10:41 +0200)]
tool: remove protocol count limitation
Replace bit mask protocol sets by null-terminated arrays of protocol
tokens. These are the addresses of the protocol names returned by
curl_version_info().
Protocol names are sorted case-insensitively before output to satisfy CI
tests matches consistency.
The protocol list returned by curl_version_info() is augmented with all
RTMP protocol variants.
Daniel Stenberg [Wed, 21 Sep 2022 10:32:16 +0000 (12:32 +0200)]
test972: verify the output without using external tool
It seems too restrictive to assume and use an external tool to verify
the JSON. This now verifies the outut byte per byte. We could consider
building a local "JSON verifyer" in a future.
Daniel Stenberg [Wed, 21 Sep 2022 06:52:57 +0000 (08:52 +0200)]
hostip: lazily wait to figure out if IPv6 works until needed
The check may take many milliseconds, so now it is performed once the
value is first needed. Also, this change makes sure that the value is
not used if the resolve is set to be IPv4-only.
Daniel Stenberg [Tue, 20 Sep 2022 12:50:09 +0000 (14:50 +0200)]
symbol-scan.pl: scan and verify .3 man pages
This script now also finds all .3 man pages in docs/include and
docs/include/opts, extracts all uses of CURL* symbols and verifies that all
symbols mentioned in docs are defined in public headers.
A "global symbol" is one of those matching a known prefix and the script makes
an attempt to check all/most of them. Just using *all* symbols that match
CURL* proved matching a little too many other references as well and turned
difficult turning into something useful.
Daniel Stenberg [Tue, 20 Sep 2022 11:32:58 +0000 (13:32 +0200)]
test1119: scan all public headers
Previously this test only scanned a subset of the headers, which made us
accidentally miss symbols that were provided in the others. Now, the script
iterates over all headers present in include/curl.
Daniel Stenberg [Tue, 20 Sep 2022 21:30:02 +0000 (23:30 +0200)]
GHA: spellcheck
This spellchecker checks markdown files. For this reason this job
converts all man pages in the repository to markdown with pandoc before
the check runs.
The perl script 'cleanspell' filters out details from the man page in
the process, to avoid the spellchecker trying to spellcheck things it
can't. Like curl specific symbols and the SYNOPSIS and EXAMPLE sections
of libcurl man pages.
The spell checker does not check words in sections that are within pre,
strong and em tags.
'spellcheck.words' is a custom word list with additional accepted words.
Daniel Stenberg [Wed, 21 Sep 2022 07:06:49 +0000 (09:06 +0200)]
connect: fix the wrong error message on connect failures
The "Failed to connect to" message after a connection failure would
include the strerror message based on the presumed previous socket
error, but in times it seems that error number is not set when reaching
this code and therefore it would include the wrong error message.
The strerror message is now removed from here and the curl_easy_strerror
error is used instead.
Daniel Stenberg [Wed, 21 Sep 2022 07:32:12 +0000 (09:32 +0200)]
httpput-postfields.c: shorten string for C89 compliance
httpput-postfields.c:41:3: error: string length ‘522’ is greater than the length ‘509’ ISO C90 compilers are required to support [-Woverlength-strings]
41 | "this chapter.";
| ^~~~~~~~~~~~~~~
Patrick Monnerat [Wed, 21 Sep 2022 01:00:30 +0000 (03:00 +0200)]
unit test 1655: make it C89-compliant
Initializations performed in unit test 1655 use automatic variables in
aggregates and thus can only be computed at run-time. Using gcc in C89
dialect mode produces warning messages like:
unit1655.c:96:7: warning: initializer element is not computable at load time [-Wpedantic]
96 | { toolong, DOH_DNS_NAME_TOO_LONG }, /* expect early failure */
| ^~~~~~~
Fix the problem by converting these automatic pointer variables to
static arrays.
resolve: make forced IPv4 resolve only use A queries
This protects IPv4-only transfers from undesired bad IPv6-related side
effects and make IPv4 transfers in dual-stack libcurl behave the same
way as in IPv4 single-stack libcurl.