Clarify that decoding too many Huffman weights is a failure condition

Merge pull request #3806 from elasota/fmt-clarifications

Correct FSE probability bit consumption in specification

Merge pull request #3800 from facebook/dependabot/github_actions/actions/checkout-4.1.1

Bump actions/checkout from 4.1.0 to 4.1.1

Merge pull request #3795 from Saverio976/doc/cmake-fetch-content

Add doc on how to use it with cmake FetchContent

Add definition of "log2sup" function

Clarify that the log2 of the largest possible symbol is the maximum number of bits consumed

Add target_include_directories because windows and macos need it for me

Bump actions/checkout from 4.1.0 to 4.1.1

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.0 to 4.1.1.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/8ade135a41bc03ea155e62e844d188df1ea18608...b4ffde65f46336ab88eb53be808477a3936bae11)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #3789 from facebook/fix_flexArray_fse

solving flexArray issue #3785 in fse

revert to manually defining DTable

thus avoiding the analyzer and ubsan to associate DTable to a size of 1.

baby-step towards solving flexArray issue #3785

the flexArray in structure FSE_DecompressWksp
is just a way to derive a pointer easily,
without risk/complexity of calculating it manually.

Not sure if this change is good enough to avoid ubsan warnings though.

Add doc on how to use it with cmake FetchContent

Merge pull request #3786 from facebook/fix_flexarray_cctx

Remove FlexArray pattern from ZSTDMT

extended the fix to ZSTDMT's Buffer Pool

removed unused macro constant

fixes suggested by @ebiggers

removed FlexArray pattern from CCtxPool

within ZSTDMT_.
This pattern is flagged by less forgiving variants of ubsan
notably used during compilation of the Linux Kernel.

There are 2 other places in the code where this pattern is used.
This fixes just one of them.

Merge pull request #3752 from paulmenzel/fix-pzstd-makefile

Fix pzstd Makefile to allow setting `DESTDIR` and `BINDIR` separately

Merge pull request #3763 from dloidolt/fix_lib/README.md

Fix a very small formatting typo in the lib/README.md file

Merge pull request #3772 from DimitriPapadopoulos/WIN32

Do not test WIN32, instead test _WIN32

Merge pull request #3771 from DimitriPapadopoulos/codespell

Fix new typos found by codespell

Merge pull request #3777 from facebook/fix_x32

fix x32 tests on Github CI

Stop suppressing pointer-overflow UBSAN errors

* Remove all pointer-overflow suppressions from our UBSAN builds/tests.
* Add `ZSTD_ALLOW_POINTER_OVERFLOW_ATTR` macro to suppress
  pointer-overflow at a per-function level. This is a superior approach
  because it also applies to users who build zstd with UBSAN.
* Add `ZSTD_wrappedPtr{Diff,Add,Sub}()` that use these suppressions.
  The end goal is to only tag these functions with
  `ZSTD_ALLOW_POINTER_OVERFLOW`. But we can start by annoting functions
  that rely on pointer overflow, and gradually transition to using
  these.
* Add `ZSTD_maybeNullPtrAdd()` to simplify pointer addition when the
  pointer may be `NULL`.
* Fix all the fuzzer issues that came up. I'm sure there will be a lot
  more, but these are the ones that came up within a few minutes of
  running the fuzzers, and while running GitHub CI.

Revert "Work around nullptr-with-nonzero-offset warning"

This reverts commit c27fa399042f466080e79bb4fd8a4871bc0bcf28.

fix x32 tests on Github CI

ubuntu-22.04 seems to have problems with x32 recently
switching to ubuntu-20.04 which seems to work fine so far

https://github.com/actions/runner-images/issues/8397

Merge pull request #3774 from facebook/dependabot/github_actions/actions/checkout-4.1.0

Bump actions/checkout from 4.0.0 to 4.1.0

Bump actions/checkout from 4.0.0 to 4.1.0

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.0.0 to 4.1.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/3df4ab11eba7bda6032a0b82a6bb43b11571feac...8ade135a41bc03ea155e62e844d188df1ea18608)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Do not test WIN32, instead test _WIN32

To the best of my knowledge:
* `_WIN32` and `_WIN64` are defined by the compiler,
* `WIN32` and `WIN64` are defined by the user, to indicate whatever
  the user chooses them to indicate. They mean 32-bit and 64-bit Windows
  compilation by convention only.

See:
https://accu.org/journals/overload/24/132/wilson_2223/

Windows compilers in general, and MSVC in particular, have been defining
`_WIN32` and `_WIN64` for a long time, provably at least since Visual Studio
2015, and in practice as early as in the days of 16-bit Windows.

See:
https://learn.microsoft.com/en-us/cpp/preprocessor/predefined-macros?view=msvc-140
https://learn.microsoft.com/en-us/windows/win32/winprog64/the-tools

Tests used to be inconsistent, sometimes testing `_WIN32`, sometimes
`_WIN32` and `WIN32`. This brings consistency to Windows detection.

Fix new typos found by codespell

Improve macro guards for ZSTD_assertValidSequence

Refine the macro guards to define the functions exactly when they are
needed.

This fixes the chromium build with zstd.

Thanks to @GregTho for reporting!

Fix a very small formatting typo in the lib/README.md file

Merge pull request #3755 from facebook/estimate_doc

added some documentation on ZSTD_estimate*Size() variants

added some documentation on ZSTD_estimate*Size() variants

as a follow up for #3747

Merge pull request #3749 from facebook/dependabot/github_actions/actions/checkout-4.0.0

Bump actions/checkout from 3.5.3 to 4.0.0

Merge pull request #3745 from klausholstjacobsen/qnx-support

Added qnx in the posix test section of platform.h

Merge pull request #3750 from facebook/dependabot/github_actions/actions/upload-artifact-3.1.3

Bump actions/upload-artifact from 3.1.2 to 3.1.3

[pzstd]: Fix `DESTDIR` handling to allow setting `BINDIR`

Currently, setting `BINDIR` and `DESTDIR` separately is not possible, so
the command below fails, as BINDIR is set explicitly:

    $ make -j80 install PREFIX=/usr EPREFIX=/usr BINDIR=/usr/bin SBINDIR=/usr/sbin LIBEXECDIR=/usr/libexec SYSCONFDIR=/etc SHAREDSTATEDIR=/var LOCALST ATEDIR=/var LIBDIR=/usr/lib INCLUDEDIR=/usr/include DATAROOTDIR=/usr/share DATADIR=/usr/share INFODIR=/usr/share/info LOCALEDIR=/usr/share/locale MAND IR=/usr/share/man DOCDIR=/usr/share/doc/zstd DESTDIR=/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/image -C contrib/pzstd DESTDIR=/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/image
    make: Entering directory
    '/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/source/contrib/pzstd' CFLAGS="  -I../../lib -I../../lib/common -I../../programs -I. -DNDEBUG -O3 -Wall -Wextra -Wno-deprecated-declarations   " LDFLAGS=" -O3 -Wall -Wextra -pedantic  " make -C ../../lib libzstd.a
    make[1]: Entering directory '/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/source/lib'
    make[1]: Leaving directory '/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/source/lib'
    g++ main.o ../../programs/util.o Options.o Pzstd.o SkippableFrame.o ../../lib/libzstd.a           -O3 -Wall -Wextra -pedantic    -pthread -o pzstd
    install -d -m 755 /usr/bin/
    install -m 755 pzstd /usr/bin/pzstd
    install: cannot create regular file '/usr/bin/pzstd': Permission denied
    make: *** [Makefile:116: install] Error 1

So, do not prefix `BINDIR` with `DESTDIR`, and adapt all paths for
installation. This is more common, and, for example, `programs/Makefile`
does the same.

Fixes: 8b4e84249b ("[pzstd] Fix Makefile")

Bump actions/upload-artifact from 3.1.2 to 3.1.3

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/0b7f8abb1508181956e8e162db84b466c27e18ce...a8a3f3ad30e3422c9c7b888a15615d19a852ae32)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Bump actions/checkout from 3.5.3 to 4.0.0

Bumps [actions/checkout](https://github.com/actions/checkout) from 3.5.3 to 4.0.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/c85c95e3d7251135ab7dc9ce3241c5835cc595a9...3df4ab11eba7bda6032a0b82a6bb43b11571feac)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #3739 from JohanMabille/cmake

Fixed zstd cmake shared build on windows

Merge pull request #3733 from ldv-alt/zdictlib_fix_prototype_mismatch

zdictlib: fix prototype mismatch

Added qnx in the posix test section of platform.h

Fixed zstd cmake shared build on windows

Fix & refactor Huffman repeat tables for dictionaries

The Huffman repeat mode checker assumed that the CTable was zeroed in the region `[maxSymbolValue + 1, 256)`.
This assumption didn't hold for tables built in the dictionaries, because it didn't go through the same codepath.

Since this code was originally written, we added a header to the CTable that specifies the `tableLog`.
Add `maxSymbolValue` to that header, and check that the table's `maxSymbolValue` is at least the block's `maxSymbolValue`.

This solution is cleaner because we write this header for every CTable we build, so it can't be missed in any code path.

Credit to OSS-Fuzz

Work around nullptr-with-nonzero-offset warning

See comment.

zdictlib: fix prototype mismatch

Fix the following warnings reported by the compiler when
ZDICTLIB_STATIC_API is not defined to ZDICTLIB_API:

lib/dictBuilder/cover.c:1122:21: warning: redeclaration of 'ZDICT_optimizeTrainFromBuffer_cover' with different visibility (old visibility
preserved)
lib/dictBuilder/cover.c:736:21: warning: redeclaration of 'ZDICT_trainFromBuffer_cover' with different visibility (old visibility
+preserved)
lib/dictBuilder/fastcover.c:549:1: warning: redeclaration of 'ZDICT_trainFromBuffer_fastCover' with different visibility (old visibility
preserved)
lib/dictBuilder/fastcover.c:618:1: warning: redeclaration of 'ZDICT_optimizeTrainFromBuffer_fastCover' with different visibility (old
visibility preserved)

Merge pull request #3720 from QBos07/cygwin-msys2-support

Updated Makefiles for full MSYS2 and Cygwin installation and testing …

Merge pull request #3728 from 0o001/dev

fix: ZSTD_BUILD_DECOMPRESSION message

No longer reject dictionaries with literals maxSymbolValue < 255

We already have logic in our Huffman encoder to validate Huffman tables with missing symbols.
We use this for higher compression levels to re-use the previous blocks statistics, or when the dictionaries table has zero-weighted symbols.
This check was leftover as an oversight from before we added validation for Huffman tables.

I validated that the `dictionary_loader` fuzzer has coverage of every line in the `ZSTD_loadCEntropy()` function to validate that it is correctly testing this function.

Merge pull request #3726 from facebook/fullbench_dctx

added ZSTD_decompressDCtx() benchmark option to fullbench

Merge pull request #3730 from facebook/dependabot/github_actions/github/codeql-action-2.21.4

Bump github/codeql-action from 2.20.3 to 2.21.4

Bump github/codeql-action from 2.20.3 to 2.21.4

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.20.3 to 2.21.4.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/46ed16ded91731b2df79a2893d3aea8e9f03b5c4...a09933a12a80f87b87005513f0abb1494c27a716)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

fix: ZSTD_BUILD_DECOMPRESSION message

added ZSTD_decompressDCtx() benchmark option to fullbench

useful to compare the difference between ZSTD_decompress
and ZSTD_decompressDCtx().

Merge pull request #3725 from felixhandte/msan-unpoison-cwksp-on-free

Unpoison Workspace Memory Before Custom-Free

Unpoison Workspace Memory Before Freeing to Custom Free

MSAN is hooked into the system malloc, but when the user provides a custom
allocator, it may not provide the same cleansing behavior. So if we leave
memory poisoned and return it to the user's allocator, where it is re-used
elsewhere, our poisoning can blow up in some other context.

Easy: Move Helper Functions Up

Improve dual license wording in README

We are licensed under BSD or GPLv2. It is clear in our headers, but not in the README.

Fixes #3717

Updated Makefiles for full MSYS2 and Cygwin installation and testing support.

They are Linux-like environments under Windows and have all the tools needed to support staged installation and testing.

Beware: this only affects the make build system.

Merge pull request #3712 from alexsifivetw/fix_typo

Fixed typo

Fixed typo

Merge pull request #3701 from nikohoffren/grammar-fix

Fix typographical error in README.md

Merge pull request #3704 from void0red/dev

fileio_asyncio: handle malloc fails in AIO_ReadPool_create

fileio_asyncio: handle malloc fails in AIO_ReadPool_create

Update fileio.c: fix build failure with enabled LTO

For some reasons when LTO is enabled, the compiler complains about statbuf variable not being correctly initialized, even though the variable has an assert != NULL just few lines below (FIO_getDictFileStat)

This is the fixed build failure:
x86_64-linux-gnu-gcc -g -O2 -ffile-prefix-map=/<<PKGBUILDDIR>>=. -flto=auto -ffat-lto-objects -fstack-protector-strong -Wformat -Werror=format-security -fdebug-prefix-map=/<<PKGBUILDDIR>>=/usr/src/libzstd-1.5.5+dfsg2-1 -Wall -Wextra -Wcast-qual -Wcast-align -Wshadow -Wstrict-aliasing=1 -Wswitch-enum -Wdeclaration-after-statement -Wstrict-prototypes -Wundef -Wpointer-arith -Wvla -Wformat=2 -Winit-self -Wfloat-equal -Wwrite-strings -Wredundant-decls -Wmissing-prototypes -Wc++-compat -g -Werror -Wa,--noexecstack -Wdate-time -D_FORTIFY_SOURCE=2 -DXXH_NAMESPACE=ZSTD_ -DDEBUGLEVEL=1 -DZSTD_LEGACY_SUPPORT=5 -DZSTD_MULTITHREAD -DZSTD_GZCOMPRESS -DZSTD_GZDECOMPRESS -DZSTD_LZMACOMPRESS -DZSTD_LZMADECOMPRESS -DZSTD_LZ4COMPRESS -DZSTD_LZ4DECOMPRESS -DZSTD_LEGACY_SUPPORT=5  -c -MT obj/conf_086c46a51a716b674719b8acb8484eb8/zstdcli_trace.o -MMD -MP -MF obj/conf_086c46a51a716b674719b8acb8484eb8/zstdcli_trace.d -o obj/conf_086c46a51a716b674719b8acb8484eb8/zstdcli_trace.o zstdcli_trace.c
In function ‘UTIL_isRegularFileStat’,
    inlined from ‘UTIL_getFileSizeStat’ at util.c:524:10,
    inlined from ‘FIO_createDResources’ at fileio.c:2230:30:
util.c:209:12: error: ‘statbuf.st_mode’ may be used uninitialized [-Werror=maybe-uninitialized]
  209 |     return S_ISREG(statbuf->st_mode) != 0;
      |            ^
fileio.c: In function ‘FIO_createDResources’:
fileio.c:2223:12: note: ‘statbuf’ declared here
 2223 |     stat_t statbuf;
      |            ^
lto1: all warnings being treated as errors

Save one byte on the frame epilogue

Merge pull request #3665 from gjasny/fix-asm-for-xcode

Fix Intel Xcode builds with assembly

Fix typographical error in README.md

Merge pull request #3697 from facebook/dependabot/github_actions/github/codeql-action-2.20.3

Bump github/codeql-action from 2.20.1 to 2.20.3

Bump github/codeql-action from 2.20.1 to 2.20.3

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.20.1 to 2.20.3.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/f6e388ebf0efc915c6c5b165b019ee61a6746a38...46ed16ded91731b2df79a2893d3aea8e9f03b5c4)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #3677 from facebook/detectOverflow

Changed the decoding loop to detect more invalid cases of corruption sooner

Merge pull request #3688 from nidhijaju/hide-asm-apple

Hide ASM symbols on Apple platforms

Merge pull request #3689 from facebook/dependabot/github_actions/github/codeql-action-2.20.1

Bump github/codeql-action from 2.3.2 to 2.20.1

Merge pull request #3690 from facebook/dependabot/github_actions/ossf/scorecard-action-2.2.0

Bump ossf/scorecard-action from 2.1.3 to 2.2.0

Merge pull request #3686 from embg/ldm_error

Clean up a false error message in the LDM debug log

Bump ossf/scorecard-action from 2.1.3 to 2.2.0

Bumps [ossf/scorecard-action](https://github.com/ossf/scorecard-action) from 2.1.3 to 2.2.0.
- [Release notes](https://github.com/ossf/scorecard-action/releases)
- [Changelog](https://github.com/ossf/scorecard-action/blob/main/RELEASE.md)
- [Commits](https://github.com/ossf/scorecard-action/compare/80e868c13c90f172d68d1f4501dee99e2479f7af...08b4669551908b1024bb425080c797723083c031)

---
updated-dependencies:
- dependency-name: ossf/scorecard-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Bump github/codeql-action from 2.3.2 to 2.20.1

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.3.2 to 2.20.1.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/f3feb00acb00f31a6f60280e6ace9ca31d91c76a...f6e388ebf0efc915c6c5b165b019ee61a6746a38)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

hide asm functions on apple platforms

Merge pull request #3684 from emaste/dev

Update FreeBSD CI images to latest supported releases

suppress false error message in LDM mode

Update FreeBSD CI images to latest supported releases

fixed incorrect test in Win32 pthread wrapper

reported by @Banzai24-yht in #3683

fixed static analyzer false positive regarding @sequence initialization

make a mock initialization to please the tool

adapted long decoder to new decodeSequences

removed older decodeSequences

changed ZSTD_decompressSequences_bodySplitLitBuffer() decoding loop

to behave more like the regular decoding loop.

removed _old variant from splitLit

changed (partially) the decodeSequences flow logic

this allows detecting overflow events without a checksum.

Merge pull request #3676 from facebook/overflow_zeroes

Bitstream produces only zeroes after an overflow event

fixed MEM_STATIC already defined in Linux Kernel mode

fix : unused attribute for FORCE_INLINE functions

fix2 : reloadDStreamFast is used by decompress4x2,
modified the entry point, so that it works fine in this case too.

make the bitstream generate only 0-value bits after an overflow

Merge pull request #3674 from facebook/zeroSeq_noExtra

detect extraneous bytes in the Sequences section

detect extraneous bytes in the Sequences section

when nbSeq == 0.

Reported by @ip7z

Merge pull request #3671 from facebook/dependabot/github_actions/actions/checkout-3.5.3

Bump actions/checkout from 3.5.2 to 3.5.3

Bump actions/checkout from 3.5.2 to 3.5.3

Bumps [actions/checkout](https://github.com/actions/checkout) from 3.5.2 to 3.5.3.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/8e5e7e5ab8b370d6c329ec480221332ada57f0ab...c85c95e3d7251135ab7dc9ce3241c5835cc595a9)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #3669 from facebook/zeroSeq_2B

fixed decoder behavior when nbSeqs==0 is encoded using 2 bytes

fixed decoder behavior when nbSeqs==0 is encoded using 2 bytes

The sequence section starts with a number, which tells how sequences are present in the section.
If this number if 0, the section automatically ends.

The number 0 can be represented using the 1 byte or the 2 bytes formats.
That's because the 2-bytes formats fully overlaps the 1 byte format.

However, when 0 is represented using the 2-bytes format,
the decoder was expecting the sequence section to continue,
and was looking for FSE tables, which is incorrect.

Fixed this behavior, in both the reference decoder and the educational behavior.

In practice, this behavior never happens,
because the encoder will always select the 1-byte format to represent 0,
since this is more efficient.

Completed the fix with a new golden sample for tests,
a clarification of the specification,
and a decoder errata paragraph.

Merge pull request #3664 from facebook/llu

changed LLU suffix into ULL for Visual 2012 and lower

Merge pull request #3659 from facebook/fixHarness

Fixed a bug in the educational decoder

Merge pull request #3668 from facebook/fix3667

fix a minor inefficiency in compress_superblock