]> git.ipfire.org Git - thirdparty/zstd.git/log
thirdparty/zstd.git
18 months agomade playTests.sh more compatible with older versions of grep 3877/head
Yann Collet [Tue, 16 Jan 2024 20:14:35 +0000 (12:14 -0800)] 
made playTests.sh more compatible with older versions of grep

replaced `\+` by `*`.
`\+` means `[1-N]`,
while `*` means `[0-N]`,
so it's not strictly equivalent
but `\+` happens to be badly supported on some flavors of grep,
and for the purpose of these tests, `*` is good enough.

18 months agoMerge pull request #3865 from facebook/grep_e
Yann Collet [Mon, 15 Jan 2024 20:44:46 +0000 (12:44 -0800)] 
Merge pull request #3865 from facebook/grep_e

playTests.sh does no longer needs grep -E

18 months agoplayTests.sh does no longer needs grep -E 3865/head
Yann Collet [Mon, 15 Jan 2024 19:16:46 +0000 (11:16 -0800)] 
playTests.sh does no longer needs grep -E

it makes the test script more portable across posix systems
because `grep -E` is not guaranteed
while `grep` is fairly common.

18 months agoMerge pull request #3863 from facebook/dependabot/github_actions/github/codeql-action...
Yann Collet [Mon, 15 Jan 2024 17:44:19 +0000 (09:44 -0800)] 
Merge pull request #3863 from facebook/dependabot/github_actions/github/codeql-action-3.23.0

Bump github/codeql-action from 2.21.4 to 3.23.0

18 months agoMerge pull request #3864 from facebook/dependabot/github_actions/actions/upload-artif...
Yann Collet [Mon, 15 Jan 2024 07:44:56 +0000 (23:44 -0800)] 
Merge pull request #3864 from facebook/dependabot/github_actions/actions/upload-artifact-4.1.0

Bump actions/upload-artifact from 4.0.0 to 4.1.0

18 months agoBump actions/upload-artifact from 4.0.0 to 4.1.0 3864/head
dependabot[bot] [Mon, 15 Jan 2024 05:24:46 +0000 (05:24 +0000)] 
Bump actions/upload-artifact from 4.0.0 to 4.1.0

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 4.0.0 to 4.1.0.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/c7d193f32edcb7bfad88892161225aeda64e9392...1eb3cb2b3e0f29609092a73eb033bb759a334595)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
18 months agoBump github/codeql-action from 2.21.4 to 3.23.0 3863/head
dependabot[bot] [Mon, 15 Jan 2024 05:24:41 +0000 (05:24 +0000)] 
Bump github/codeql-action from 2.21.4 to 3.23.0

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 2.21.4 to 3.23.0.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/a09933a12a80f87b87005513f0abb1494c27a716...e5f05b81d5b6ff8cfa111c80c22c5fd02a384118)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
18 months agoMerge pull request #3760 from jcelerier/patch-1
Yann Collet [Sat, 13 Jan 2024 19:41:30 +0000 (11:41 -0800)] 
Merge pull request #3760 from jcelerier/patch-1

[x-compile] Fix cross-compiling for AArch64 with lld

18 months agoMerge pull request #3817 from elasota/oversized-probs-clarification
Yann Collet [Sat, 13 Jan 2024 19:37:54 +0000 (11:37 -0800)] 
Merge pull request #3817 from elasota/oversized-probs-clarification

Clarify that probability tables must not contain non-zero probabilities for invalid values

18 months agoMerge pull request #3850 from KapJI/better-errors
Yann Collet [Sat, 13 Jan 2024 19:37:25 +0000 (11:37 -0800)] 
Merge pull request #3850 from KapJI/better-errors

cli: better errors on argument parsing

18 months agoMerge pull request #3858 from eli-schwartz/msvc-dev-cmd
Yann Collet [Sat, 13 Jan 2024 19:36:20 +0000 (11:36 -0800)] 
Merge pull request #3858 from eli-schwartz/msvc-dev-cmd

CI: meson: use builtin handling for MSVC

19 months agoCI: meson: use builtin handling for MSVC 3858/head
Eli Schwartz [Tue, 2 Jan 2024 06:36:45 +0000 (01:36 -0500)] 
CI: meson: use builtin handling for MSVC

This avoids downloading -- and periodically bumping the checksum for --
a third-party action that isn't strictly required, and thus helps keep
down dependencies and reduce update churn.

19 months agoMerge pull request #3855 from emaste/freebsd-14-ci
Yann Collet [Sun, 31 Dec 2023 06:16:42 +0000 (22:16 -0800)] 
Merge pull request #3855 from emaste/freebsd-14-ci

Cirrus-CI: Add FreeBSD 14

19 months agoMerge pull request #3847 from michoecho/fix_nullptr_deref_in_createCDict
Yann Collet [Sat, 30 Dec 2023 21:23:39 +0000 (13:23 -0800)] 
Merge pull request #3847 from michoecho/fix_nullptr_deref_in_createCDict

Fix a nullptr dereference in ZSTD_createCDict_advanced2()

19 months agoCirrus-CI: Add FreeBSD 14 3855/head
Ed Maste [Thu, 14 Dec 2023 00:56:14 +0000 (19:56 -0500)] 
Cirrus-CI: Add FreeBSD 14

19 months agoMerge pull request #3846 from emaste/c89
Yann Collet [Fri, 29 Dec 2023 20:34:47 +0000 (12:34 -0800)] 
Merge pull request #3846 from emaste/c89

zlibWrapper: convert to C89 / ANSI C

19 months agoMake offload API compatible with static CCtx (#3854)
Elliot Gorokhovsky [Thu, 28 Dec 2023 19:48:46 +0000 (14:48 -0500)] 
Make offload API compatible with static CCtx (#3854)

* Add ZSTD_CCtxParams_registerSequenceProducer() to public API

* add unit test

* add docs to zstd.h

* nits

* Add ZSTDLIB_STATIC_API prefix

* Add asserts

19 months agoMerge pull request #3849 from facebook/dependabot/github_actions/actions/upload-artif...
Yann Collet [Mon, 18 Dec 2023 17:49:57 +0000 (09:49 -0800)] 
Merge pull request #3849 from facebook/dependabot/github_actions/actions/upload-artifact-4.0.0

Bump actions/upload-artifact from 3.1.3 to 4.0.0

19 months agoUpdate windows-artifacts.yml 3849/head
Yann Collet [Mon, 18 Dec 2023 17:39:42 +0000 (09:39 -0800)] 
Update windows-artifacts.yml

and fixed version number in comment

19 months agocli: better errors on arguent parsing 3850/head
Ruslan Sayfutdinov [Sat, 16 Dec 2023 15:28:19 +0000 (15:28 +0000)] 
cli: better errors on arguent parsing

19 months agoBump actions/upload-artifact from 3.1.3 to 4.0.0
dependabot[bot] [Mon, 18 Dec 2023 05:26:39 +0000 (05:26 +0000)] 
Bump actions/upload-artifact from 3.1.3 to 4.0.0

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.3 to 4.0.0.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/a8a3f3ad30e3422c9c7b888a15615d19a852ae32...c7d193f32edcb7bfad88892161225aeda64e9392)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
19 months agoFix a nullptr dereference in ZSTD_createCDict_advanced2() 3847/head
Michał Chojnowski [Sat, 16 Dec 2023 11:32:52 +0000 (12:32 +0100)] 
Fix a nullptr dereference in ZSTD_createCDict_advanced2()

If the relevant allocation returns NULL, ZSTD_createCDict_advanced_internal()
will return NULL. But ZSTD_createCDict_advanced2() doesn't check for
this and attempts to use the returned pointer anyway, which leads to
a segfault.

19 months agozlibWrapper: convert to C89 / ANSI C 3846/head
Ed Maste [Thu, 14 Dec 2023 00:54:29 +0000 (19:54 -0500)] 
zlibWrapper: convert to C89 / ANSI C

Clang 16 (which is the system compiler in FreeBSD 14.0) no longer allows
K&R function definitions.  Formatting of the changes matches current
zlib.

19 months agoMerge pull request #3845 from emaste/dev
Yann Collet [Thu, 14 Dec 2023 07:54:41 +0000 (23:54 -0800)] 
Merge pull request #3845 from emaste/dev

Update FreeBSD CI: drop 12.4 (nearly EOL)

19 months agoUpdate FreeBSD CI: drop 12.4 as it is nearly EOL 3845/head
Ed Maste [Wed, 13 Dec 2023 15:53:13 +0000 (10:53 -0500)] 
Update FreeBSD CI: drop 12.4 as it is nearly EOL

12.4 is EOL as of the end of December 2023, and pkg installation will
start failing some time after that so remove those jobs now.

20 months agoMerge pull request #3839 from embg/offload_refactor
Elliot Gorokhovsky [Wed, 29 Nov 2023 01:04:08 +0000 (20:04 -0500)] 
Merge pull request #3839 from embg/offload_refactor

Move offload API params into ZSTD_CCtx_params

20 months agoMove offload API params into ZSTD_CCtx_params 3839/head
Elliot Gorokhovsky [Mon, 27 Nov 2023 14:52:52 +0000 (06:52 -0800)] 
Move offload API params into ZSTD_CCtx_params

20 months agoRefactor ZSTD_sequenceProducer_F typedef to ZSTD_sequenceProducer_F*
Elliot Gorokhovsky [Mon, 27 Nov 2023 14:52:36 +0000 (06:52 -0800)] 
Refactor ZSTD_sequenceProducer_F typedef to ZSTD_sequenceProducer_F*

20 months agoClarify that there must be at least 2 weights, i.e. encoding all weights as 0 is...
elasota [Thu, 9 Nov 2023 04:46:37 +0000 (23:46 -0500)] 
Clarify that there must be at least 2 weights, i.e. encoding all weights as 0 is invalid

20 months agoClarify that the presence of a value with weight 1 is required
elasota [Thu, 9 Nov 2023 04:25:00 +0000 (23:25 -0500)] 
Clarify that the presence of a value with weight 1 is required

20 months agoModernize macros to use `do { } while (0)`
Nick Terrell [Tue, 21 Nov 2023 21:26:25 +0000 (13:26 -0800)] 
Modernize macros to use `do { } while (0)`

This PR introduces no functional changes. It attempts to change all
macros currently using `{ }` or some variant of that to to
`do { } while (0)`, and introduces trailing `;` where necessary.
There were no bugs found during this migration.

The bug in Visual Studios warning on this has been fixed since VS2015.
Additionally, we have several instances of `do { } while (0)` which have
been present for several releases, so we don't have to worry about
breaking peoples builds.

Fixes Issue #3830.

20 months agoMerge pull request #3820 from facebook/xxh082
Yann Collet [Tue, 21 Nov 2023 17:11:40 +0000 (09:11 -0800)] 
Merge pull request #3820 from facebook/xxh082

update xxhash library to v0.8.2

20 months ago[huf] Fix null pointer addition
Nick Terrell [Mon, 20 Nov 2023 20:04:30 +0000 (12:04 -0800)] 
[huf] Fix null pointer addition

`HUF_DecompressFastArgs_init()` was adding 0 to NULL. Fix it by exiting
early for empty outputs. This is no change in behavior, because the
function was already exiting 0 in this case, just slightly later.

20 months ago[huf] Improve fast C & ASM performance on small data
Nick Terrell [Mon, 20 Nov 2023 19:33:57 +0000 (11:33 -0800)] 
[huf] Improve fast C & ASM performance on small data

* Rename `ilimit` to `ilowest` and set it equal to `src` instead of
  `src + 6 + 8`. This is safe because the fast decoding loops guarantee
  to never read below `ilowest` already. This allows the fast decoder to
  run for at least two more iterations, because it consumes at most 7
  bytes per iteration.
* Continue the fast loop all the way until the number of safe iterations
 is 0. Initially, I thought that when it got towards the end, the
 computation of how many iterations of safe might become expensive. But
 it ends up being slower to have to decode each of the 4 streams
 individually, which makes sense.

This drastically speeds up the Huffman decoder on the `github` dataset
for the issue raised in #3762, measured with `zstd -b1e1r github/`.

| Decoder  | Speed before | Speed after |
|----------|--------------|-------------|
| Fallback | 477 MB/s     | 477 MB/s    |
| Fast C   | 384 MB/s     | 492 MB/s    |
| Assembly | 385 MB/s     | 501 MB/s    |

We can also look at the speed delta for different block sizes of silesia
using `zstd -b1e1r silesia.tar -B#`.

| Decoder  | -B1K ∆ | -B2K ∆ | -B4K ∆ | -B8K ∆ | -B16K ∆ | -B32K ∆ | -B64K ∆ | -B128K ∆ |
|----------|--------|--------|--------|--------|---------|---------|---------|----------|
| Fast C   | +11.2% | +8.2%  | +6.1%  | +4.4%  | +2.7%   | +1.5%   | +0.6%   | +0.2%    |
| Assembly | +12.5% | +9.0%  | +6.2%  | +3.6%  | +1.5%   | +0.7%   | +0.2%   | +0.03%   |

20 months ago[huf] Improve fast huffman decoding speed in linux kernel
Nick Terrell [Sat, 18 Nov 2023 02:20:19 +0000 (18:20 -0800)] 
[huf] Improve fast huffman decoding speed in linux kernel

gcc in the linux kernel was not unrolling the inner loops of the Huffman
decoder, which was destroying decoding performance. The compiler was
generating crazy code with all sorts of branches. I suspect because of
Spectre mitigations, but I'm not certain. Once the loops were manually
unrolled, performance was restored.

Additionally, when gcc couldn't prove that the variable left shift in
the 4X2 decode loop wasn't greater than 63, it inserted checks to verify
it. To fix this, mask `entry.nbBits & 0x3F`, which allows gcc to eliete
this check. This is a no op, because `entry.nbBits` is guaranteed to be
less than 64.

Lastly, introduce the `HUF_DISABLE_FAST_DECODE` macro to disable the
fast C loops for Issue #3762. So if even after this change, there is a
performance regression, users can opt-out at compile time.

20 months ago[debug] Don't define g_debuglevel in the kernel
Nick Terrell [Fri, 17 Nov 2023 01:15:25 +0000 (17:15 -0800)] 
[debug] Don't define g_debuglevel in the kernel

We only use this constant when `DEBUGLEVEL>=2`, but we get
-Werror=pedantic errors for empty translation units, so still define it
except in kernel environments.

Backport from the kernel:

https://lore.kernel.org/lkml/20230616144400.172683-1-ben.dooks@codethink.co.uk/

20 months ago[linux] Remove usage of deprecated function
Nick Terrell [Fri, 17 Nov 2023 00:53:44 +0000 (16:53 -0800)] 
[linux] Remove usage of deprecated function

ZSTD_resetDStream() is deprecated and replaced by ZSTD_DCtx_reset().
This removes deprecation warnings from the kernel build.

This change is a no-op, see the docs suggesting this replacement.

https://github.com/facebook/zstd/blob/fcbf2fde9ac7ce1562c7b3a394350e764bcb580f/lib/zstd.h#L2655-L2663

20 months ago[linux] Backport intptr_t removal
Nick Terrell [Thu, 16 Nov 2023 20:00:11 +0000 (12:00 -0800)] 
[linux] Backport intptr_t removal

Linux started providing intptr_t in <linux/types.h> so we no longer need
to define it here.

https://lkml.kernel.org/r/ed66b9e4-1fb7-45be-9bb9-d4bc291c691f@p183

20 months agoupdate license text 3820/head
Yann Collet [Fri, 17 Nov 2023 00:19:25 +0000 (16:19 -0800)] 
update license text

20 months agoMerge pull request #3807 from gruenich/increase_minimum_cmake_version_3.5
Yann Collet [Tue, 14 Nov 2023 18:48:11 +0000 (10:48 -0800)] 
Merge pull request #3807 from gruenich/increase_minimum_cmake_version_3.5

[cmake] Require CMake version 3.5 or newer

20 months agofix the copyright linter
Yann Collet [Mon, 13 Nov 2023 23:46:59 +0000 (15:46 -0800)] 
fix the copyright linter

20 months agoupdate xxhash to v0.8.2
Yann Collet [Mon, 13 Nov 2023 23:42:07 +0000 (15:42 -0800)] 
update xxhash to v0.8.2

List of updates : https://github.com/Cyan4973/xxHash/releases/tag/v0.8.2

This is also a preparation task before taking care of #3819

20 months agoMerge pull request #3813 from elasota/overflow-clarification
Yann Collet [Mon, 13 Nov 2023 08:03:42 +0000 (00:03 -0800)] 
Merge pull request #3813 from elasota/overflow-clarification

Clarify that a stream containing too many Huffman weights is invalid

20 months agoMerge pull request #3812 from jondo2010/bazel_build_doc
Yann Collet [Mon, 13 Nov 2023 08:03:10 +0000 (00:03 -0800)] 
Merge pull request #3812 from jondo2010/bazel_build_doc

Add Bazel module instructions to README.md

20 months agoClarify that a non-zero probability for an invalid symbol is invalid 3817/head
elasota [Mon, 13 Nov 2023 05:03:56 +0000 (00:03 -0500)] 
Clarify that a non-zero probability for an invalid symbol is invalid

20 months agoClarify that decoding too many Huffman weights is a failure condition 3813/head
elasota [Thu, 9 Nov 2023 01:06:58 +0000 (20:06 -0500)] 
Clarify that decoding too many Huffman weights is a failure condition

20 months agoAdd Bazel module instructions to README.md 3812/head
John Hughes [Wed, 8 Nov 2023 09:08:21 +0000 (09:08 +0000)] 
Add Bazel module instructions to README.md

21 months ago[cmake] Remove code for compatibility with CMake < 3.0 3807/head
Christoph Grüninger [Wed, 1 Nov 2023 07:51:45 +0000 (08:51 +0100)] 
[cmake] Remove code for compatibility with CMake < 3.0

The required version of CMake is now 3.5.

21 months agoMerge pull request #3806 from elasota/fmt-clarifications
Yann Collet [Wed, 1 Nov 2023 17:59:54 +0000 (10:59 -0700)] 
Merge pull request #3806 from elasota/fmt-clarifications

Correct FSE probability bit consumption in specification

21 months ago[cmake] Require CMake version 3.5 or newer
Christoph Grüninger [Tue, 31 Oct 2023 22:30:43 +0000 (23:30 +0100)] 
[cmake] Require CMake version 3.5 or newer

More recent versions of CMake emit the following warning:
CMake Deprecation Warning at cmake/CMakeLists.txt:10 (cmake_minimum_required):
  Compatibility with CMake < 3.5 will be removed from a future version of
  CMake.

  Update the VERSION argument <min> value or use a ...<max> suffix to tell
  CMake that the project does not need compatibility with older versions.

21 months agoMerge pull request #3800 from facebook/dependabot/github_actions/actions/checkout...
Yann Collet [Tue, 31 Oct 2023 16:23:47 +0000 (09:23 -0700)] 
Merge pull request #3800 from facebook/dependabot/github_actions/actions/checkout-4.1.1

Bump actions/checkout from 4.1.0 to 4.1.1

21 months agoMerge pull request #3795 from Saverio976/doc/cmake-fetch-content
Yann Collet [Tue, 31 Oct 2023 16:23:28 +0000 (09:23 -0700)] 
Merge pull request #3795 from Saverio976/doc/cmake-fetch-content

Add doc on how to use it with cmake FetchContent

21 months agoAdd definition of "log2sup" function 3806/head
elasota [Tue, 31 Oct 2023 15:42:00 +0000 (11:42 -0400)] 
Add definition of "log2sup" function

21 months agoClarify that the log2 of the largest possible symbol is the maximum number of bits...
elasota [Tue, 31 Oct 2023 05:17:23 +0000 (01:17 -0400)] 
Clarify that the log2 of the largest possible symbol is the maximum number of bits consumed

21 months agoAdd target_include_directories because windows and macos need it for me 3795/head
Xavier Mitault [Mon, 23 Oct 2023 15:38:09 +0000 (17:38 +0200)] 
Add target_include_directories because windows and macos need it for me

21 months agoBump actions/checkout from 4.1.0 to 4.1.1 3800/head
dependabot[bot] [Mon, 23 Oct 2023 05:56:05 +0000 (05:56 +0000)] 
Bump actions/checkout from 4.1.0 to 4.1.1

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.1.0 to 4.1.1.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/8ade135a41bc03ea155e62e844d188df1ea18608...b4ffde65f46336ab88eb53be808477a3936bae11)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
21 months agoMerge pull request #3789 from facebook/fix_flexArray_fse
Yann Collet [Fri, 20 Oct 2023 19:40:13 +0000 (12:40 -0700)] 
Merge pull request #3789 from facebook/fix_flexArray_fse

solving flexArray issue #3785 in fse

21 months agorevert to manually defining DTable 3789/head
Yann Collet [Thu, 19 Oct 2023 05:45:57 +0000 (22:45 -0700)] 
revert to manually defining DTable

thus avoiding the analyzer and ubsan to associate DTable to a size of 1.

21 months agobaby-step towards solving flexArray issue #3785
Yann Collet [Mon, 9 Oct 2023 23:47:52 +0000 (16:47 -0700)] 
baby-step towards solving flexArray issue #3785

the flexArray in structure FSE_DecompressWksp
is just a way to derive a pointer easily,
without risk/complexity of calculating it manually.

Not sure if this change is good enough to avoid ubsan warnings though.

21 months agoAdd doc on how to use it with cmake FetchContent
Xavier Mitault [Wed, 18 Oct 2023 11:22:15 +0000 (13:22 +0200)] 
Add doc on how to use it with cmake FetchContent

21 months agoMerge pull request #3786 from facebook/fix_flexarray_cctx
Yann Collet [Fri, 13 Oct 2023 16:08:20 +0000 (09:08 -0700)] 
Merge pull request #3786 from facebook/fix_flexarray_cctx

Remove FlexArray pattern from ZSTDMT

21 months agoextended the fix to ZSTDMT's Buffer Pool 3786/head
Yann Collet [Sun, 8 Oct 2023 07:25:17 +0000 (00:25 -0700)] 
extended the fix to ZSTDMT's Buffer Pool

21 months agoremoved unused macro constant
Yann Collet [Sun, 8 Oct 2023 06:32:22 +0000 (23:32 -0700)] 
removed unused macro constant

21 months agofixes suggested by @ebiggers
Yann Collet [Sun, 8 Oct 2023 06:29:42 +0000 (23:29 -0700)] 
fixes suggested by @ebiggers

21 months agoremoved FlexArray pattern from CCtxPool
Yann Collet [Sun, 8 Oct 2023 03:19:45 +0000 (20:19 -0700)] 
removed FlexArray pattern from CCtxPool

within ZSTDMT_.
This pattern is flagged by less forgiving variants of ubsan
notably used during compilation of the Linux Kernel.

There are 2 other places in the code where this pattern is used.
This fixes just one of them.

21 months agoMerge pull request #3752 from paulmenzel/fix-pzstd-makefile
Yann Collet [Sun, 8 Oct 2023 02:32:22 +0000 (19:32 -0700)] 
Merge pull request #3752 from paulmenzel/fix-pzstd-makefile

Fix pzstd Makefile to allow setting `DESTDIR` and `BINDIR` separately

21 months agoMerge pull request #3763 from dloidolt/fix_lib/README.md
Yann Collet [Sun, 8 Oct 2023 02:31:36 +0000 (19:31 -0700)] 
Merge pull request #3763 from dloidolt/fix_lib/README.md

Fix a very small formatting typo in the lib/README.md file

21 months agoMerge pull request #3772 from DimitriPapadopoulos/WIN32
Yann Collet [Sun, 8 Oct 2023 02:30:18 +0000 (19:30 -0700)] 
Merge pull request #3772 from DimitriPapadopoulos/WIN32

Do not test WIN32, instead test _WIN32

21 months agoMerge pull request #3771 from DimitriPapadopoulos/codespell
Yann Collet [Sun, 8 Oct 2023 02:29:41 +0000 (19:29 -0700)] 
Merge pull request #3771 from DimitriPapadopoulos/codespell

Fix new typos found by codespell

21 months agoMerge pull request #3777 from facebook/fix_x32 3760/head
Yann Collet [Mon, 2 Oct 2023 23:25:43 +0000 (16:25 -0700)] 
Merge pull request #3777 from facebook/fix_x32

fix x32 tests on Github CI

22 months agoMerge pull request #3777 from facebook/fix_x32
Yann Collet [Mon, 2 Oct 2023 23:25:43 +0000 (16:25 -0700)] 
Merge pull request #3777 from facebook/fix_x32

fix x32 tests on Github CI

22 months agoStop suppressing pointer-overflow UBSAN errors
Nick Terrell [Wed, 27 Sep 2023 00:53:26 +0000 (17:53 -0700)] 
Stop suppressing pointer-overflow UBSAN errors

* Remove all pointer-overflow suppressions from our UBSAN builds/tests.
* Add `ZSTD_ALLOW_POINTER_OVERFLOW_ATTR` macro to suppress
  pointer-overflow at a per-function level. This is a superior approach
  because it also applies to users who build zstd with UBSAN.
* Add `ZSTD_wrappedPtr{Diff,Add,Sub}()` that use these suppressions.
  The end goal is to only tag these functions with
  `ZSTD_ALLOW_POINTER_OVERFLOW`. But we can start by annoting functions
  that rely on pointer overflow, and gradually transition to using
  these.
* Add `ZSTD_maybeNullPtrAdd()` to simplify pointer addition when the
  pointer may be `NULL`.
* Fix all the fuzzer issues that came up. I'm sure there will be a lot
  more, but these are the ones that came up within a few minutes of
  running the fuzzers, and while running GitHub CI.

22 months agoRevert "Work around nullptr-with-nonzero-offset warning"
Nick Terrell [Fri, 25 Aug 2023 18:18:56 +0000 (11:18 -0700)] 
Revert "Work around nullptr-with-nonzero-offset warning"

This reverts commit c27fa399042f466080e79bb4fd8a4871bc0bcf28.

22 months agofix x32 tests on Github CI 3777/head
Yann Collet [Thu, 28 Sep 2023 04:18:20 +0000 (21:18 -0700)] 
fix x32 tests on Github CI

ubuntu-22.04 seems to have problems with x32 recently
switching to ubuntu-20.04 which seems to work fine so far

https://github.com/actions/runner-images/issues/8397

22 months agoMerge pull request #3774 from facebook/dependabot/github_actions/actions/checkout...
Yann Collet [Mon, 25 Sep 2023 06:34:06 +0000 (23:34 -0700)] 
Merge pull request #3774 from facebook/dependabot/github_actions/actions/checkout-4.1.0

Bump actions/checkout from 4.0.0 to 4.1.0

22 months agoBump actions/checkout from 4.0.0 to 4.1.0 3774/head
dependabot[bot] [Mon, 25 Sep 2023 05:18:07 +0000 (05:18 +0000)] 
Bump actions/checkout from 4.0.0 to 4.1.0

Bumps [actions/checkout](https://github.com/actions/checkout) from 4.0.0 to 4.1.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/3df4ab11eba7bda6032a0b82a6bb43b11571feac...8ade135a41bc03ea155e62e844d188df1ea18608)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>
22 months agoDo not test WIN32, instead test _WIN32 3772/head
Dimitri Papadopoulos [Sat, 23 Sep 2023 17:03:18 +0000 (19:03 +0200)] 
Do not test WIN32, instead test _WIN32

To the best of my knowledge:
* `_WIN32` and `_WIN64` are defined by the compiler,
* `WIN32` and `WIN64` are defined by the user, to indicate whatever
  the user chooses them to indicate. They mean 32-bit and 64-bit Windows
  compilation by convention only.

See:
https://accu.org/journals/overload/24/132/wilson_2223/

Windows compilers in general, and MSVC in particular, have been defining
`_WIN32` and `_WIN64` for a long time, provably at least since Visual Studio
2015, and in practice as early as in the days of 16-bit Windows.

See:
https://learn.microsoft.com/en-us/cpp/preprocessor/predefined-macros?view=msvc-140
https://learn.microsoft.com/en-us/windows/win32/winprog64/the-tools

Tests used to be inconsistent, sometimes testing `_WIN32`, sometimes
`_WIN32` and `WIN32`. This brings consistency to Windows detection.

22 months agoFix new typos found by codespell 3771/head
Dimitri Papadopoulos [Sat, 23 Sep 2023 16:56:01 +0000 (18:56 +0200)] 
Fix new typos found by codespell

22 months agoImprove macro guards for ZSTD_assertValidSequence
Nick Terrell [Fri, 22 Sep 2023 18:51:15 +0000 (11:51 -0700)] 
Improve macro guards for ZSTD_assertValidSequence

Refine the macro guards to define the functions exactly when they are
needed.

This fixes the chromium build with zstd.

Thanks to @GregTho for reporting!

22 months agoFix a very small formatting typo in the lib/README.md file 3763/head
Dominik Loidolt [Tue, 19 Sep 2023 14:22:47 +0000 (16:22 +0200)] 
Fix a very small formatting typo in the lib/README.md file

22 months agoMerge pull request #3755 from facebook/estimate_doc
Yann Collet [Tue, 19 Sep 2023 05:32:40 +0000 (22:32 -0700)] 
Merge pull request #3755 from facebook/estimate_doc

added some documentation on ZSTD_estimate*Size() variants

22 months agoadded some documentation on ZSTD_estimate*Size() variants 3755/head
Yann Collet [Wed, 13 Sep 2023 18:35:19 +0000 (11:35 -0700)] 
added some documentation on ZSTD_estimate*Size() variants

as a follow up for #3747

22 months agoMerge pull request #3749 from facebook/dependabot/github_actions/actions/checkout...
Yann Collet [Tue, 12 Sep 2023 20:58:08 +0000 (13:58 -0700)] 
Merge pull request #3749 from facebook/dependabot/github_actions/actions/checkout-4.0.0

Bump actions/checkout from 3.5.3 to 4.0.0

22 months agoMerge pull request #3745 from klausholstjacobsen/qnx-support
Yann Collet [Tue, 12 Sep 2023 20:43:59 +0000 (13:43 -0700)] 
Merge pull request #3745 from klausholstjacobsen/qnx-support

Added qnx in the posix test section of platform.h

22 months agoMerge pull request #3750 from facebook/dependabot/github_actions/actions/upload-artif...
Yann Collet [Tue, 12 Sep 2023 20:43:00 +0000 (13:43 -0700)] 
Merge pull request #3750 from facebook/dependabot/github_actions/actions/upload-artifact-3.1.3

Bump actions/upload-artifact from 3.1.2 to 3.1.3

22 months ago[pzstd]: Fix `DESTDIR` handling to allow setting `BINDIR` 3752/head
Paul Menzel [Tue, 12 Sep 2023 17:12:19 +0000 (19:12 +0200)] 
[pzstd]: Fix `DESTDIR` handling to allow setting `BINDIR`

Currently, setting `BINDIR` and `DESTDIR` separately is not possible, so
the command below fails, as BINDIR is set explicitly:

    $ make -j80 install PREFIX=/usr EPREFIX=/usr BINDIR=/usr/bin SBINDIR=/usr/sbin LIBEXECDIR=/usr/libexec SYSCONFDIR=/etc SHAREDSTATEDIR=/var LOCALST ATEDIR=/var LIBDIR=/usr/lib INCLUDEDIR=/usr/include DATAROOTDIR=/usr/share DATADIR=/usr/share INFODIR=/usr/share/info LOCALEDIR=/usr/share/locale MAND IR=/usr/share/man DOCDIR=/usr/share/doc/zstd DESTDIR=/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/image -C contrib/pzstd DESTDIR=/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/image
    make: Entering directory
    '/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/source/contrib/pzstd' CFLAGS="  -I../../lib -I../../lib/common -I../../programs -I. -DNDEBUG -O3 -Wall -Wextra -Wno-deprecated-declarations   " LDFLAGS=" -O3 -Wall -Wextra -pedantic  " make -C ../../lib libzstd.a
    make[1]: Entering directory '/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/source/lib'
    make[1]: Leaving directory '/dev/shm/bee-pmenzel/zstd/zstd-1.5.5-0/source/lib'
    g++ main.o ../../programs/util.o Options.o Pzstd.o SkippableFrame.o ../../lib/libzstd.a           -O3 -Wall -Wextra -pedantic    -pthread -o pzstd
    install -d -m 755 /usr/bin/
    install -m 755 pzstd /usr/bin/pzstd
    install: cannot create regular file '/usr/bin/pzstd': Permission denied
    make: *** [Makefile:116: install] Error 1

So, do not prefix `BINDIR` with `DESTDIR`, and adapt all paths for
installation. This is more common, and, for example, `programs/Makefile`
does the same.

Fixes: 8b4e84249b ("[pzstd] Fix Makefile")
22 months agoBump actions/upload-artifact from 3.1.2 to 3.1.3 3750/head
dependabot[bot] [Mon, 11 Sep 2023 05:09:05 +0000 (05:09 +0000)] 
Bump actions/upload-artifact from 3.1.2 to 3.1.3

Bumps [actions/upload-artifact](https://github.com/actions/upload-artifact) from 3.1.2 to 3.1.3.
- [Release notes](https://github.com/actions/upload-artifact/releases)
- [Commits](https://github.com/actions/upload-artifact/compare/0b7f8abb1508181956e8e162db84b466c27e18ce...a8a3f3ad30e3422c9c7b888a15615d19a852ae32)

---
updated-dependencies:
- dependency-name: actions/upload-artifact
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>
22 months agoBump actions/checkout from 3.5.3 to 4.0.0 3749/head
dependabot[bot] [Mon, 11 Sep 2023 05:09:02 +0000 (05:09 +0000)] 
Bump actions/checkout from 3.5.3 to 4.0.0

Bumps [actions/checkout](https://github.com/actions/checkout) from 3.5.3 to 4.0.0.
- [Release notes](https://github.com/actions/checkout/releases)
- [Changelog](https://github.com/actions/checkout/blob/main/CHANGELOG.md)
- [Commits](https://github.com/actions/checkout/compare/c85c95e3d7251135ab7dc9ce3241c5835cc595a9...3df4ab11eba7bda6032a0b82a6bb43b11571feac)

---
updated-dependencies:
- dependency-name: actions/checkout
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>
22 months agoMerge pull request #3739 from JohanMabille/cmake
Yann Collet [Sun, 10 Sep 2023 13:32:24 +0000 (06:32 -0700)] 
Merge pull request #3739 from JohanMabille/cmake

Fixed zstd cmake shared build on windows

22 months agoMerge pull request #3733 from ldv-alt/zdictlib_fix_prototype_mismatch
Yann Collet [Sun, 10 Sep 2023 13:31:52 +0000 (06:31 -0700)] 
Merge pull request #3733 from ldv-alt/zdictlib_fix_prototype_mismatch

zdictlib: fix prototype mismatch

23 months agoAdded qnx in the posix test section of platform.h 3745/head
klausholstjacobsen [Sun, 3 Sep 2023 08:10:23 +0000 (10:10 +0200)] 
Added qnx in the posix test section of platform.h

23 months agoFixed zstd cmake shared build on windows 3739/head
Johan Mabille [Fri, 25 Aug 2023 22:40:35 +0000 (00:40 +0200)] 
Fixed zstd cmake shared build on windows

23 months agoFix & refactor Huffman repeat tables for dictionaries
Nick Terrell [Thu, 24 Aug 2023 21:41:21 +0000 (14:41 -0700)] 
Fix & refactor Huffman repeat tables for dictionaries

The Huffman repeat mode checker assumed that the CTable was zeroed in the region `[maxSymbolValue + 1, 256)`.
This assumption didn't hold for tables built in the dictionaries, because it didn't go through the same codepath.

Since this code was originally written, we added a header to the CTable that specifies the `tableLog`.
Add `maxSymbolValue` to that header, and check that the table's `maxSymbolValue` is at least the block's `maxSymbolValue`.

This solution is cleaner because we write this header for every CTable we build, so it can't be missed in any code path.

Credit to OSS-Fuzz

23 months agoWork around nullptr-with-nonzero-offset warning
Nick Terrell [Thu, 24 Aug 2023 23:33:42 +0000 (16:33 -0700)] 
Work around nullptr-with-nonzero-offset warning

See comment.

23 months agozdictlib: fix prototype mismatch 3733/head
Dmitry V. Levin [Tue, 8 Aug 2023 08:00:00 +0000 (08:00 +0000)] 
zdictlib: fix prototype mismatch

Fix the following warnings reported by the compiler when
ZDICTLIB_STATIC_API is not defined to ZDICTLIB_API:

lib/dictBuilder/cover.c:1122:21: warning: redeclaration of 'ZDICT_optimizeTrainFromBuffer_cover' with different visibility (old visibility
preserved)
lib/dictBuilder/cover.c:736:21: warning: redeclaration of 'ZDICT_trainFromBuffer_cover' with different visibility (old visibility
+preserved)
lib/dictBuilder/fastcover.c:549:1: warning: redeclaration of 'ZDICT_trainFromBuffer_fastCover' with different visibility (old visibility
preserved)
lib/dictBuilder/fastcover.c:618:1: warning: redeclaration of 'ZDICT_optimizeTrainFromBuffer_fastCover' with different visibility (old
visibility preserved)

23 months agoMerge pull request #3720 from QBos07/cygwin-msys2-support
Yann Collet [Tue, 22 Aug 2023 23:29:34 +0000 (16:29 -0700)] 
Merge pull request #3720 from QBos07/cygwin-msys2-support

Updated Makefiles for full MSYS2 and Cygwin installation and testing …

23 months agoMerge pull request #3728 from 0o001/dev
Yann Collet [Tue, 22 Aug 2023 20:33:47 +0000 (13:33 -0700)] 
Merge pull request #3728 from 0o001/dev

fix: ZSTD_BUILD_DECOMPRESSION message

23 months agoNo longer reject dictionaries with literals maxSymbolValue < 255
Nick Terrell [Mon, 21 Aug 2023 18:33:29 +0000 (11:33 -0700)] 
No longer reject dictionaries with literals maxSymbolValue < 255

We already have logic in our Huffman encoder to validate Huffman tables with missing symbols.
We use this for higher compression levels to re-use the previous blocks statistics, or when the dictionaries table has zero-weighted symbols.
This check was leftover as an oversight from before we added validation for Huffman tables.

I validated that the `dictionary_loader` fuzzer has coverage of every line in the `ZSTD_loadCEntropy()` function to validate that it is correctly testing this function.

23 months agoMerge pull request #3726 from facebook/fullbench_dctx
Yann Collet [Mon, 21 Aug 2023 19:35:51 +0000 (12:35 -0700)] 
Merge pull request #3726 from facebook/fullbench_dctx

added ZSTD_decompressDCtx() benchmark option to fullbench

23 months agoMerge pull request #3730 from facebook/dependabot/github_actions/github/codeql-action...
Felix Handte [Mon, 21 Aug 2023 16:47:26 +0000 (09:47 -0700)] 
Merge pull request #3730 from facebook/dependabot/github_actions/github/codeql-action-2.21.4

Bump github/codeql-action from 2.20.3 to 2.21.4