added a faster block splitter variant

that samples 1 in 5 positions.

This variant is fast enough for lazy2 and btlazy2,
but it's less good in combination with post-splitter at higher levels (>= btopt).

minor split optimization

let's fill the initial stats directly into target fingerprint

updated regression test results

fixed workspace alignment on non 64-bit systems

splitter workspace is now provided by ZSTD_CCtx*

fix alignment test

for non 64-bit systems

fixed minor conversion warnings on Visual

ZSTD_splitBlock_4k() uses externally provided workspace

ideally, this workspace would be provided from the ZSTD_CCtx* state

replaced uasan32 test by asan32 test

fixing minor formatting issue in 32-bit mode with logs enabled

fixed VS2010 solution

new Makefile target mesonbuild

for easier local testing

fixed meson build

fixed zstreamtest

fixed c90 comment style

fix assert

only split full blocks

short term simplification

fixed single-library build

fixed kernel build

fixed RLE detection test

fix overlap write scenario in presence of incompressible data

more ZSTD_memset() to apply

minor C++-ism

though I really wonder if this is a property worth maintaining.

use ZSTD_memset()

for better portability on Linux kernel

do not use `new` as variable name

fixed strict C90 semantic

XP: add a pre-splitter

instead of ingesting only full blocks, make an analysis of data, and infer where to split.

Merge pull request #4171 from facebook/lvl3_ratio+

Improve compression ratio of levels 3 & 4

updated regression tests results

enable regression tests at pull request time

was transferred from circleci,
but was only triggered on push into dev,
i.e. after pull request is merged.

rewrite code in the manner suggested by @terrelln

fix test

a margin of 4 is insufficient to guarantee compression success.

slightly improved compression ratio at levels 3 & 4

The compression ratio benefits are small but consistent, i.e. always positive.
On `silesia.tar` corpus, this modification saves ~75 KB at level 3.
The measured speed cost is negligible, i.e. below noise level, between 0 and -1%.

Merge pull request #4170 from facebook/dict_cSpeed

Improve dictionary compression speed

fix test

minor improvement to level 3 dictionary compression ratio

small dictionary compression speed improvement

not as good as small-blocks improvement,
but generally positive.

Merge pull request #4167 from facebook/ci_m32test_faster

attempt to make 32-bit tests faster

attempt parallel test running with -j

measure if -O2 makes the test complete faster

attempt to make 32-bit tests faster

this is the longest CI test, reaching ~40mn on last PR

Merge pull request #4165 from facebook/cspeed_cmov

Improve compression speed on small blocks

rename variable name

findMatch -> matchFound
since it's a test, as opposed to an active search operation.
suggested by @terrelln

fixed parameter ordering in `dfast`

noticed by @terrelln

Merge pull request #4164 from facebook/spec_043

spec update: huffman prefix code paragraph

minor: better variable naming

added ascending order example

make __asm__ a __GNUC__ specific

store dummy bytes within ZSTD_match4Found_cmov()

feels more logical, better contained

introduce memory barrier to force test order

suggested by @terrelln

made search strategy switchable

between cmov and branch
and use a simple heuristic based on wlog to select between them.

note: performance is not good on clang (yet)

refactor search into an inline function

for easier swapping with a parameter

refactor huffman prefix code paragraph

minor refactor zstd_fast

make hot variables more local

Merge pull request #4160 from facebook/fix_nightly

fix dependency for nightly github actions tests

fix dependency for nightly github actions tests

Merge pull request #4159 from facebook/spec_refactor_fse

specification update

refactor documentation of the FSE decoding table build process

Merge pull request #4153 from artem/fix-meson-includes

meson: Do not export private headers in libzstd_dep to avoid name clash

Merge pull request #4156 from facebook/rm_circleci

removing nightly tests built on circleci

Merge pull request #4157 from facebook/fix_result_c

fix incorrect pointer manipulation

Merge pull request #4158 from facebook/benchzstd_fclose

fix missing fclose()

fix missing fclose()

fix #4151

fix incorrect pointer manipulation

fix #4155

removing nightly tests built on circleci

Merge pull request #4154 from dearblue/freebsd-14.1

Update FreeBSD VM image to 14.1

meson: Fix contrib and tests build

meson: Do not export private headers in libzstd_dep to avoid name clash

This way libzstd_dep does not override, for instance, <xxhash.h>

Update FreeBSD VM image to 14.1

FreeBSD 14.0 will reach the end of life on 2024-09-30.
The updated 14.1 is scheduled to end-of-life on 2025-03-31.

ref. https://www.freebsd.org/releases/14.2R/schedule/

Merge pull request #4146 from facebook/dictBench_Doc

update documentation: specify that Dictionary can be used for benchmark

Merge pull request #4013 from elasota/spec-clarify-offset-code-overflow

Specify that decoders may reject non-zero probabilities for larger offset codes than implementation supports

update documentation to specify that Dictionary can be used for benchmark

fix #4139

Included suggestion from @neheb

Fixed warning when compiling pzstd with CPPFLAGS=-Wunused-result and CXXFLAGS=-std=c++17

Merge pull request #4119 from xionghul/dev

Fix zstd-pgo run error

Merge pull request #4143 from facebook/fix_dictsizemin_dic

fix doc nit: ZDICT_DICTSIZE_MIN

Optimize compression by avoiding unpredictable branches

Avoid unpredictable branch. Use conditional move to generate the address
that is guaranteed to be safe and compare unconditionally.
Instead of

if (idx < limit && x[idx] == val ) // mispredicted idx < limit branch

Do

addr = cmov(safe,x+idx)
if (*addr == val && idx < limit) // almost always false so well predicted

Using microbenchmarks from https://github.com/google/fleetbench,
I get about ~10% speed-up:

name                                                                                          old cpu/op   new cpu/op    delta
BM_ZSTD_COMPRESS_Fleet/compression_level:-7/window_log:15                                     1.46ns ± 3%   1.31ns ± 7%   -9.88%  (p=0.000 n=35+38)
BM_ZSTD_COMPRESS_Fleet/compression_level:-7/window_log:16                                     1.41ns ± 3%   1.28ns ± 3%   -9.56%  (p=0.000 n=36+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-5/window_log:15                                     1.61ns ± 1%   1.43ns ± 3%  -10.70%  (p=0.000 n=30+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-5/window_log:16                                     1.54ns ± 2%   1.39ns ± 3%   -9.21%  (p=0.000 n=37+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-3/window_log:15                                     1.82ns ± 2%   1.61ns ± 3%  -11.31%  (p=0.000 n=37+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:-3/window_log:16                                     1.73ns ± 3%   1.56ns ± 3%   -9.50%  (p=0.000 n=38+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-1/window_log:15                                     2.12ns ± 2%   1.79ns ± 3%  -15.55%  (p=0.000 n=34+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-1/window_log:16                                     1.99ns ± 3%   1.72ns ± 3%  -13.70%  (p=0.000 n=38+38)
BM_ZSTD_COMPRESS_Fleet/compression_level:0/window_log:15                                      3.22ns ± 3%   2.94ns ± 3%   -8.67%  (p=0.000 n=38+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:0/window_log:16                                      3.19ns ± 4%   2.86ns ± 4%  -10.55%  (p=0.000 n=40+38)
BM_ZSTD_COMPRESS_Fleet/compression_level:1/window_log:15                                      2.60ns ± 3%   2.22ns ± 3%  -14.53%  (p=0.000 n=40+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:1/window_log:16                                      2.46ns ± 3%   2.13ns ± 2%  -13.67%  (p=0.000 n=39+36)
BM_ZSTD_COMPRESS_Fleet/compression_level:2/window_log:15                                      2.69ns ± 3%   2.46ns ± 3%   -8.63%  (p=0.000 n=37+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:2/window_log:16                                      2.63ns ± 3%   2.36ns ± 3%  -10.47%  (p=0.000 n=40+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:3/window_log:15                                      3.20ns ± 2%   2.95ns ± 3%   -7.94%  (p=0.000 n=35+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:3/window_log:16                                      3.20ns ± 4%   2.87ns ± 4%  -10.33%  (p=0.000 n=40+40)

I've also measured the impact on internal workloads and saw similar
~10% improvement in performance, measured by cpu usage/byte of data.

fix doc nit: ZDICT_DICTSIZE_MIN

fix #4142

Merge pull request #4129 from facebook/mitigate_32bit

Limit range of operations on Indexes in 32-bit mode

Limit range of operations on Indexes in 32-bit mode

and use unsigned type.
This reduce risks that an operation produces a negative number when crossing the 2 GB limit.

Merge pull request #4128 from facebook/dependabot/github_actions/github/codeql-action-3.26.2

Bump github/codeql-action from 3.25.1 to 3.26.2

Bump github/codeql-action from 3.25.1 to 3.26.2

Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.25.1 to 3.26.2.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/c7f9125735019aa87cfc361530512d50ea439c71...429e1977040da7a23b6822b13c129cd1ba93dbb2)

---
updated-dependencies:
- dependency-name: github/codeql-action
  dependency-type: direct:production
  update-type: version-update:semver-minor
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #4122 from facebook/dependabot/github_actions/actions/setup-java-4

Bump actions/setup-java from 3 to 4

Bump actions/setup-java from 3 to 4

Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v3...v4)

---
updated-dependencies:
- dependency-name: actions/setup-java
  dependency-type: direct:production
  update-type: version-update:semver-major
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #4115 from Adenilson/leak01

[zstd][leak] Avoid memory leak on early return of ZSTD_generateSequence

Merge pull request #4114 from Adenilson/trace01

[riscv] Enable support for weak symbols

Merge pull request #4113 from facebook/fix4110

formatString_u() can display numbers > 100

Fix zstd-pgo run error

The -Werror=missing-profile caused thread/zlib/lzma/lz4 detection failure
when build with profile-use, thus caused ZSTD_MULTITHREAD etc. is not
defined for profile-use, then there will be many profile mismatch information
in output and the final binary reports run error sometimes as below:

Error : ZSTD_CCtx_setParameter(ctx, ZSTD_c_nbWorkers, adv->nbWorkers) failed : Unsupported parameter

Signed-off-by: Xionghu Luo <xionghuluo@tencent.com>

Merge pull request #4109 from facebook/actionsTest

added android-ndk-build

[zstd][leak] Avoid memory leak on early return of ZSTD_generateSequence

Sanity checks on a few of the context parameters (i.e. workers and block size)
may prompt an early return on ZSTD_generateSequences.

Allocating the destination buffer past those return points avoids a potential
memory leak.

This patch should fix issue #4112.

[riscv] Enable support for weak symbols

Both gcc and clang support weak symbols on RISC-V, therefore
let's enable it.

This should fix issue #4069.

fix c90 comment style

formatString_u() can display numbers > 100

fixes #4110

Merge pull request #4111 from facebook/dependabot/github_actions/msys2/setup-msys2-2.24.1

Bump msys2/setup-msys2 from 2.24.0 to 2.24.1

Bump msys2/setup-msys2 from 2.24.0 to 2.24.1

Bumps [msys2/setup-msys2](https://github.com/msys2/setup-msys2) from 2.24.0 to 2.24.1.
- [Release notes](https://github.com/msys2/setup-msys2/releases)
- [Changelog](https://github.com/msys2/setup-msys2/blob/main/CHANGELOG.md)
- [Commits](https://github.com/msys2/setup-msys2/compare/5df0ca6cbf14efcd08f8d5bd5e049a3cc8e07fd2...ddf331adaebd714795f1042345e6ca57bd66cea8)

---
updated-dependencies:
- dependency-name: msys2/setup-msys2
  dependency-type: direct:production
  update-type: version-update:semver-patch
...

Signed-off-by: dependabot[bot] <support@github.com>

Merge pull request #4102 from proppy/decodecorpus-ext

tests/decodecorpus: add more advanced options

added android-ndk-build

Merge pull request #4107 from Adenilson/ndk01

[zstd][android] Fix build with NDK r27

[zstd][android] Fix build with NDK r27

The NDK cross compiler declares the target as __linux (which is
not technically incorrect), which triggers the enablement of _GNU_SOURCE
in the newly added code that requires the presence of qsort_r() used
in the COVER dictionary code.

Even though the NDK uses llvm/libc, it doesn't declare qsort_r()
in the stdlib.h header.

The build fix is to only activate the _GNU_SOURCE macro if the OS is
*not* Android, as then we will fallback to the C90 compliant code.

This patch should solve the reported issue number #4103.

Merge pull request #4094 from RubenKelevra/patchfrom_singlethread_man_update

clarify when to use '--single-thread' with '--patch-from'