]>
git.ipfire.org Git - thirdparty/zstd.git/log
Yann Collet [Mon, 21 Oct 2024 21:56:43 +0000 (14:56 -0700)]
added a faster block splitter variant
that samples 1 in 5 positions.
This variant is fast enough for lazy2 and btlazy2,
but it's less good in combination with post-splitter at higher levels (>= btopt).
Yann Collet [Mon, 21 Oct 2024 00:16:17 +0000 (17:16 -0700)]
minor split optimization
let's fill the initial stats directly into target fingerprint
Yann Collet [Mon, 21 Oct 2024 00:07:43 +0000 (17:07 -0700)]
updated regression test results
Yann Collet [Fri, 18 Oct 2024 23:47:41 +0000 (16:47 -0700)]
fixed workspace alignment on non 64-bit systems
Yann Collet [Fri, 18 Oct 2024 23:00:36 +0000 (16:00 -0700)]
splitter workspace is now provided by ZSTD_CCtx*
Yann Collet [Fri, 18 Oct 2024 18:20:54 +0000 (11:20 -0700)]
fix alignment test
for non 64-bit systems
Yann Collet [Fri, 18 Oct 2024 18:16:49 +0000 (11:16 -0700)]
fixed minor conversion warnings on Visual
Yann Collet [Fri, 18 Oct 2024 01:40:47 +0000 (18:40 -0700)]
ZSTD_splitBlock_4k() uses externally provided workspace
ideally, this workspace would be provided from the ZSTD_CCtx* state
Yann Collet [Fri, 18 Oct 2024 00:05:17 +0000 (17:05 -0700)]
replaced uasan32 test by asan32 test
Yann Collet [Thu, 17 Oct 2024 23:23:05 +0000 (16:23 -0700)]
fixing minor formatting issue in 32-bit mode with logs enabled
Yann Collet [Thu, 17 Oct 2024 22:44:51 +0000 (15:44 -0700)]
fixed VS2010 solution
Yann Collet [Thu, 17 Oct 2024 22:42:02 +0000 (15:42 -0700)]
new Makefile target mesonbuild
for easier local testing
Yann Collet [Thu, 17 Oct 2024 22:35:56 +0000 (15:35 -0700)]
fixed meson build
Yann Collet [Thu, 17 Oct 2024 22:16:57 +0000 (15:16 -0700)]
fixed zstreamtest
Yann Collet [Thu, 17 Oct 2024 21:52:49 +0000 (14:52 -0700)]
fixed c90 comment style
Yann Collet [Thu, 17 Oct 2024 21:50:46 +0000 (14:50 -0700)]
fix assert
Yann Collet [Thu, 17 Oct 2024 21:46:47 +0000 (14:46 -0700)]
only split full blocks
short term simplification
Yann Collet [Thu, 17 Oct 2024 21:41:26 +0000 (14:41 -0700)]
fixed single-library build
Yann Collet [Thu, 17 Oct 2024 20:26:57 +0000 (13:26 -0700)]
fixed kernel build
Yann Collet [Thu, 17 Oct 2024 20:21:55 +0000 (13:21 -0700)]
fixed RLE detection test
Yann Collet [Thu, 17 Oct 2024 19:55:08 +0000 (12:55 -0700)]
fix overlap write scenario in presence of incompressible data
Yann Collet [Tue, 3 Sep 2024 23:59:06 +0000 (16:59 -0700)]
more ZSTD_memset() to apply
Yann Collet [Tue, 3 Sep 2024 23:54:36 +0000 (16:54 -0700)]
minor C++-ism
though I really wonder if this is a property worth maintaining.
Yann Collet [Tue, 3 Sep 2024 23:52:10 +0000 (16:52 -0700)]
use ZSTD_memset()
for better portability on Linux kernel
Yann Collet [Tue, 3 Sep 2024 23:48:52 +0000 (16:48 -0700)]
do not use `new` as variable name
Yann Collet [Tue, 3 Sep 2024 23:44:30 +0000 (16:44 -0700)]
fixed strict C90 semantic
Yann Collet [Tue, 3 Sep 2024 21:35:29 +0000 (14:35 -0700)]
XP: add a pre-splitter
instead of ingesting only full blocks, make an analysis of data, and infer where to split.
Yann Collet [Thu, 17 Oct 2024 18:39:41 +0000 (11:39 -0700)]
Merge pull request #4171 from facebook/lvl3_ratio+
Improve compression ratio of levels 3 & 4
Yann Collet [Thu, 17 Oct 2024 17:50:28 +0000 (10:50 -0700)]
updated regression tests results
Yann Collet [Thu, 17 Oct 2024 16:45:16 +0000 (09:45 -0700)]
enable regression tests at pull request time
was transferred from circleci,
but was only triggered on push into dev,
i.e. after pull request is merged.
Yann Collet [Thu, 17 Oct 2024 16:36:56 +0000 (09:36 -0700)]
rewrite code in the manner suggested by @terrelln
Yann Collet [Wed, 16 Oct 2024 19:55:52 +0000 (12:55 -0700)]
fix test
a margin of 4 is insufficient to guarantee compression success.
Yann Collet [Wed, 16 Oct 2024 19:13:57 +0000 (12:13 -0700)]
slightly improved compression ratio at levels 3 & 4
The compression ratio benefits are small but consistent, i.e. always positive.
On `silesia.tar` corpus, this modification saves ~75 KB at level 3.
The measured speed cost is negligible, i.e. below noise level, between 0 and -1%.
Yann Collet [Thu, 17 Oct 2024 00:36:49 +0000 (17:36 -0700)]
Merge pull request #4170 from facebook/dict_cSpeed
Improve dictionary compression speed
Yann Collet [Wed, 16 Oct 2024 01:44:40 +0000 (18:44 -0700)]
fix test
Yann Collet [Wed, 16 Oct 2024 00:58:33 +0000 (17:58 -0700)]
minor improvement to level 3 dictionary compression ratio
Yann Collet [Wed, 16 Oct 2024 00:46:01 +0000 (17:46 -0700)]
small dictionary compression speed improvement
not as good as small-blocks improvement,
but generally positive.
Yann Collet [Sat, 12 Oct 2024 08:57:55 +0000 (01:57 -0700)]
Merge pull request #4167 from facebook/ci_m32test_faster
attempt to make 32-bit tests faster
Yann Collet [Sat, 12 Oct 2024 01:01:28 +0000 (18:01 -0700)]
attempt parallel test running with -j
Yann Collet [Sat, 12 Oct 2024 00:30:55 +0000 (17:30 -0700)]
measure if -O2 makes the test complete faster
Yann Collet [Fri, 11 Oct 2024 23:24:25 +0000 (16:24 -0700)]
attempt to make 32-bit tests faster
this is the longest CI test, reaching ~40mn on last PR
Yann Collet [Fri, 11 Oct 2024 23:20:19 +0000 (16:20 -0700)]
Merge pull request #4165 from facebook/cspeed_cmov
Improve compression speed on small blocks
Yann Collet [Fri, 11 Oct 2024 22:38:12 +0000 (15:38 -0700)]
rename variable name
findMatch -> matchFound
since it's a test, as opposed to an active search operation.
suggested by @terrelln
Yann Collet [Fri, 11 Oct 2024 22:36:15 +0000 (15:36 -0700)]
fixed parameter ordering in `dfast`
noticed by @terrelln
Yann Collet [Thu, 10 Oct 2024 23:56:02 +0000 (16:56 -0700)]
Merge pull request #4164 from facebook/spec_043
spec update: huffman prefix code paragraph
Yann Collet [Thu, 10 Oct 2024 23:07:20 +0000 (16:07 -0700)]
minor: better variable naming
Yann Collet [Wed, 9 Oct 2024 08:06:24 +0000 (01:06 -0700)]
added ascending order example
Yann Collet [Tue, 8 Oct 2024 23:38:35 +0000 (16:38 -0700)]
make __asm__ a __GNUC__ specific
Yann Collet [Tue, 8 Oct 2024 23:02:54 +0000 (16:02 -0700)]
store dummy bytes within ZSTD_match4Found_cmov()
feels more logical, better contained
Yann Collet [Tue, 8 Oct 2024 22:54:48 +0000 (15:54 -0700)]
introduce memory barrier to force test order
suggested by @terrelln
Yann Collet [Tue, 8 Oct 2024 18:43:07 +0000 (11:43 -0700)]
made search strategy switchable
between cmov and branch
and use a simple heuristic based on wlog to select between them.
note: performance is not good on clang (yet)
Yann Collet [Tue, 8 Oct 2024 18:10:48 +0000 (11:10 -0700)]
refactor search into an inline function
for easier swapping with a parameter
Yann Collet [Tue, 8 Oct 2024 00:15:07 +0000 (17:15 -0700)]
refactor huffman prefix code paragraph
Yann Collet [Mon, 7 Oct 2024 18:22:40 +0000 (11:22 -0700)]
minor refactor zstd_fast
make hot variables more local
Yann Collet [Fri, 4 Oct 2024 04:02:39 +0000 (21:02 -0700)]
Merge pull request #4160 from facebook/fix_nightly
fix dependency for nightly github actions tests
Yann Collet [Thu, 3 Oct 2024 22:10:16 +0000 (15:10 -0700)]
fix dependency for nightly github actions tests
Yann Collet [Thu, 3 Oct 2024 21:54:16 +0000 (14:54 -0700)]
Merge pull request #4159 from facebook/spec_refactor_fse
specification update
Yann Collet [Thu, 3 Oct 2024 05:57:36 +0000 (22:57 -0700)]
refactor documentation of the FSE decoding table build process
Yann Collet [Wed, 2 Oct 2024 23:51:44 +0000 (16:51 -0700)]
Merge pull request #4153 from artem/fix-meson-includes
meson: Do not export private headers in libzstd_dep to avoid name clash
Yann Collet [Wed, 2 Oct 2024 23:51:15 +0000 (16:51 -0700)]
Merge pull request #4156 from facebook/rm_circleci
removing nightly tests built on circleci
Yann Collet [Wed, 2 Oct 2024 23:50:45 +0000 (16:50 -0700)]
Merge pull request #4157 from facebook/fix_result_c
fix incorrect pointer manipulation
Yann Collet [Wed, 2 Oct 2024 23:49:43 +0000 (16:49 -0700)]
Merge pull request #4158 from facebook/benchzstd_fclose
fix missing fclose()
Yann Collet [Tue, 1 Oct 2024 16:52:45 +0000 (09:52 -0700)]
fix missing fclose()
fix #4151
Yann Collet [Tue, 1 Oct 2024 16:25:26 +0000 (09:25 -0700)]
fix incorrect pointer manipulation
fix #4155
Yann Collet [Tue, 1 Oct 2024 04:38:29 +0000 (21:38 -0700)]
removing nightly tests built on circleci
Yann Collet [Mon, 30 Sep 2024 18:54:32 +0000 (11:54 -0700)]
Merge pull request #4154 from dearblue/freebsd-14.1
Update FreeBSD VM image to 14.1
Artem Labazov [Mon, 30 Sep 2024 15:05:57 +0000 (18:05 +0300)]
meson: Fix contrib and tests build
Artem Labazov [Mon, 30 Sep 2024 14:00:42 +0000 (17:00 +0300)]
meson: Do not export private headers in libzstd_dep to avoid name clash
This way libzstd_dep does not override, for instance, <xxhash.h>
dearblue [Mon, 30 Sep 2024 13:45:17 +0000 (22:45 +0900)]
Update FreeBSD VM image to 14.1
FreeBSD 14.0 will reach the end of life on 2024-09-30.
The updated 14.1 is scheduled to end-of-life on 2025-03-31.
ref. https://www.freebsd.org/releases/14.2R/schedule/
Yann Collet [Fri, 27 Sep 2024 20:44:42 +0000 (13:44 -0700)]
Merge pull request #4146 from facebook/dictBench_Doc
update documentation: specify that Dictionary can be used for benchmark
Yann Collet [Fri, 27 Sep 2024 20:42:32 +0000 (13:42 -0700)]
Merge pull request #4013 from elasota/spec-clarify-offset-code-overflow
Specify that decoders may reject non-zero probabilities for larger offset codes than implementation supports
Yann Collet [Wed, 25 Sep 2024 23:56:01 +0000 (16:56 -0700)]
update documentation to specify that Dictionary can be used for benchmark
fix #4139
inventor500 [Wed, 25 Sep 2024 15:21:20 +0000 (10:21 -0500)]
Included suggestion from @neheb
inventor500 [Sat, 31 Aug 2024 12:23:51 +0000 (07:23 -0500)]
Fixed warning when compiling pzstd with CPPFLAGS=-Wunused-result and CXXFLAGS=-std=c++17
Yann Collet [Wed, 25 Sep 2024 00:55:43 +0000 (17:55 -0700)]
Merge pull request #4119 from xionghul/dev
Fix zstd-pgo run error
Yann Collet [Wed, 25 Sep 2024 00:55:25 +0000 (17:55 -0700)]
Merge pull request #4143 from facebook/fix_dictsizemin_dic
fix doc nit: ZDICT_DICTSIZE_MIN
Ilya Tokar [Wed, 18 Sep 2024 21:36:37 +0000 (17:36 -0400)]
Optimize compression by avoiding unpredictable branches
Avoid unpredictable branch. Use conditional move to generate the address
that is guaranteed to be safe and compare unconditionally.
Instead of
if (idx < limit && x[idx] == val ) // mispredicted idx < limit branch
Do
addr = cmov(safe,x+idx)
if (*addr == val && idx < limit) // almost always false so well predicted
Using microbenchmarks from https://github.com/google/fleetbench,
I get about ~10% speed-up:
name old cpu/op new cpu/op delta
BM_ZSTD_COMPRESS_Fleet/compression_level:-7/window_log:15 1.46ns ± 3% 1.31ns ± 7% -9.88% (p=0.000 n=35+38)
BM_ZSTD_COMPRESS_Fleet/compression_level:-7/window_log:16 1.41ns ± 3% 1.28ns ± 3% -9.56% (p=0.000 n=36+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-5/window_log:15 1.61ns ± 1% 1.43ns ± 3% -10.70% (p=0.000 n=30+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-5/window_log:16 1.54ns ± 2% 1.39ns ± 3% -9.21% (p=0.000 n=37+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-3/window_log:15 1.82ns ± 2% 1.61ns ± 3% -11.31% (p=0.000 n=37+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:-3/window_log:16 1.73ns ± 3% 1.56ns ± 3% -9.50% (p=0.000 n=38+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-1/window_log:15 2.12ns ± 2% 1.79ns ± 3% -15.55% (p=0.000 n=34+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:-1/window_log:16 1.99ns ± 3% 1.72ns ± 3% -13.70% (p=0.000 n=38+38)
BM_ZSTD_COMPRESS_Fleet/compression_level:0/window_log:15 3.22ns ± 3% 2.94ns ± 3% -8.67% (p=0.000 n=38+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:0/window_log:16 3.19ns ± 4% 2.86ns ± 4% -10.55% (p=0.000 n=40+38)
BM_ZSTD_COMPRESS_Fleet/compression_level:1/window_log:15 2.60ns ± 3% 2.22ns ± 3% -14.53% (p=0.000 n=40+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:1/window_log:16 2.46ns ± 3% 2.13ns ± 2% -13.67% (p=0.000 n=39+36)
BM_ZSTD_COMPRESS_Fleet/compression_level:2/window_log:15 2.69ns ± 3% 2.46ns ± 3% -8.63% (p=0.000 n=37+39)
BM_ZSTD_COMPRESS_Fleet/compression_level:2/window_log:16 2.63ns ± 3% 2.36ns ± 3% -10.47% (p=0.000 n=40+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:3/window_log:15 3.20ns ± 2% 2.95ns ± 3% -7.94% (p=0.000 n=35+40)
BM_ZSTD_COMPRESS_Fleet/compression_level:3/window_log:16 3.20ns ± 4% 2.87ns ± 4% -10.33% (p=0.000 n=40+40)
I've also measured the impact on internal workloads and saw similar
~10% improvement in performance, measured by cpu usage/byte of data.
Yann Collet [Thu, 19 Sep 2024 16:50:30 +0000 (09:50 -0700)]
fix doc nit: ZDICT_DICTSIZE_MIN
fix #4142
Yann Collet [Thu, 22 Aug 2024 18:00:50 +0000 (11:00 -0700)]
Merge pull request #4129 from facebook/mitigate_32bit
Limit range of operations on Indexes in 32-bit mode
Yann Collet [Wed, 21 Aug 2024 17:53:25 +0000 (10:53 -0700)]
Limit range of operations on Indexes in 32-bit mode
and use unsigned type.
This reduce risks that an operation produces a negative number when crossing the 2 GB limit.
Yann Collet [Mon, 19 Aug 2024 20:54:06 +0000 (13:54 -0700)]
Merge pull request #4128 from facebook/dependabot/github_actions/github/codeql-action-3.26.2
Bump github/codeql-action from 3.25.1 to 3.26.2
dependabot[bot] [Mon, 19 Aug 2024 05:37:33 +0000 (05:37 +0000)]
Bump github/codeql-action from 3.25.1 to 3.26.2
Bumps [github/codeql-action](https://github.com/github/codeql-action) from 3.25.1 to 3.26.2.
- [Release notes](https://github.com/github/codeql-action/releases)
- [Changelog](https://github.com/github/codeql-action/blob/main/CHANGELOG.md)
- [Commits](https://github.com/github/codeql-action/compare/
c7f9125735019aa87cfc361530512d50ea439c71 ...
429e1977040da7a23b6822b13c129cd1ba93dbb2 )
---
updated-dependencies:
- dependency-name: github/codeql-action
dependency-type: direct:production
update-type: version-update:semver-minor
...
Signed-off-by: dependabot[bot] <support@github.com>
Yann Collet [Mon, 12 Aug 2024 06:07:43 +0000 (23:07 -0700)]
Merge pull request #4122 from facebook/dependabot/github_actions/actions/setup-java-4
Bump actions/setup-java from 3 to 4
dependabot[bot] [Mon, 12 Aug 2024 05:16:05 +0000 (05:16 +0000)]
Bump actions/setup-java from 3 to 4
Bumps [actions/setup-java](https://github.com/actions/setup-java) from 3 to 4.
- [Release notes](https://github.com/actions/setup-java/releases)
- [Commits](https://github.com/actions/setup-java/compare/v3...v4)
---
updated-dependencies:
- dependency-name: actions/setup-java
dependency-type: direct:production
update-type: version-update:semver-major
...
Signed-off-by: dependabot[bot] <support@github.com>
Yann Collet [Fri, 9 Aug 2024 21:09:17 +0000 (14:09 -0700)]
Merge pull request #4115 from Adenilson/leak01
[zstd][leak] Avoid memory leak on early return of ZSTD_generateSequence
Yann Collet [Fri, 9 Aug 2024 21:08:57 +0000 (14:08 -0700)]
Merge pull request #4114 from Adenilson/trace01
[riscv] Enable support for weak symbols
Yann Collet [Fri, 9 Aug 2024 00:19:08 +0000 (17:19 -0700)]
Merge pull request #4113 from facebook/fix4110
formatString_u() can display numbers > 100
Xionghu Luo [Thu, 8 Aug 2024 07:13:27 +0000 (15:13 +0800)]
Fix zstd-pgo run error
The -Werror=missing-profile caused thread/zlib/lzma/lz4 detection failure
when build with profile-use, thus caused ZSTD_MULTITHREAD etc. is not
defined for profile-use, then there will be many profile mismatch information
in output and the final binary reports run error sometimes as below:
Error : ZSTD_CCtx_setParameter(ctx, ZSTD_c_nbWorkers, adv->nbWorkers) failed : Unsupported parameter
Signed-off-by: Xionghu Luo <xionghuluo@tencent.com>
Yann Collet [Thu, 8 Aug 2024 01:17:18 +0000 (18:17 -0700)]
Merge pull request #4109 from facebook/actionsTest
added android-ndk-build
Adenilson Cavalcanti [Wed, 7 Aug 2024 00:16:28 +0000 (17:16 -0700)]
[zstd][leak] Avoid memory leak on early return of ZSTD_generateSequence
Sanity checks on a few of the context parameters (i.e. workers and block size)
may prompt an early return on ZSTD_generateSequences.
Allocating the destination buffer past those return points avoids a potential
memory leak.
This patch should fix issue #4112.
Adenilson Cavalcanti [Tue, 6 Aug 2024 23:55:32 +0000 (16:55 -0700)]
[riscv] Enable support for weak symbols
Both gcc and clang support weak symbols on RISC-V, therefore
let's enable it.
This should fix issue #4069.
Yann Collet [Tue, 6 Aug 2024 19:47:30 +0000 (12:47 -0700)]
fix c90 comment style
Yann Collet [Tue, 6 Aug 2024 18:44:37 +0000 (11:44 -0700)]
formatString_u() can display numbers > 100
fixes #4110
Yann Collet [Mon, 5 Aug 2024 07:39:06 +0000 (00:39 -0700)]
Merge pull request #4111 from facebook/dependabot/github_actions/msys2/setup-msys2-2.24.1
Bump msys2/setup-msys2 from 2.24.0 to 2.24.1
dependabot[bot] [Mon, 5 Aug 2024 05:17:43 +0000 (05:17 +0000)]
Bump msys2/setup-msys2 from 2.24.0 to 2.24.1
Bumps [msys2/setup-msys2](https://github.com/msys2/setup-msys2) from 2.24.0 to 2.24.1.
- [Release notes](https://github.com/msys2/setup-msys2/releases)
- [Changelog](https://github.com/msys2/setup-msys2/blob/main/CHANGELOG.md)
- [Commits](https://github.com/msys2/setup-msys2/compare/
5df0ca6cbf14efcd08f8d5bd5e049a3cc8e07fd2 ...
ddf331adaebd714795f1042345e6ca57bd66cea8 )
---
updated-dependencies:
- dependency-name: msys2/setup-msys2
dependency-type: direct:production
update-type: version-update:semver-patch
...
Signed-off-by: dependabot[bot] <support@github.com>
Yann Collet [Wed, 31 Jul 2024 08:43:15 +0000 (01:43 -0700)]
Merge pull request #4102 from proppy/decodecorpus-ext
tests/decodecorpus: add more advanced options
Yann Collet [Tue, 30 Jul 2024 18:02:16 +0000 (11:02 -0700)]
added android-ndk-build
Yann Collet [Tue, 30 Jul 2024 18:34:27 +0000 (11:34 -0700)]
Merge pull request #4107 from Adenilson/ndk01
[zstd][android] Fix build with NDK r27
Adenilson Cavalcanti [Mon, 29 Jul 2024 23:37:41 +0000 (16:37 -0700)]
[zstd][android] Fix build with NDK r27
The NDK cross compiler declares the target as __linux (which is
not technically incorrect), which triggers the enablement of _GNU_SOURCE
in the newly added code that requires the presence of qsort_r() used
in the COVER dictionary code.
Even though the NDK uses llvm/libc, it doesn't declare qsort_r()
in the stdlib.h header.
The build fix is to only activate the _GNU_SOURCE macro if the OS is
*not* Android, as then we will fallback to the C90 compliant code.
This patch should solve the reported issue number #4103.
Yann Collet [Mon, 29 Jul 2024 16:34:55 +0000 (09:34 -0700)]
Merge pull request #4094 from RubenKelevra/patchfrom_singlethread_man_update
clarify when to use '--single-thread' with '--patch-from'