]> git.ipfire.org Git - thirdparty/zstd.git/log
thirdparty/zstd.git
4 years agoExplicit cast for visual warnings 2395/head
sen [Thu, 19 Nov 2020 22:32:40 +0000 (17:32 -0500)] 
Explicit cast for visual warnings

Github has automatic commits now! Cool

Co-authored-by: Nick Terrell <nickrterrell@gmail.com>
4 years agoUnroll isRLE loop
senhuang42 [Thu, 19 Nov 2020 16:56:16 +0000 (11:56 -0500)] 
Unroll isRLE loop

4 years agoMerge pull request #2393 from senhuang42/fix_sequence_extractions_api
sen [Sun, 15 Nov 2020 23:29:52 +0000 (18:29 -0500)] 
Merge pull request #2393 from senhuang42/fix_sequence_extractions_api

Improve repcode handling in sequence extraction API

4 years agoReduce number of memcpy() calls 2393/head
senhuang42 [Fri, 13 Nov 2020 14:41:44 +0000 (09:41 -0500)] 
Reduce number of memcpy() calls

4 years agoUse existing repcode update function to implement updates
senhuang42 [Thu, 12 Nov 2020 21:38:23 +0000 (16:38 -0500)] 
Use existing repcode update function to implement updates

4 years agoAdd in proper block repcode histories
senhuang42 [Thu, 12 Nov 2020 19:37:47 +0000 (14:37 -0500)] 
Add in proper block repcode histories

4 years agoLet block reps persist
senhuang42 [Thu, 12 Nov 2020 17:22:58 +0000 (12:22 -0500)] 
Let block reps persist

4 years agoFix incorrect repcode setting
senhuang42 [Thu, 12 Nov 2020 16:57:01 +0000 (11:57 -0500)] 
Fix incorrect repcode setting

4 years agoImprove unit test
senhuang42 [Thu, 12 Nov 2020 16:09:01 +0000 (11:09 -0500)] 
Improve unit test

4 years agoOverhaul repcode handling logic
senhuang42 [Thu, 12 Nov 2020 15:59:35 +0000 (10:59 -0500)] 
Overhaul repcode handling logic

4 years agoMerge pull request #2388 from facebook/fix2386
Yann Collet [Fri, 6 Nov 2020 19:38:08 +0000 (11:38 -0800)] 
Merge pull request #2388 from facebook/fix2386

fix incorrect assert

4 years agoMerge pull request #2381 from senhuang42/expand_sequence_extraction_api
sen [Fri, 6 Nov 2020 18:00:31 +0000 (13:00 -0500)] 
Merge pull request #2381 from senhuang42/expand_sequence_extraction_api

Add enum to define ZSTD_Sequence type and update sequence extraction API

4 years agofix multiple minor conversion warnings 2388/head
Yann Collet [Fri, 6 Nov 2020 17:57:05 +0000 (09:57 -0800)] 
fix multiple minor conversion warnings

unrelated to #2386, just cleaning up while I'm updating this file ...

4 years agofix incorrect assert
Yann Collet [Fri, 6 Nov 2020 17:44:04 +0000 (09:44 -0800)] 
fix incorrect assert

fix #2386, reported by @Neumann-A

4 years agoUpdate unit tests 2381/head
senhuang42 [Fri, 6 Nov 2020 15:56:56 +0000 (10:56 -0500)] 
Update unit tests

4 years agoImplement mergeGeneratedSequences()
senhuang42 [Fri, 6 Nov 2020 15:55:46 +0000 (10:55 -0500)] 
Implement mergeGeneratedSequences()

4 years agoRename getSequences() to generateSequences()
senhuang42 [Fri, 6 Nov 2020 15:53:22 +0000 (10:53 -0500)] 
Rename getSequences() to generateSequences()

4 years agoAdd new mergeGeneratedSequences() function
senhuang42 [Fri, 6 Nov 2020 15:52:34 +0000 (10:52 -0500)] 
Add new mergeGeneratedSequences() function

4 years agoMerge pull request #2385 from LuAPi/add-ZSTD_getDictID_fromCDict-single-commit
Nick Terrell [Thu, 5 Nov 2020 18:36:13 +0000 (10:36 -0800)] 
Merge pull request #2385 from LuAPi/add-ZSTD_getDictID_fromCDict-single-commit

Add ZSTD_getDictID_fromCDict function to experimental section

4 years agoAdd ZSTD_getDictID_fromCDict function to experimental section 2385/head
Luke Pitt [Wed, 4 Nov 2020 11:37:37 +0000 (11:37 +0000)] 
Add ZSTD_getDictID_fromCDict function to experimental section

4 years agoChange block delimiter removing to linear time approach
senhuang42 [Mon, 2 Nov 2020 21:59:16 +0000 (16:59 -0500)] 
Change block delimiter removing to linear time approach

4 years agoRemove trailing comma
senhuang42 [Mon, 2 Nov 2020 16:53:04 +0000 (11:53 -0500)] 
Remove trailing comma

4 years agoUse ZSTD_memmove() instead of memmove()
senhuang42 [Mon, 2 Nov 2020 16:43:19 +0000 (11:43 -0500)] 
Use ZSTD_memmove() instead of memmove()

4 years agoRevert compressibility change
senhuang42 [Mon, 2 Nov 2020 16:35:27 +0000 (11:35 -0500)] 
Revert compressibility change

4 years agoUpdate name of enum, clarify documentation
senhuang42 [Mon, 2 Nov 2020 16:32:56 +0000 (11:32 -0500)] 
Update name of enum, clarify documentation

4 years agoUpdate unit test
senhuang42 [Mon, 2 Nov 2020 16:30:31 +0000 (11:30 -0500)] 
Update unit test

4 years agoRevert unnecessary seqCollector adjustment
senhuang42 [Mon, 2 Nov 2020 15:59:06 +0000 (10:59 -0500)] 
Revert unnecessary seqCollector adjustment

4 years agoFix incorrect index increment in merge algorithm
senhuang42 [Mon, 2 Nov 2020 15:58:18 +0000 (10:58 -0500)] 
Fix incorrect index increment in merge algorithm

4 years agoAdd algorithm to remove all delimiters
senhuang42 [Mon, 2 Nov 2020 15:46:52 +0000 (10:46 -0500)] 
Add algorithm to remove all delimiters

4 years agoUpdate seqCollector definition
senhuang42 [Mon, 2 Nov 2020 15:19:26 +0000 (10:19 -0500)] 
Update seqCollector definition

4 years agoUpdate ZSTD_getSequences function signature
senhuang42 [Mon, 2 Nov 2020 15:17:59 +0000 (10:17 -0500)] 
Update ZSTD_getSequences function signature

4 years agoAdd new enum for different sequence formats for ingestion/extraction
senhuang42 [Mon, 2 Nov 2020 15:15:53 +0000 (10:15 -0500)] 
Add new enum for different sequence formats for ingestion/extraction

4 years agoMerge pull request #2378 from senhuang42/free_cress_ptr
sen [Mon, 2 Nov 2020 01:33:25 +0000 (20:33 -0500)] 
Merge pull request #2378 from senhuang42/free_cress_ptr

[minor] Pass cRess_t by const ptr instead of by value

4 years agoMerge pull request #2379 from terrelln/regression-test
Nick Terrell [Fri, 30 Oct 2020 22:09:38 +0000 (15:09 -0700)] 
Merge pull request #2379 from terrelln/regression-test

[regression] Updates results.csv & add README

4 years agoMerge pull request #2354 from terrelln/stable-buffer
Nick Terrell [Fri, 30 Oct 2020 22:06:56 +0000 (15:06 -0700)] 
Merge pull request #2354 from terrelln/stable-buffer

Add ZSTD_c_stable{In,Out}Buffer and optimize when set

4 years ago[regression] Add README explaining the test 2379/head
Nick Terrell [Fri, 30 Oct 2020 20:55:52 +0000 (13:55 -0700)] 
[regression] Add README explaining the test

4 years ago[regression] Update results.csv
Nick Terrell [Fri, 30 Oct 2020 20:54:30 +0000 (13:54 -0700)] 
[regression] Update results.csv

https://github.com/facebook/zstd/pull/2339 removes the single-pass zstdmt API.
This changes the compressed size, because we no longer take the # of threads into
account when deciding the job size.

4 years agoMerge pull request #2376 from senhuang42/clarify_sequence_extraction_api
sen [Fri, 30 Oct 2020 19:47:25 +0000 (15:47 -0400)] 
Merge pull request #2376 from senhuang42/clarify_sequence_extraction_api

Refine external ZSTD_Sequence API

4 years ago[test] Add unit tests for ZSTD_c_stable{In,Out}Buffer 2354/head
Nick Terrell [Tue, 13 Oct 2020 01:40:14 +0000 (18:40 -0700)] 
[test] Add unit tests for ZSTD_c_stable{In,Out}Buffer

4 years ago[lib] Avoid allocating the input buffer when ZSTD_c_stableInBuffer is set
Nick Terrell [Mon, 12 Oct 2020 21:47:55 +0000 (14:47 -0700)] 
[lib] Avoid allocating the input buffer when ZSTD_c_stableInBuffer is set

We don't use it when we have a stable input buffer, so don't allocate
it. I had to slightly modify `ZSTD_copyCCtx()` by storing the
`ZSTD_buffered_policy_e` in the `ZSTD_CCtx`, since `inBuffSize > 0` is
no longer the correct signal for the buffered mode.

4 years ago[lib] Skip the input window buffer when ZSTD_c_stableInBuffer is set
Nick Terrell [Mon, 12 Oct 2020 21:36:30 +0000 (14:36 -0700)] 
[lib] Skip the input window buffer when ZSTD_c_stableInBuffer is set

Compress directly from the `ZSTD_inBuffer`. We still allocate the input
buffer. A following commit will remove that allocation.

4 years ago[cwksp] Return NULL when 0 bytes are requested
Nick Terrell [Mon, 12 Oct 2020 21:19:04 +0000 (14:19 -0700)] 
[cwksp] Return NULL when 0 bytes are requested

This ensures that the buffer is never used.

4 years ago[lib] Avoid allocating output buffer when ZSTD_c_stableOutBuffer is set
Nick Terrell [Mon, 12 Oct 2020 21:17:22 +0000 (14:17 -0700)] 
[lib] Avoid allocating output buffer when ZSTD_c_stableOutBuffer is set

We compress directly to the `ZSTD_outBuffer` so we don't need to
allocate it.

4 years ago[lib] Compress directly into output when ZSTD_c_stableOutBuffer is set
Nick Terrell [Mon, 12 Oct 2020 21:12:23 +0000 (14:12 -0700)] 
[lib] Compress directly into output when ZSTD_c_stableOutBuffer is set

When we have a stable output buffer always compress directly into the
`ZSTD_outBuffer`. We are allowed to return `dstSizeTooSmall`.

4 years ago[lib] Take the shortcut when ZSTD_c_stableOutBuffer is set
Nick Terrell [Mon, 12 Oct 2020 21:09:12 +0000 (14:09 -0700)] 
[lib] Take the shortcut when ZSTD_c_stableOutBuffer is set

When we have a stable output buffer take the single-pass shortcut.
It is okay to return `dstSizeTooSmall` if the output buffer isn't
big enough, because we know it will never grow.

4 years ago[lib] Set ZSTD_c_stable{In,Out}Buffer in ZSTD_compress2()
Nick Terrell [Mon, 12 Oct 2020 20:51:35 +0000 (13:51 -0700)] 
[lib] Set ZSTD_c_stable{In,Out}Buffer in ZSTD_compress2()

Sets these parameters in ZSTD_compress2() then resets them to their
orignal values after the compression call.

An alternative design could be to add a flush mode `ZSTD_e_singlePass`
which implies `ZSTD_c_stable{In,Out}Buffer` but only for a single
compression call, by directly setting the applied parameters. I've opted
for the smaller change, but this is open for discussion.

4 years ago[lib] Validate buffers when ZSTD_c_stable{In,Out}Buffer is set
Nick Terrell [Mon, 12 Oct 2020 20:45:33 +0000 (13:45 -0700)] 
[lib] Validate buffers when ZSTD_c_stable{In,Out}Buffer is set

Adds the validation of the input/output buffers only. They are still
unused.

4 years ago[API] Add ZSTD_c_stable{In,Out}Buffer parameters
Nick Terrell [Mon, 12 Oct 2020 20:15:39 +0000 (13:15 -0700)] 
[API] Add ZSTD_c_stable{In,Out}Buffer parameters

This commit adds the parameters and sets the value in the CCtxParams
but it does not do anything with the value.

4 years ago[lib] Set appliedParams in zstdmt mode
Nick Terrell [Tue, 13 Oct 2020 02:30:23 +0000 (19:30 -0700)] 
[lib] Set appliedParams in zstdmt mode

Previously only `nbWorkers` was set. Set all parameters, because that is
what is expected. This is needed for the `ZSTD_c_stable{In,Out}Buffer`
parameters.

4 years agoSpecify that getSequences() will always emit block boundary sequences 2376/head
senhuang42 [Fri, 30 Oct 2020 16:30:50 +0000 (12:30 -0400)] 
Specify that getSequences() will always emit block boundary sequences

4 years agoSequence extractor should update CBlockState
senhuang42 [Fri, 30 Oct 2020 16:13:19 +0000 (12:13 -0400)] 
Sequence extractor should update CBlockState

4 years agoEmit last literals of 0 size as well, to indicate block boundary
senhuang42 [Thu, 29 Oct 2020 20:41:17 +0000 (16:41 -0400)] 
Emit last literals of 0 size as well, to indicate block boundary

4 years agoCorrect literalsRead calculation to include longLength
senhuang42 [Thu, 29 Oct 2020 18:49:37 +0000 (14:49 -0400)] 
Correct literalsRead calculation to include longLength

4 years agoMerge pull request #2375 from senhuang42/ldm_oss_fuzz_testcase
sen [Thu, 29 Oct 2020 13:32:05 +0000 (09:32 -0400)] 
Merge pull request #2375 from senhuang42/ldm_oss_fuzz_testcase

Add a test case for LDM + opt parser with small uncompressible block

4 years agoUn-mix declarations and code 2375/head
senhuang42 [Wed, 28 Oct 2020 21:49:54 +0000 (17:49 -0400)] 
Un-mix declarations and code

4 years agoRemove implicit typecast to appease appVeyor windows build
senhuang42 [Wed, 28 Oct 2020 20:25:09 +0000 (16:25 -0400)] 
Remove implicit typecast to appease appVeyor windows build

4 years agoAdd check for allocation
senhuang42 [Wed, 28 Oct 2020 20:22:22 +0000 (16:22 -0400)] 
Add check for allocation

4 years agoMerge pull request #2373 from facebook/libzstd_autoconf_full
Yann Collet [Wed, 28 Oct 2020 18:57:08 +0000 (11:57 -0700)] 
Merge pull request #2373 from facebook/libzstd_autoconf_full

Automatic detection of configuration changes

4 years agoMerge branch 'dev' into libzstd_autoconf_full 2373/head
Yann Collet [Wed, 28 Oct 2020 17:53:08 +0000 (10:53 -0700)] 
Merge branch 'dev' into libzstd_autoconf_full

4 years agoMerge branch 'libzstd_autoconf_full' of github.com:facebook/zstd into libzstd_autocon...
Yann Collet [Wed, 28 Oct 2020 17:25:55 +0000 (10:25 -0700)] 
Merge branch 'libzstd_autoconf_full' of github.com:facebook/zstd into libzstd_autoconf_full

4 years agominor Makefile refactor
Yann Collet [Wed, 28 Oct 2020 16:39:15 +0000 (09:39 -0700)] 
minor Makefile refactor

hopefully improving readability

4 years agoClarify comments in zstd.h some more
senhuang42 [Tue, 27 Oct 2020 21:45:23 +0000 (17:45 -0400)] 
Clarify comments in zstd.h some more

4 years agoMerge pull request #2339 from terrelln/zstdmt-stability
Nick Terrell [Wed, 28 Oct 2020 02:43:13 +0000 (19:43 -0700)] 
Merge pull request #2339 from terrelln/zstdmt-stability

Fix zstdmt stability issues and clean up the zstdmt code

4 years agoMerge branch 'dev' into libzstd_autoconf_full
Yann Collet [Tue, 27 Oct 2020 22:45:30 +0000 (15:45 -0700)] 
Merge branch 'dev' into libzstd_autoconf_full

4 years agoMerge pull request #2374 from bket/portability
Yann Collet [Tue, 27 Oct 2020 21:15:35 +0000 (14:15 -0700)] 
Merge pull request #2374 from bket/portability

'head -c BYTES' is non-portable

4 years agoMove test to appropriate location
senhuang42 [Tue, 27 Oct 2020 20:59:43 +0000 (16:59 -0400)] 
Move test to appropriate location

4 years agoAdd test to fuzzer.c
senhuang42 [Tue, 27 Oct 2020 20:57:24 +0000 (16:57 -0400)] 
Add test to fuzzer.c

4 years agoMerge pull request #2366 from senhuang42/enable_ldm_by_default
sen [Tue, 27 Oct 2020 18:59:28 +0000 (14:59 -0400)] 
Merge pull request #2366 from senhuang42/enable_ldm_by_default

Enable LDM by default if window size >= 128MB and strategy uses opt parser

4 years agoMerge pull request #2362 from senhuang42/fix_ldm_fuzz_issue
Nick Terrell [Tue, 27 Oct 2020 18:13:03 +0000 (11:13 -0700)] 
Merge pull request #2362 from senhuang42/fix_ldm_fuzz_issue

Fix long distance matcher OSS-fuzz issue

4 years agoRemove unused variable position
senhuang42 [Tue, 27 Oct 2020 16:58:12 +0000 (12:58 -0400)] 
Remove unused variable position

4 years agoAdd test compatibility with last literals in sequences
senhuang42 [Tue, 27 Oct 2020 16:28:46 +0000 (12:28 -0400)] 
Add test compatibility with last literals in sequences

4 years ago'head -c BYTES' is non-portable 2374/head
Björn Ketelaars [Tue, 27 Oct 2020 15:51:24 +0000 (16:51 +0100)] 
'head -c BYTES' is non-portable

4 years agoAdd support for representing last literals in the extracted seqs
senhuang42 [Tue, 27 Oct 2020 15:19:48 +0000 (11:19 -0400)] 
Add support for representing last literals in the extracted seqs

4 years agoImprove documentation of seqStore_t
senhuang42 [Tue, 27 Oct 2020 14:43:37 +0000 (10:43 -0400)] 
Improve documentation of seqStore_t

4 years agoImprove documentation regarding various operations in copyBlockSequences
senhuang42 [Tue, 27 Oct 2020 14:36:06 +0000 (10:36 -0400)] 
Improve documentation regarding various operations in copyBlockSequences

4 years agoModify ZSTD_copyBlockSequences to agree with new API
senhuang42 [Tue, 27 Oct 2020 14:07:26 +0000 (10:07 -0400)] 
Modify ZSTD_copyBlockSequences to agree with new API

4 years agoClarify and modify ZSTD_Sequence definition
senhuang42 [Tue, 27 Oct 2020 13:41:32 +0000 (09:41 -0400)] 
Clarify and modify ZSTD_Sequence definition

4 years agoMerge branch 'dev' into libzstd_autoconf_full
Yann Collet [Mon, 26 Oct 2020 22:17:39 +0000 (15:17 -0700)] 
Merge branch 'dev' into libzstd_autoconf_full

4 years agoMerge pull request #2370 from facebook/libzstd_autoconf
Yann Collet [Mon, 26 Oct 2020 22:16:04 +0000 (15:16 -0700)] 
Merge pull request #2370 from facebook/libzstd_autoconf

Automatic configuration detection

4 years agomake install only rebuild binaries if they don't exist
Yann Collet [Fri, 23 Oct 2020 23:46:49 +0000 (16:46 -0700)] 
make install only rebuild binaries if they don't exist

Now `make` followed by `make install` doesn't rebuild binaries

also : only generated target directories if they don't already exist

4 years agomake zstd is now differentiated from zstd-nomt
Yann Collet [Fri, 23 Oct 2020 23:08:21 +0000 (16:08 -0700)] 
make zstd is now differentiated from zstd-nomt

avoids mixing object files using different flags

4 years agosimplified silent mode maintenance
Yann Collet [Fri, 23 Oct 2020 17:41:17 +0000 (10:41 -0700)] 
simplified silent mode maintenance

4 years agofix partial lib test
Yann Collet [Fri, 23 Oct 2020 17:27:12 +0000 (10:27 -0700)] 
fix partial lib test

4 years agosimplified silent mode
Yann Collet [Fri, 23 Oct 2020 17:22:52 +0000 (10:22 -0700)] 
simplified silent mode

4 years agofixed building libzstd with manual BUILD_DIR
Yann Collet [Fri, 23 Oct 2020 17:14:04 +0000 (10:14 -0700)] 
fixed building libzstd with manual BUILD_DIR

and when HASH is not found

4 years agofix partial-build test 2370/head
Yann Collet [Fri, 23 Oct 2020 02:53:01 +0000 (19:53 -0700)] 
fix partial-build test

sometimes, the scope difference is solely determined by the list of source files,
not by the flags.

4 years agoMerge branch 'dev' into libzstd_autoconf
Yann Collet [Fri, 23 Oct 2020 01:55:30 +0000 (18:55 -0700)] 
Merge branch 'dev' into libzstd_autoconf

4 years agocan integrate later dynamic flags changes
Yann Collet [Fri, 23 Oct 2020 01:48:06 +0000 (18:48 -0700)] 
can integrate later dynamic flags changes

for example `libzstd-mt` is `differentiated from `libzstd`

4 years agoMerge pull request #2368 from facebook/progressive_libzstd
Yann Collet [Fri, 23 Oct 2020 00:36:56 +0000 (17:36 -0700)] 
Merge pull request #2368 from facebook/progressive_libzstd

faster rebuild of libzstd

4 years agoupdated build documentation
Yann Collet [Thu, 22 Oct 2020 19:31:23 +0000 (12:31 -0700)] 
updated build documentation

4 years agofixed libzstd.dll compilation on mingw
Yann Collet [Thu, 22 Oct 2020 18:28:46 +0000 (11:28 -0700)] 
fixed libzstd.dll compilation on mingw

and zstd linking

4 years agofix DEBUGLEVEL redefinition from tests/
Yann Collet [Thu, 22 Oct 2020 07:20:40 +0000 (00:20 -0700)] 
fix DEBUGLEVEL redefinition from tests/

4 years agofix directory creation for Windows' libzstd
Yann Collet [Thu, 22 Oct 2020 07:15:31 +0000 (00:15 -0700)] 
fix directory creation for Windows' libzstd

4 years agofix test
Yann Collet [Thu, 22 Oct 2020 06:51:13 +0000 (23:51 -0700)] 
fix test

DEBUGLEVEL redefinition

4 years agoprograms/zstd also automatically generate object dir per conf
Yann Collet [Thu, 22 Oct 2020 06:38:33 +0000 (23:38 -0700)] 
programs/zstd also automatically generate object dir per conf

same rules as lib/libzstd
can also be controlled via HASH and BUILD_DIR

4 years agozstd in programs/ can also receive a DEBUGLEVEL value
Yann Collet [Thu, 22 Oct 2020 06:13:46 +0000 (23:13 -0700)] 
zstd in programs/ can also receive a DEBUGLEVEL value

simplify tests : only set DEBUGLEVEL, not the flags directly

4 years agoMerge branch 'dev' into libzstd_autoconf
Yann Collet [Thu, 22 Oct 2020 05:36:09 +0000 (22:36 -0700)] 
Merge branch 'dev' into libzstd_autoconf

4 years agoautomatically detect configuration changes
Yann Collet [Thu, 22 Oct 2020 02:22:45 +0000 (19:22 -0700)] 
automatically detect configuration changes

Makefile now automatically detects modifications of compilation flags,
and produce object files in directories dedicated to this compilation flags.
This makes it possible, for example, to compile libzstd with different DEBUGLEVEL.
Object files sharing the same configration will be generated into their dedicated directories.

Also : new compilation variables
- DEBUGLEVEL : select the debug level (assert & traces) inserted during compilation (default == 0 == release)
- HASH : select a hash function to differentiate configuration (default == md5sum)
- BUILD_DIR : skip the hash stage, store object files into manually specified directory

4 years agoautomatic %.h header dependency tracking 2368/head
Yann Collet [Thu, 22 Oct 2020 00:25:07 +0000 (17:25 -0700)] 
automatic %.h header dependency tracking

also : BUILD_DIR can be manually specified

4 years agoMerge pull request #2367 from facebook/progressive_build
Yann Collet [Wed, 21 Oct 2020 22:43:14 +0000 (15:43 -0700)] 
Merge pull request #2367 from facebook/progressive_build

faster rebuild of zstd