]> git.ipfire.org Git - thirdparty/zstd.git/log
thirdparty/zstd.git
9 years agoadded "Makefile is validated"
Przemyslaw Skibinski [Tue, 17 Jan 2017 11:40:06 +0000 (12:40 +0100)] 
added "Makefile is validated"

9 years agoMerge pull request #511 from indygreg/cdict-dictid
Yann Collet [Tue, 17 Jan 2017 04:17:44 +0000 (20:17 -0800)] 
Merge pull request #511 from indygreg/cdict-dictid

Set dictionary ID in ZSTD_initCStream_usingCDict()

9 years agoadded test checking dictID when using ZSTD_initCStream_usingCDict()
Yann Collet [Tue, 17 Jan 2017 03:46:22 +0000 (19:46 -0800)] 
added test checking dictID when using ZSTD_initCStream_usingCDict()

It shows that dictID is not properly added into frame header

9 years agoSet dictionary ID in ZSTD_initCStream_usingCDict() 511/head
Gregory Szorc [Sun, 15 Jan 2017 01:44:54 +0000 (17:44 -0800)] 
Set dictionary ID in ZSTD_initCStream_usingCDict()

When porting python-zstandard to use ZSTD_initCStream_usingCDict()
so compression dictionaries could be reused, an automated test
failed due to compressed content changing.

I tracked this down to ZSTD_initCStream_usingCDict() not
setting the dictID field of the ZSTD_CCtx attached to the
ZSTD_CStream instance.

I'm not 100% convinced this is the correct or full solution,
as I'm still seeing one automated test failing with this change.

9 years agoMerge pull request #510 from iburinoc/baddict
Yann Collet [Thu, 12 Jan 2017 21:10:04 +0000 (22:10 +0100)] 
Merge pull request #510 from iburinoc/baddict

Fixed decompress_usingDict not propagating corrupted dictionary error

9 years agoFix missing 'OK' logging on fuzzer testcase 510/head
Sean Purcell [Thu, 12 Jan 2017 17:38:29 +0000 (09:38 -0800)] 
Fix missing 'OK' logging on fuzzer testcase

9 years agofix gcc-arm warning "suggest braces around empty body"
Yann Collet [Thu, 12 Jan 2017 16:46:46 +0000 (17:46 +0100)] 
fix gcc-arm warning "suggest braces around empty body"

9 years agozstdmt : fix : resources properly collected even when early fail
Yann Collet [Thu, 12 Jan 2017 02:06:35 +0000 (03:06 +0100)] 
zstdmt : fix : resources properly collected even when early fail

In previous version, main function would return early when detecting a job error.
Late threads resources were therefore not collected back into pools.
New version just register the error, but continue the collecting process.
All buffers and context should be released back to pool before leaving main function.

9 years agoFixed decompress_usingDict not propagating corrupted dictionary error
Sean Purcell [Thu, 12 Jan 2017 01:31:06 +0000 (17:31 -0800)] 
Fixed decompress_usingDict not propagating corrupted dictionary error

9 years agozstdmt : correctly check for cctx and buffer allocation
Yann Collet [Thu, 12 Jan 2017 01:01:28 +0000 (02:01 +0100)] 
zstdmt : correctly check for cctx and buffer allocation

Result from getBuffer and getCCtx could be NULL when allocation fails.
Now correctly checks : job creation stop and last job reports an allocation error.
releaseBuffer and releaseCCtx are now also compatible with NULL input.

Identified a new potential issue :
when early job fails, later jobs are not collected for resource retrieval.

9 years agozstdmt : changed internal naming from frame to chunk
Yann Collet [Thu, 12 Jan 2017 00:25:46 +0000 (01:25 +0100)] 
zstdmt : changed internal naming from frame to chunk

Since the result of mt compression is a single frame,
changed naming, which implied the concatenation of multiple frames.

minor : ensures that content size is written in header

9 years agoZSTDMT_compress() creates a single frame
Yann Collet [Wed, 11 Jan 2017 17:21:25 +0000 (18:21 +0100)] 
ZSTDMT_compress() creates a single frame

The new strategy involves cutting frame at block level.
The result is a single frame, preserving ZSTD_getDecompressedSize()

As a consequence, bench can now make a full round-trip,
since the result is compatible with ZSTD_decompress().

This strategy will not make it possible to decode the frame with multiple threads
since the exact cut between independent blocks is not known.
MT decoding needs further discussions.

9 years agominor refactor (release CCtx 1st) and comment clarification
Yann Collet [Wed, 11 Jan 2017 15:08:08 +0000 (16:08 +0100)] 
minor refactor (release CCtx 1st) and comment clarification

9 years agofixed ZSTDMT_createCCtx() : checked inner objects are properly created
Yann Collet [Wed, 11 Jan 2017 14:58:05 +0000 (15:58 +0100)] 
fixed ZSTDMT_createCCtx() : checked inner objects are properly created

9 years agoimproved ZSTD_createCCtxPool() cancellation
Yann Collet [Wed, 11 Jan 2017 14:44:26 +0000 (15:44 +0100)] 
improved ZSTD_createCCtxPool() cancellation

use ZSTD_freeCCtxPool() to release the partially created pool.
avoids to duplicate logic.

Also : identified a new difficult corner case :
when freeing the Pool, all CCtx should be previously released back to the pool.
Otherwise, it means some CCtx are still in use.
There is currently no clear policy on what to do in such a case.
Note : it's supposed to never happen.
Since pool creation/usage is static, it has no external user,
which limits risks.

9 years agofixed ZSTDMT_createCCtxPool() when inner CCtx creation fails
Yann Collet [Wed, 11 Jan 2017 14:35:56 +0000 (15:35 +0100)] 
fixed ZSTDMT_createCCtxPool() when inner CCtx creation fails

9 years agoMerge pull request #509 from terrelln/dict-builder-32
Yann Collet [Tue, 10 Jan 2017 05:30:28 +0000 (06:30 +0100)] 
Merge pull request #509 from terrelln/dict-builder-32

Handle cover dictionary builder maximum input size for 32-bit mode

9 years agoDocument memory requirements for COVER algorithm 509/head
Nick Terrell [Tue, 10 Jan 2017 01:00:12 +0000 (17:00 -0800)] 
Document memory requirements for COVER algorithm

9 years agoHandle large input size in 32-bit mode correctly
Nick Terrell [Tue, 10 Jan 2017 00:50:00 +0000 (16:50 -0800)] 
Handle large input size in 32-bit mode correctly

9 years agoUse COVER_MEMMULT when training with COVER.
Nick Terrell [Tue, 10 Jan 2017 00:49:04 +0000 (16:49 -0800)] 
Use COVER_MEMMULT when training with COVER.

9 years agominor man page update
Yann Collet [Mon, 9 Jan 2017 18:47:09 +0000 (19:47 +0100)] 
minor man page update

9 years agoMerge pull request #507 from terrelln/dict-builder-optimize
Yann Collet [Tue, 3 Jan 2017 19:33:33 +0000 (20:33 +0100)] 
Merge pull request #507 from terrelln/dict-builder-optimize

Add new dictionary builder

9 years agoAdd COVER dictionary builder to fuzzer unit tests 507/head
Nick Terrell [Tue, 3 Jan 2017 02:45:19 +0000 (18:45 -0800)] 
Add COVER dictionary builder to fuzzer unit tests

9 years agoSimplify COVER parameters
Nick Terrell [Mon, 2 Jan 2017 20:40:43 +0000 (12:40 -0800)] 
Simplify COVER parameters

9 years agoseparated ppc and ppc64 tests, for more regular timing
Yann Collet [Mon, 2 Jan 2017 23:25:01 +0000 (00:25 +0100)] 
separated ppc and ppc64 tests, for more regular timing

9 years agoAdd cover cli to playtests
Nick Terrell [Sun, 1 Jan 2017 06:59:51 +0000 (01:59 -0500)] 
Add cover cli to playtests

9 years agoAdd COVER to the zstd cli
Nick Terrell [Sun, 1 Jan 2017 05:08:24 +0000 (21:08 -0800)] 
Add COVER to the zstd cli

9 years agoAdd cover.c to build files
Nick Terrell [Sun, 1 Jan 2017 05:08:12 +0000 (21:08 -0800)] 
Add cover.c to build files

9 years agoAdd COVER dictionary builder
Nick Terrell [Sun, 1 Jan 2017 05:07:44 +0000 (21:07 -0800)] 
Add COVER dictionary builder

9 years agocompile object files, for faster recompilation
Yann Collet [Mon, 2 Jan 2017 02:22:18 +0000 (03:22 +0100)] 
compile object files, for faster recompilation

9 years agofixed cmake tests
Yann Collet [Mon, 2 Jan 2017 01:05:45 +0000 (02:05 +0100)] 
fixed cmake tests

9 years agonew Makefile target zstdmt
Yann Collet [Mon, 2 Jan 2017 00:43:56 +0000 (01:43 +0100)] 
new Makefile target zstdmt

9 years agofixed linux warnings
Yann Collet [Mon, 2 Jan 2017 00:11:55 +0000 (01:11 +0100)] 
fixed linux warnings

9 years agominor fixes
Yann Collet [Sun, 1 Jan 2017 23:49:42 +0000 (00:49 +0100)] 
minor fixes

9 years agouse pthread_cond to send signals between threads
Yann Collet [Sun, 1 Jan 2017 16:31:33 +0000 (17:31 +0100)] 
use pthread_cond to send signals between threads

9 years agoMerge pull request #505 from terrelln/pthread_cond_t
Yann Collet [Sun, 1 Jan 2017 14:53:51 +0000 (15:53 +0100)] 
Merge pull request #505 from terrelln/pthread_cond_t

pthread_cond_t

9 years agoShorten thread pool tests 505/head
Nick Terrell [Sun, 1 Jan 2017 03:39:32 +0000 (22:39 -0500)] 
Shorten thread pool tests

9 years agoFix pool for threading.h
Nick Terrell [Sun, 1 Jan 2017 00:10:47 +0000 (19:10 -0500)] 
Fix pool for threading.h

9 years agoAdd threading.h condition variables
Nick Terrell [Sun, 1 Jan 2017 00:10:29 +0000 (19:10 -0500)] 
Add threading.h condition variables

9 years agoSwitch thread pool test to threading.h
Nick Terrell [Sun, 1 Jan 2017 00:10:13 +0000 (19:10 -0500)] 
Switch thread pool test to threading.h

9 years agoextended ZSTDMT code support for non-MT systems and WIN32 (preliminary)
Yann Collet [Sat, 31 Dec 2016 15:32:19 +0000 (16:32 +0100)] 
extended ZSTDMT code support for non-MT systems and WIN32 (preliminary)

9 years agosimplified Buffer Pool
Yann Collet [Sat, 31 Dec 2016 13:45:33 +0000 (14:45 +0100)] 
simplified Buffer Pool

9 years agonew zstdmt version using generic treadpool
Yann Collet [Sat, 31 Dec 2016 05:04:25 +0000 (06:04 +0100)] 
new zstdmt version using generic treadpool

9 years agobench correctly measures time for multi-threaded compression (posix only)
Yann Collet [Sat, 31 Dec 2016 02:31:26 +0000 (03:31 +0100)] 
bench correctly measures time for multi-threaded compression (posix only)

9 years agoMerge pull request #504 from terrelln/thread-pool
Yann Collet [Fri, 30 Dec 2016 14:31:49 +0000 (15:31 +0100)] 
Merge pull request #504 from terrelln/thread-pool

[zstdmt] Add thread pool

9 years agoAdd thread pool tests 504/head
Nick Terrell [Fri, 30 Dec 2016 07:41:03 +0000 (23:41 -0800)] 
Add thread pool tests

9 years agoAdd a thread pool for ZSTDMT and COVER
Nick Terrell [Fri, 30 Dec 2016 07:39:44 +0000 (23:39 -0800)] 
Add a thread pool for ZSTDMT and COVER

9 years agoMerge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Przemyslaw Skibinski [Thu, 29 Dec 2016 14:24:18 +0000 (15:24 +0100)] 
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11

9 years agoChanged : input divided into roughly equal parts.
Yann Collet [Thu, 29 Dec 2016 00:24:01 +0000 (01:24 +0100)] 
Changed : input divided into roughly equal parts.
Debug : can measure time waiting for mutexes to unlock.

9 years agocompression threads use ZSTD_compressCCtx()
Yann Collet [Wed, 28 Dec 2016 16:08:28 +0000 (17:08 +0100)] 
compression threads use ZSTD_compressCCtx()

9 years agoMerge pull request #503 from inikep/dev11
Yann Collet [Wed, 28 Dec 2016 15:50:39 +0000 (16:50 +0100)] 
Merge pull request #503 from inikep/dev11

Dev11

9 years agoadded -T command , to set nb of threads
Yann Collet [Wed, 28 Dec 2016 15:11:09 +0000 (16:11 +0100)] 
added -T command , to set nb of threads

9 years agoprotect buffer pool with a mutex
Yann Collet [Wed, 28 Dec 2016 14:31:19 +0000 (15:31 +0100)] 
protect buffer pool with a mutex

9 years agoMerge pull request #502 from Chocobo1/cpp11
Yann Collet [Wed, 28 Dec 2016 13:38:48 +0000 (14:38 +0100)] 
Merge pull request #502 from Chocobo1/cpp11

Move -std=c++11 cxxflag to PZSTD_CXXFLAGS

9 years agochanged default PREFIX and MANDIR 503/head
Przemyslaw Skibinski [Wed, 28 Dec 2016 11:32:41 +0000 (12:32 +0100)] 
changed default PREFIX and MANDIR

9 years agofirst zstdmt sketch
Yann Collet [Tue, 27 Dec 2016 06:19:36 +0000 (07:19 +0100)] 
first zstdmt sketch

9 years agoMove -std=c++11 cxxflag to PZSTD_CXXFLAGS 502/head
Chocobo1 [Mon, 26 Dec 2016 15:04:59 +0000 (23:04 +0800)] 
Move -std=c++11 cxxflag to PZSTD_CXXFLAGS

Fixes the problem that the compiler doesn't enable c++11 mode by default
and the package build system has its own CXXFLAGS

9 years agoMerge branch 'dev' of github.com:facebook/zstd into dev
Yann Collet [Fri, 23 Dec 2016 21:25:31 +0000 (22:25 +0100)] 
Merge branch 'dev' of github.com:facebook/zstd into dev

9 years agofixed ZSTD_sizeof_?Dict()
Yann Collet [Fri, 23 Dec 2016 21:25:03 +0000 (22:25 +0100)] 
fixed ZSTD_sizeof_?Dict()

9 years agoMerge pull request #499 from inikep/dev11
Yann Collet [Fri, 23 Dec 2016 20:32:03 +0000 (21:32 +0100)] 
Merge pull request #499 from inikep/dev11

improved *BSD and Solaris compatibility

9 years agoMerge pull request #497 from apjanke/zstdless-with-shebang
Yann Collet [Fri, 23 Dec 2016 10:18:10 +0000 (11:18 +0100)] 
Merge pull request #497 from apjanke/zstdless-with-shebang

zstdless: add shebang and quote $@

9 years agoMerge pull request #498 from terrelln/zdict-deprecated
Yann Collet [Fri, 23 Dec 2016 10:16:23 +0000 (11:16 +0100)] 
Merge pull request #498 from terrelln/zdict-deprecated

Fix failing test due to deprecation warning

9 years agoBSD: improved "make install" 499/head
Przemyslaw Skibinski [Fri, 23 Dec 2016 09:05:49 +0000 (10:05 +0100)] 
BSD: improved "make install"

9 years agoZDICT_finalizeDictionary() flipped comparison 498/head
Nick Terrell [Fri, 23 Dec 2016 02:14:57 +0000 (18:14 -0800)] 
ZDICT_finalizeDictionary() flipped comparison

9 years agoZDICT_finalizeDictionary() flipped comparison
Nick Terrell [Fri, 23 Dec 2016 02:01:14 +0000 (18:01 -0800)] 
ZDICT_finalizeDictionary() flipped comparison

`ZDICT_finalizeDictionary()` had a flipped comparison.
I also allowed `dictBufferCapacity == dictContentSize`.
It might be the case that the user wants to fill the dictionary
completely up, and then let zstd take exactly the space it needs
for the entropy tables.

9 years agoFix failing test due to deprecation warning
Nick Terrell [Fri, 23 Dec 2016 01:28:49 +0000 (17:28 -0800)] 
Fix failing test due to deprecation warning

9 years agozstdless: add shebang and quote $@ 497/head
Andrew Janke [Thu, 22 Dec 2016 22:40:10 +0000 (17:40 -0500)] 
zstdless: add shebang and quote $@

9 years agoadded ZDICT_finalizeDictionary()
Yann Collet [Thu, 22 Dec 2016 19:18:43 +0000 (20:18 +0100)] 
added ZDICT_finalizeDictionary()

9 years agoSolaris: working "make -C lib install"
Przemyslaw Skibinski [Thu, 22 Dec 2016 19:14:37 +0000 (20:14 +0100)] 
Solaris: working "make -C lib install"

9 years agoSolaris: working "make -C programs install"
Przemyslaw Skibinski [Thu, 22 Dec 2016 18:26:01 +0000 (19:26 +0100)] 
Solaris: working "make -C programs install"

9 years ago"make test" is now compatible with Solaris
Przemyslaw Skibinski [Thu, 22 Dec 2016 17:05:07 +0000 (18:05 +0100)] 
"make test" is now compatible with Solaris

9 years agoMerge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Przemyslaw Skibinski [Thu, 22 Dec 2016 16:59:22 +0000 (17:59 +0100)] 
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11

9 years agofixed zlib_wrapper conversion warnings
Yann Collet [Thu, 22 Dec 2016 14:26:33 +0000 (15:26 +0100)] 
fixed zlib_wrapper conversion warnings

9 years agofixed allmost target
Yann Collet [Thu, 22 Dec 2016 11:43:00 +0000 (12:43 +0100)] 
fixed allmost target

9 years agoremoved examples from standard C tests, since they contain some POSIX elements
Yann Collet [Thu, 22 Dec 2016 10:31:39 +0000 (11:31 +0100)] 
removed examples from standard C tests, since they contain some POSIX elements

9 years agofix : zlib wrapper compile in gnu90 mode
Yann Collet [Wed, 21 Dec 2016 18:57:18 +0000 (19:57 +0100)] 
fix : zlib wrapper compile in gnu90 mode

9 years agoadded ZSTD_createDDict_byReference() body
Yann Collet [Wed, 21 Dec 2016 18:25:15 +0000 (19:25 +0100)] 
added ZSTD_createDDict_byReference() body

9 years agoMerge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Przemyslaw Skibinski [Wed, 21 Dec 2016 16:02:06 +0000 (17:02 +0100)] 
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11

9 years agoadded ZSTD_createDDict_byReference()
Yann Collet [Wed, 21 Dec 2016 15:44:35 +0000 (16:44 +0100)] 
added ZSTD_createDDict_byReference()

9 years agoMerge branch 'dev' of github.com:facebook/zstd into dev
Yann Collet [Wed, 21 Dec 2016 15:44:24 +0000 (16:44 +0100)] 
Merge branch 'dev' of github.com:facebook/zstd into dev

9 years agoMerge pull request #489 from inikep/v112
Yann Collet [Wed, 21 Dec 2016 15:42:46 +0000 (16:42 +0100)] 
Merge pull request #489 from inikep/v112

improved detection of POSIX

9 years agoMerge branch 'dev' of github.com:facebook/zstd into dev
Yann Collet [Wed, 21 Dec 2016 15:20:41 +0000 (16:20 +0100)] 
Merge branch 'dev' of github.com:facebook/zstd into dev

9 years agoadded : ZSTD_createCDict_byReference()
Yann Collet [Wed, 21 Dec 2016 15:20:11 +0000 (16:20 +0100)] 
added : ZSTD_createCDict_byReference()

9 years agoMerge pull request #493 from inikep/fullbench
Yann Collet [Wed, 21 Dec 2016 14:12:06 +0000 (15:12 +0100)] 
Merge pull request #493 from inikep/fullbench

Fullbench extDict variant

9 years agoutil.h: restore BSD license for Facebook Open-Source 489/head
Przemyslaw Skibinski [Wed, 21 Dec 2016 14:08:44 +0000 (15:08 +0100)] 
util.h: restore BSD license for Facebook Open-Source

9 years agoupdated comments
Przemyslaw Skibinski [Wed, 21 Dec 2016 13:00:41 +0000 (14:00 +0100)] 
updated comments

9 years ago_CRT_SECURE_NO_WARNINGS moved to util.h
Przemyslaw Skibinski [Wed, 21 Dec 2016 12:47:11 +0000 (13:47 +0100)] 
_CRT_SECURE_NO_WARNINGS moved to util.h

9 years agoplatform.h: removed Compiler Options
Przemyslaw Skibinski [Wed, 21 Dec 2016 12:23:34 +0000 (13:23 +0100)] 
platform.h: removed Compiler Options

9 years agofixed Visual Studio compilation
Przemyslaw Skibinski [Wed, 21 Dec 2016 10:43:11 +0000 (11:43 +0100)] 
fixed Visual Studio compilation

9 years agoMerge pull request #492 from inikep/dev11
Yann Collet [Wed, 21 Dec 2016 10:23:04 +0000 (11:23 +0100)] 
Merge pull request #492 from inikep/dev11

test CMake on Windows

9 years agoMerge pull request #490 from jacquesg/cmake-fix
Yann Collet [Wed, 21 Dec 2016 10:22:28 +0000 (11:22 +0100)] 
Merge pull request #490 from jacquesg/cmake-fix

Fix cmake build (MSVC)

9 years agoimproved MinGW support
Przemyslaw Skibinski [Wed, 21 Dec 2016 10:18:45 +0000 (11:18 +0100)] 
improved MinGW support

9 years agoMerge pull request #494 from terrelln/msan-dict-load
Yann Collet [Wed, 21 Dec 2016 08:33:48 +0000 (09:33 +0100)] 
Merge pull request #494 from terrelln/msan-dict-load

Fix dictionary loading bug causing an MSAN failure

9 years agofix basic types redefinition
Przemyslaw Skibinski [Wed, 21 Dec 2016 08:26:00 +0000 (09:26 +0100)] 
fix basic types redefinition

9 years agosimplified zstdcli.c
Przemyslaw Skibinski [Wed, 21 Dec 2016 08:19:15 +0000 (09:19 +0100)] 
simplified zstdcli.c

9 years agoexecutables use new util.h and platform.h
Przemyslaw Skibinski [Wed, 21 Dec 2016 08:06:14 +0000 (09:06 +0100)] 
executables use new util.h and platform.h

9 years agoimproved util.h and platform.h
Przemyslaw Skibinski [Wed, 21 Dec 2016 08:04:59 +0000 (09:04 +0100)] 
improved util.h and platform.h

9 years agoAdd test for invalid offset rep codes 494/head
Nick Terrell [Tue, 20 Dec 2016 19:13:45 +0000 (11:13 -0800)] 
Add test for invalid offset rep codes

9 years agoFix dictionary loading bug causing an MSAN failure
Nick Terrell [Tue, 20 Dec 2016 18:47:52 +0000 (10:47 -0800)] 
Fix dictionary loading bug causing an MSAN failure

Offset rep codes must be in the range `[1, dictSize)`.
Fix dictionary loading to reject `0` as a offset rep code.

9 years agowindres updated to v1.1.3 493/head
Przemyslaw Skibinski [Tue, 20 Dec 2016 10:21:26 +0000 (11:21 +0100)] 
windres updated to v1.1.3