]> git.ipfire.org Git - thirdparty/zstd.git/log
thirdparty/zstd.git
8 years agoDon't run test-pool with QEMU 545/head
Nick Terrell [Tue, 14 Feb 2017 18:08:14 +0000 (10:08 -0800)] 
Don't run test-pool with QEMU

    > make test -n
    ...
    ./pool
    > make test -n QEMU_SYS=valgrind
    ...
    ./legacy
    # ./pool not run

8 years ago[zstdmt] Fix MSAN failure with ZSTD_p_forceWindow
Nick Terrell [Tue, 14 Feb 2017 02:27:34 +0000 (18:27 -0800)] 
[zstdmt] Fix MSAN failure with ZSTD_p_forceWindow

Reproduction steps:

```
make zstreamtest CC=clang CFLAGS="-O3 -g -fsanitize=memory -fsanitize-memory-track-origins"
./zstreamtest -vv -t4178 -i4178 -s4531
```

How to get to the error in gdb (may be a more efficient way):

* 2 breaks at zstd_compress.c:2418  -- in ZSTD_compressContinue_internal()
* 2 breaks at zstd_compress.c:2276  -- in ZSTD_compressBlock_internal()
* 1 break at zstd_compress.c:1547

Why the error occurred:

When `zc->forceWindow == 1`, after calling `ZSTD_loadDictionaryContent()` we
have `zc->loadedDictEnd == zc->nextToUpdate == 0`. But, we've really loaded up
to `iend` into the dictionary. Then in `ZSTD_compressBlock_internal()` we see
that `current > zc->nextToUpdate + 384`, so we load the last 192 bytes a second
time. In this case the bytes we are loading are a block of all 0s, starting in
the previous block. So when we are loading the last 192 bytes, we find a `match`
in the future, 183 bytes beyond `ip`. Since the block is all 0s, the match
extends to the end of the block. But in `ZSTD_count()` we only check that
`pIn < pInLoopLimit`, but since `pMatch > pIn`, `pMatch` eventually points past
the end of the buffer, causing the MSAN failure.

The fix:

The line changed sets sets `zc->nextToUpdate` to the end of the dictionary.
This is the behavior that existed before `ZSTD_p_forceWindow` was introduced.
This fixes the exposing test case. Since the code doesn't fail without
`zc->forceWindow`, it makes sense that this works. I've run the command
`./zstreamtest -T2mn` 64 times without failures. CI should also verify nothing
obvious broke.

8 years agoMerge pull request #536 from iburinoc/multiframe
Yann Collet [Mon, 13 Feb 2017 20:18:16 +0000 (12:18 -0800)] 
Merge pull request #536 from iburinoc/multiframe

Simple API multiframe decoding

8 years agoMerge pull request #542 from ds77/large-sparse-fix
Yann Collet [Mon, 13 Feb 2017 19:57:35 +0000 (11:57 -0800)] 
Merge pull request #542 from ds77/large-sparse-fix

zstdcli: fix writing 2GB+ sparse files under Windows

8 years agofix indentation in previous commit 542/head
ds77 [Mon, 13 Feb 2017 11:00:59 +0000 (12:00 +0100)] 
fix indentation in previous commit

8 years agofix seeking 2GB+ files under Windows
ds77 [Sun, 12 Feb 2017 09:27:18 +0000 (10:27 +0100)] 
fix seeking 2GB+ files under Windows

Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes  fallback workaround using Win32 API).

8 years agoMerge pull request #540 from ds77/dev-stat64-fix
Yann Collet [Sun, 12 Feb 2017 05:08:08 +0000 (21:08 -0800)] 
Merge pull request #540 from ds77/dev-stat64-fix

zstdcli: Fix reporting incorrect sizes of large flies on MinGW

8 years agoMerge pull request #541 from inikep/mingw
Yann Collet [Sun, 12 Feb 2017 05:07:53 +0000 (21:07 -0800)] 
Merge pull request #541 from inikep/mingw

fixed "mingw32" AppVeyor target

8 years agorevert last commit 541/head
Przemyslaw Skibinski [Fri, 10 Feb 2017 20:15:49 +0000 (21:15 +0100)] 
revert last commit

8 years agoExpose frameSrcSize to experimental API 536/head
Sean Purcell [Fri, 10 Feb 2017 19:38:57 +0000 (11:38 -0800)] 
Expose frameSrcSize to experimental API

8 years agofix for original MinGW
Przemyslaw Skibinski [Fri, 10 Feb 2017 19:09:28 +0000 (20:09 +0100)] 
fix for original MinGW

8 years agoMerge branch 'dev' into multiframe
Sean Purcell [Fri, 10 Feb 2017 18:08:55 +0000 (10:08 -0800)] 
Merge branch 'dev' into multiframe

8 years agouse _stati64() in UTIL_getFileSize() when compiling with mingw, get rid of introduces... 540/head
ds77 [Fri, 10 Feb 2017 17:37:57 +0000 (18:37 +0100)] 
use _stati64() in UTIL_getFileSize() when compiling with mingw, get rid of introduces previously preprocessor checks.

8 years agoMerge remote-tracking branch 'refs/remotes/facebook/dev' into mingw
Przemyslaw Skibinski [Fri, 10 Feb 2017 12:38:54 +0000 (13:38 +0100)] 
Merge remote-tracking branch 'refs/remotes/facebook/dev' into mingw

# Conflicts:
# programs/platform.h

8 years agoMerge pull request #537 from terrelln/small-bugs
Yann Collet [Fri, 10 Feb 2017 12:35:43 +0000 (04:35 -0800)] 
Merge pull request #537 from terrelln/small-bugs

Fix small bugs

8 years agofix previous commit
- [Fri, 10 Feb 2017 12:27:43 +0000 (13:27 +0100)] 
fix previous commit

* struct _stat64 is not defined by (non-w64) MinGW releases, __stat64 should be everywhere
* proper detection of _stat64() availability (as in MinGW sys/stat.h)

8 years agoappveyor.yml: fixed clang test
Przemyslaw Skibinski [Fri, 10 Feb 2017 12:10:25 +0000 (13:10 +0100)] 
appveyor.yml: fixed clang test

8 years agoMerge pull request #538 from iburinoc/errorstring
Yann Collet [Fri, 10 Feb 2017 11:59:56 +0000 (03:59 -0800)] 
Merge pull request #538 from iburinoc/errorstring

Fix ZSTD_getErrorString and add tests

8 years agoMerge pull request #533 from inikep/dev11
Yann Collet [Fri, 10 Feb 2017 11:50:08 +0000 (03:50 -0800)] 
Merge pull request #533 from inikep/dev11

--priority=rt

8 years agoappveyor.yml: reordering of tests
Przemyslaw Skibinski [Fri, 10 Feb 2017 11:41:31 +0000 (12:41 +0100)] 
appveyor.yml: reordering of tests

8 years agofix missing "
Przemyslaw Skibinski [Fri, 10 Feb 2017 11:32:30 +0000 (12:32 +0100)] 
fix missing "

8 years agoappveyor.yml: add clang target
Przemyslaw Skibinski [Fri, 10 Feb 2017 11:01:14 +0000 (12:01 +0100)] 
appveyor.yml: add clang target

8 years agofixed "mingw32" AppVeyor target
Przemyslaw Skibinski [Fri, 10 Feb 2017 10:01:52 +0000 (11:01 +0100)] 
fixed "mingw32" AppVeyor target

8 years agouse _stat64 only when targetting Win2k or later
- [Fri, 10 Feb 2017 09:56:45 +0000 (10:56 +0100)] 
use _stat64 only when targetting Win2k or later

8 years agofix 64-bit file support for MinGW
Przemyslaw Skibinski [Fri, 10 Feb 2017 09:36:44 +0000 (10:36 +0100)] 
fix 64-bit file support for MinGW

8 years agoImprovement from @inikep 537/head
Nick Terrell [Thu, 9 Feb 2017 22:20:52 +0000 (14:20 -0800)] 
Improvement from @inikep

8 years agouse _stat64() on MinGW
ds77 [Thu, 9 Feb 2017 17:12:00 +0000 (18:12 +0100)] 
use _stat64() on MinGW

On MinGW, use _stat64() and struct _stat64 instead of stat() and struct stat_t. This fixes reporting incorrect sizes for large files.

8 years agoDocumentation updates
Sean Purcell [Thu, 9 Feb 2017 21:25:30 +0000 (13:25 -0800)] 
Documentation updates

8 years agoFix failing unit test
Sean Purcell [Thu, 9 Feb 2017 20:27:32 +0000 (12:27 -0800)] 
Fix failing unit test

8 years agoMerge branch 'dev' into multiframe
Sean Purcell [Thu, 9 Feb 2017 19:54:22 +0000 (11:54 -0800)] 
Merge branch 'dev' into multiframe

8 years agoMake pledgedSrcSize meaning clear for other functions
Sean Purcell [Thu, 9 Feb 2017 18:50:43 +0000 (10:50 -0800)] 
Make pledgedSrcSize meaning clear for other functions

- Added tests
- Moved new size functions to static link only

8 years agoecho replaced with printf 533/head
Przemyslaw Skibinski [Thu, 9 Feb 2017 16:01:17 +0000 (17:01 +0100)] 
echo replaced with printf

8 years agodon't use "echo -e"
Przemyslaw Skibinski [Thu, 9 Feb 2017 10:32:14 +0000 (11:32 +0100)] 
don't use "echo -e"

8 years agoMerge branch 'dev' into dev11
Przemyslaw Skibinski [Thu, 9 Feb 2017 08:42:16 +0000 (09:42 +0100)] 
Merge branch 'dev' into dev11

8 years agoMerge pull request #535 from terrelln/tests 539/head
Yann Collet [Thu, 9 Feb 2017 08:36:03 +0000 (00:36 -0800)] 
Merge pull request #535 from terrelln/tests

Remove test-longmatch from test target and only run it once

8 years agoFix deprecation warnings for clang with C++14
Nick Terrell [Thu, 9 Feb 2017 01:38:17 +0000 (17:38 -0800)] 
Fix deprecation warnings for clang with C++14

8 years agoFix ZSTD_getErrorString and add tests 538/head
Sean Purcell [Wed, 8 Feb 2017 23:31:47 +0000 (15:31 -0800)] 
Fix ZSTD_getErrorString and add tests

8 years ago#undef _POSIX_C_SOURCE if already defined
Nick Terrell [Thu, 9 Feb 2017 01:25:01 +0000 (17:25 -0800)] 
#undef _POSIX_C_SOURCE if already defined

8 years agoTest multiframe legacy decoding with simple and streaming APIs
Sean Purcell [Wed, 8 Feb 2017 21:56:32 +0000 (13:56 -0800)] 
Test multiframe legacy decoding with simple and streaming APIs

8 years agoDisambiguate pledgedSrcSize == 0
Sean Purcell [Wed, 8 Feb 2017 00:33:48 +0000 (16:33 -0800)] 
Disambiguate pledgedSrcSize == 0

- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize

8 years agoZSTD_decompress now handles multiple frames
Sean Purcell [Wed, 8 Feb 2017 00:16:55 +0000 (16:16 -0800)] 
ZSTD_decompress now handles multiple frames

8 years agoDecompressed size functions now handle multiframes and distinguish cases
Sean Purcell [Tue, 7 Feb 2017 21:50:09 +0000 (13:50 -0800)] 
Decompressed size functions now handle multiframes and distinguish cases

- Add ZSTD_findDecompressedSize
    - Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
    - Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize

8 years agoremove redundant "ifeq ($(HAVE_ZLIB), 1)"
Przemyslaw Skibinski [Wed, 8 Feb 2017 20:11:18 +0000 (21:11 +0100)] 
remove redundant "ifeq ($(HAVE_ZLIB), 1)"

8 years agoRemove test-longmatch from test target and only run it once 535/head
Nick Terrell [Wed, 8 Feb 2017 20:00:21 +0000 (12:00 -0800)] 
Remove test-longmatch from test target and only run it once

8 years agofixed zlib detection with MinGW
Przemyslaw Skibinski [Wed, 8 Feb 2017 13:32:49 +0000 (14:32 +0100)] 
fixed zlib detection with MinGW

8 years agoturn off test-pool for qemu-ppc64-static
Przemyslaw Skibinski [Wed, 8 Feb 2017 12:58:04 +0000 (13:58 +0100)] 
turn off test-pool for qemu-ppc64-static

8 years agoMerge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Przemyslaw Skibinski [Wed, 8 Feb 2017 12:49:35 +0000 (13:49 +0100)] 
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11

8 years agoadded description of "--priority=rt"
Przemyslaw Skibinski [Wed, 8 Feb 2017 12:49:06 +0000 (13:49 +0100)] 
added description of "--priority=rt"

8 years agoMerge pull request #526 from iburinoc/educational
Yann Collet [Tue, 7 Feb 2017 23:34:33 +0000 (15:34 -0800)] 
Merge pull request #526 from iburinoc/educational

Add educational decoder to /contrib

8 years agoMinor changes to educational decoder 526/head
Sean Purcell [Tue, 7 Feb 2017 22:44:11 +0000 (14:44 -0800)] 
Minor changes to educational decoder

8 years agocompleted NEWS for v1.1.3
Yann Collet [Tue, 7 Feb 2017 20:05:28 +0000 (12:05 -0800)] 
completed NEWS for v1.1.3

8 years agoMerge pull request #534 from terrelln/dev
Yann Collet [Tue, 7 Feb 2017 19:58:59 +0000 (11:58 -0800)] 
Merge pull request #534 from terrelln/dev

Attribute cover dictionary code

8 years agoAttribute cover dictionary code 534/head
Nick Terrell [Tue, 7 Feb 2017 19:35:07 +0000 (11:35 -0800)] 
Attribute cover dictionary code

8 years ago"--rt-prio" renamed to "--priority=rt"
Przemyslaw Skibinski [Tue, 7 Feb 2017 19:12:59 +0000 (20:12 +0100)] 
"--rt-prio" renamed to "--priority=rt"

8 years agoadded the "--rt-prio" option
Przemyslaw Skibinski [Tue, 7 Feb 2017 15:48:01 +0000 (16:48 +0100)] 
added the "--rt-prio" option

8 years agoSET_REALTIME_PRIORITY
Przemyslaw Skibinski [Tue, 7 Feb 2017 15:36:19 +0000 (16:36 +0100)] 
SET_REALTIME_PRIORITY

8 years agoMerge branch 'dev' of github.com:facebook/zstd into dev
Yann Collet [Tue, 7 Feb 2017 01:40:38 +0000 (17:40 -0800)] 
Merge branch 'dev' of github.com:facebook/zstd into dev

8 years agocorrected contributor's name
Yann Collet [Tue, 7 Feb 2017 01:39:54 +0000 (17:39 -0800)] 
corrected contributor's name

8 years agoMerge pull request #532 from terrelln/zlib
Yann Collet [Mon, 6 Feb 2017 22:44:23 +0000 (14:44 -0800)] 
Merge pull request #532 from terrelln/zlib

Change zlib include to be a system include

8 years agoEnsure <zlib.h> can be included in HAVE_ZLIB test 532/head
Nick Terrell [Mon, 6 Feb 2017 19:54:31 +0000 (11:54 -0800)] 
Ensure <zlib.h> can be included in HAVE_ZLIB test

8 years agoChange zlib include to be a system include
Nick Terrell [Mon, 6 Feb 2017 19:32:13 +0000 (11:32 -0800)] 
Change zlib include to be a system include

8 years agoplatform.h added to build_package.bat
Przemyslaw Skibinski [Mon, 6 Feb 2017 18:43:13 +0000 (19:43 +0100)] 
platform.h added to build_package.bat

8 years agochanged download URL for github_users sample set
Yann Collet [Mon, 6 Feb 2017 18:01:06 +0000 (10:01 -0800)] 
changed download URL for github_users sample set

8 years agoMerge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Przemyslaw Skibinski [Mon, 6 Feb 2017 10:21:51 +0000 (11:21 +0100)] 
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11

8 years agoupdated NEWS for v1.1.3 531/head
Yann Collet [Mon, 6 Feb 2017 02:09:35 +0000 (18:09 -0800)] 
updated NEWS for v1.1.3

8 years agoMerge pull request #529 from facebook/gzstd_default
Yann Collet [Mon, 6 Feb 2017 02:03:36 +0000 (18:03 -0800)] 
Merge pull request #529 from facebook/gzstd_default

Gzstd default

8 years agofixed Mac OS-X specific directory in $(RM) list 529/head
Yann Collet [Sun, 5 Feb 2017 18:22:58 +0000 (10:22 -0800)] 
fixed Mac OS-X specific directory in $(RM) list

these directories are now removed with -r command

8 years agoSilence zlib detection routine
cyan4973 [Sun, 5 Feb 2017 07:36:12 +0000 (23:36 -0800)] 
Silence zlib detection routine

When it fails, $(CC) sends error message into stderr
redirected to /dev/null

8 years agoremoved gzstd test from travis
cyan4973 [Sun, 5 Feb 2017 05:39:41 +0000 (21:39 -0800)] 
removed gzstd test from travis

this target does no longer exist

8 years agoInlined portions of specification for clarity
Sean Purcell [Sat, 4 Feb 2017 02:04:00 +0000 (18:04 -0800)] 
Inlined portions of specification for clarity

8 years agoMerge pull request #530 from terrelln/cover-man
Yann Collet [Sat, 4 Feb 2017 01:10:06 +0000 (17:10 -0800)] 
Merge pull request #530 from terrelln/cover-man

Add cover dictionary training to zstd.1

8 years agoAdd cover dictionary training to zstd.1 530/head
Nick Terrell [Sat, 4 Feb 2017 00:42:07 +0000 (16:42 -0800)] 
Add cover dictionary training to zstd.1

Tested with `make install && man zstd` and visual inspection.

8 years agoSwitch IO to go through streams
Sean Purcell [Thu, 2 Feb 2017 01:05:45 +0000 (17:05 -0800)] 
Switch IO to go through streams

8 years agofixed zstdmt compilation under Windows minGW/MSYS2, by @inikep
Yann Collet [Fri, 3 Feb 2017 22:35:42 +0000 (14:35 -0800)] 
fixed zstdmt compilation under Windows minGW/MSYS2, by @inikep

8 years agoremove zlib detection artefact
Yann Collet [Fri, 3 Feb 2017 22:25:07 +0000 (14:25 -0800)] 
remove zlib detection artefact

result of compilation test is sent to /dev/null

8 years agofixed pointer conversion warnings (C++)
Yann Collet [Fri, 3 Feb 2017 16:43:06 +0000 (08:43 -0800)] 
fixed pointer conversion warnings (C++)

in gz module

8 years agoMerge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Przemyslaw Skibinski [Fri, 3 Feb 2017 10:03:00 +0000 (11:03 +0100)] 
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11

8 years agorelease builds use less debug symbols and warnings
Yann Collet [Fri, 3 Feb 2017 04:54:14 +0000 (20:54 -0800)] 
release builds use less debug symbols and warnings

release build are triggered through either `make`,
or their specific target `make zstd-release` and `make lib-release`.

8 years agofixed silent conversion warnings in GZDECOMPRESS path
Yann Collet [Fri, 3 Feb 2017 01:12:50 +0000 (17:12 -0800)] 
fixed silent conversion warnings in GZDECOMPRESS path

8 years agoChanged Makefile to generate zstd with .gz support by default
Yann Collet [Fri, 3 Feb 2017 00:49:34 +0000 (16:49 -0800)] 
Changed Makefile to generate zstd with .gz support by default

.gz support is detected by a runtime test.

8 years agominor : fixed zstd-frugal
Yann Collet [Fri, 3 Feb 2017 00:19:09 +0000 (16:19 -0800)] 
minor : fixed zstd-frugal

fixed a minor unused variable warning when compiling zstd-frugal target

8 years agoremoved unused chart
Yann Collet [Thu, 2 Feb 2017 21:50:51 +0000 (13:50 -0800)] 
removed unused chart

8 years agoupdated dictionary compression paragraph
Yann Collet [Thu, 2 Feb 2017 21:50:07 +0000 (13:50 -0800)] 
updated dictionary compression paragraph

8 years agoMinor security fixes
Sean Purcell [Wed, 1 Feb 2017 18:41:04 +0000 (10:41 -0800)] 
Minor security fixes

8 years agoMerge pull request #527 from facebook/zstdmt
Yann Collet [Wed, 1 Feb 2017 00:36:46 +0000 (16:36 -0800)] 
Merge pull request #527 from facebook/zstdmt

zstdmt refinements

8 years agoMore const's and readability improvements
Sean Purcell [Tue, 31 Jan 2017 23:57:18 +0000 (15:57 -0800)] 
More const's and readability improvements

8 years agoRemoved circular buffer, matches access destination buffer directly
Sean Purcell [Tue, 31 Jan 2017 23:54:02 +0000 (15:54 -0800)] 
Removed circular buffer, matches access destination buffer directly

8 years agoMerge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Przemyslaw Skibinski [Tue, 31 Jan 2017 22:59:17 +0000 (23:59 +0100)] 
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11

8 years agoMinor fixes according to comments
Sean Purcell [Mon, 30 Jan 2017 22:57:02 +0000 (14:57 -0800)] 
Minor fixes according to comments

- Add Facebook copyright notice
- Make max size macros more consistent
- Fix some unchecked malloc's

8 years agoAdded ZSTD_get_decompressed_size
Sean Purcell [Mon, 30 Jan 2017 22:42:21 +0000 (14:42 -0800)] 
Added ZSTD_get_decompressed_size

Since this implementation handles multiple concatenated frames,
to determine decompressed size we must traverse the entire input,
checking each frame's frame_content_size field

8 years agofixed overlapLog==0 => no overlap 527/head
Yann Collet [Mon, 30 Jan 2017 22:54:46 +0000 (14:54 -0800)] 
fixed overlapLog==0 => no overlap

8 years agofixed C constant restrictions
Yann Collet [Mon, 30 Jan 2017 22:37:08 +0000 (14:37 -0800)] 
fixed C constant restrictions

8 years agozstdmt : section size is set to be a minimum of overlapSize
Yann Collet [Mon, 30 Jan 2017 21:35:45 +0000 (13:35 -0800)] 
zstdmt : section size is set to be a minimum of overlapSize

the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.

8 years agocli : added advanced parameter overlapLog
Yann Collet [Mon, 30 Jan 2017 21:07:24 +0000 (13:07 -0800)] 
cli : added advanced parameter overlapLog

as a hidden (undocumented) parameter for now

8 years agozstreamtest uses random section sizes for fuzzing
Yann Collet [Mon, 30 Jan 2017 20:50:31 +0000 (12:50 -0800)] 
zstreamtest uses random section sizes for fuzzing

8 years agozstreamtest uses random overlapLog for fuzzing
Yann Collet [Mon, 30 Jan 2017 20:46:35 +0000 (12:46 -0800)] 
zstreamtest uses random overlapLog for fuzzing

8 years agoAdd educational decoder to /contrib
Sean Purcell [Mon, 30 Jan 2017 19:42:45 +0000 (11:42 -0800)] 
Add educational decoder to /contrib

8 years agoadded command --block-size=
Yann Collet [Mon, 30 Jan 2017 19:17:26 +0000 (11:17 -0800)] 
added command --block-size=

for Multi-threading only.
alias : -B#

8 years agochanged advanced parameter overlapLog
Yann Collet [Mon, 30 Jan 2017 19:00:00 +0000 (11:00 -0800)] 
changed advanced parameter overlapLog

Follows a positive logic (increasing value => increasing overlap)
which is easier to use

8 years agofixed : legacy decoders v04 and v05
Yann Collet [Mon, 30 Jan 2017 18:45:58 +0000 (10:45 -0800)] 
fixed : legacy decoders v04 and v05