]>
git.ipfire.org Git - thirdparty/zstd.git/log
Nick Terrell [Tue, 14 Feb 2017 18:08:14 +0000 (10:08 -0800)]
Don't run test-pool with QEMU
> make test -n
...
./pool
> make test -n QEMU_SYS=valgrind
...
./legacy
# ./pool not run
Nick Terrell [Tue, 14 Feb 2017 02:27:34 +0000 (18:27 -0800)]
[zstdmt] Fix MSAN failure with ZSTD_p_forceWindow
Reproduction steps:
```
make zstreamtest CC=clang CFLAGS="-O3 -g -fsanitize=memory -fsanitize-memory-track-origins"
./zstreamtest -vv -t4178 -i4178 -s4531
```
How to get to the error in gdb (may be a more efficient way):
* 2 breaks at zstd_compress.c:2418 -- in ZSTD_compressContinue_internal()
* 2 breaks at zstd_compress.c:2276 -- in ZSTD_compressBlock_internal()
* 1 break at zstd_compress.c:1547
Why the error occurred:
When `zc->forceWindow == 1`, after calling `ZSTD_loadDictionaryContent()` we
have `zc->loadedDictEnd == zc->nextToUpdate == 0`. But, we've really loaded up
to `iend` into the dictionary. Then in `ZSTD_compressBlock_internal()` we see
that `current > zc->nextToUpdate + 384`, so we load the last 192 bytes a second
time. In this case the bytes we are loading are a block of all 0s, starting in
the previous block. So when we are loading the last 192 bytes, we find a `match`
in the future, 183 bytes beyond `ip`. Since the block is all 0s, the match
extends to the end of the block. But in `ZSTD_count()` we only check that
`pIn < pInLoopLimit`, but since `pMatch > pIn`, `pMatch` eventually points past
the end of the buffer, causing the MSAN failure.
The fix:
The line changed sets sets `zc->nextToUpdate` to the end of the dictionary.
This is the behavior that existed before `ZSTD_p_forceWindow` was introduced.
This fixes the exposing test case. Since the code doesn't fail without
`zc->forceWindow`, it makes sense that this works. I've run the command
`./zstreamtest -T2mn` 64 times without failures. CI should also verify nothing
obvious broke.
Yann Collet [Mon, 13 Feb 2017 20:18:16 +0000 (12:18 -0800)]
Merge pull request #536 from iburinoc/multiframe
Simple API multiframe decoding
Yann Collet [Mon, 13 Feb 2017 19:57:35 +0000 (11:57 -0800)]
Merge pull request #542 from ds77/large-sparse-fix
zstdcli: fix writing 2GB+ sparse files under Windows
ds77 [Mon, 13 Feb 2017 11:00:59 +0000 (12:00 +0100)]
fix indentation in previous commit
ds77 [Sun, 12 Feb 2017 09:27:18 +0000 (10:27 +0100)]
fix seeking 2GB+ files under Windows
Replace fseek() in FIO_fwriteSparse() and FIO_fwriteSparseEnd() with macro expanding to 64-bit fseek version provided by the platform (includes fallback workaround using Win32 API).
Yann Collet [Sun, 12 Feb 2017 05:08:08 +0000 (21:08 -0800)]
Merge pull request #540 from ds77/dev-stat64-fix
zstdcli: Fix reporting incorrect sizes of large flies on MinGW
Yann Collet [Sun, 12 Feb 2017 05:07:53 +0000 (21:07 -0800)]
Merge pull request #541 from inikep/mingw
fixed "mingw32" AppVeyor target
Przemyslaw Skibinski [Fri, 10 Feb 2017 20:15:49 +0000 (21:15 +0100)]
revert last commit
Sean Purcell [Fri, 10 Feb 2017 19:38:57 +0000 (11:38 -0800)]
Expose frameSrcSize to experimental API
Przemyslaw Skibinski [Fri, 10 Feb 2017 19:09:28 +0000 (20:09 +0100)]
fix for original MinGW
Sean Purcell [Fri, 10 Feb 2017 18:08:55 +0000 (10:08 -0800)]
Merge branch 'dev' into multiframe
ds77 [Fri, 10 Feb 2017 17:37:57 +0000 (18:37 +0100)]
use _stati64() in UTIL_getFileSize() when compiling with mingw, get rid of introduces previously preprocessor checks.
Przemyslaw Skibinski [Fri, 10 Feb 2017 12:38:54 +0000 (13:38 +0100)]
Merge remote-tracking branch 'refs/remotes/facebook/dev' into mingw
# Conflicts:
# programs/platform.h
Yann Collet [Fri, 10 Feb 2017 12:35:43 +0000 (04:35 -0800)]
Merge pull request #537 from terrelln/small-bugs
Fix small bugs
- [Fri, 10 Feb 2017 12:27:43 +0000 (13:27 +0100)]
fix previous commit
* struct _stat64 is not defined by (non-w64) MinGW releases, __stat64 should be everywhere
* proper detection of _stat64() availability (as in MinGW sys/stat.h)
Przemyslaw Skibinski [Fri, 10 Feb 2017 12:10:25 +0000 (13:10 +0100)]
appveyor.yml: fixed clang test
Yann Collet [Fri, 10 Feb 2017 11:59:56 +0000 (03:59 -0800)]
Merge pull request #538 from iburinoc/errorstring
Fix ZSTD_getErrorString and add tests
Yann Collet [Fri, 10 Feb 2017 11:50:08 +0000 (03:50 -0800)]
Merge pull request #533 from inikep/dev11
--priority=rt
Przemyslaw Skibinski [Fri, 10 Feb 2017 11:41:31 +0000 (12:41 +0100)]
appveyor.yml: reordering of tests
Przemyslaw Skibinski [Fri, 10 Feb 2017 11:32:30 +0000 (12:32 +0100)]
fix missing "
Przemyslaw Skibinski [Fri, 10 Feb 2017 11:01:14 +0000 (12:01 +0100)]
appveyor.yml: add clang target
Przemyslaw Skibinski [Fri, 10 Feb 2017 10:01:52 +0000 (11:01 +0100)]
fixed "mingw32" AppVeyor target
- [Fri, 10 Feb 2017 09:56:45 +0000 (10:56 +0100)]
use _stat64 only when targetting Win2k or later
Przemyslaw Skibinski [Fri, 10 Feb 2017 09:36:44 +0000 (10:36 +0100)]
fix 64-bit file support for MinGW
Nick Terrell [Thu, 9 Feb 2017 22:20:52 +0000 (14:20 -0800)]
Improvement from @inikep
ds77 [Thu, 9 Feb 2017 17:12:00 +0000 (18:12 +0100)]
use _stat64() on MinGW
On MinGW, use _stat64() and struct _stat64 instead of stat() and struct stat_t. This fixes reporting incorrect sizes for large files.
Sean Purcell [Thu, 9 Feb 2017 21:25:30 +0000 (13:25 -0800)]
Documentation updates
Sean Purcell [Thu, 9 Feb 2017 20:27:32 +0000 (12:27 -0800)]
Fix failing unit test
Sean Purcell [Thu, 9 Feb 2017 19:54:22 +0000 (11:54 -0800)]
Merge branch 'dev' into multiframe
Sean Purcell [Thu, 9 Feb 2017 18:50:43 +0000 (10:50 -0800)]
Make pledgedSrcSize meaning clear for other functions
- Added tests
- Moved new size functions to static link only
Przemyslaw Skibinski [Thu, 9 Feb 2017 16:01:17 +0000 (17:01 +0100)]
echo replaced with printf
Przemyslaw Skibinski [Thu, 9 Feb 2017 10:32:14 +0000 (11:32 +0100)]
don't use "echo -e"
Przemyslaw Skibinski [Thu, 9 Feb 2017 08:42:16 +0000 (09:42 +0100)]
Merge branch 'dev' into dev11
Yann Collet [Thu, 9 Feb 2017 08:36:03 +0000 (00:36 -0800)]
Merge pull request #535 from terrelln/tests
Remove test-longmatch from test target and only run it once
Nick Terrell [Thu, 9 Feb 2017 01:38:17 +0000 (17:38 -0800)]
Fix deprecation warnings for clang with C++14
Sean Purcell [Wed, 8 Feb 2017 23:31:47 +0000 (15:31 -0800)]
Fix ZSTD_getErrorString and add tests
Nick Terrell [Thu, 9 Feb 2017 01:25:01 +0000 (17:25 -0800)]
#undef _POSIX_C_SOURCE if already defined
Sean Purcell [Wed, 8 Feb 2017 21:56:32 +0000 (13:56 -0800)]
Test multiframe legacy decoding with simple and streaming APIs
Sean Purcell [Wed, 8 Feb 2017 00:33:48 +0000 (16:33 -0800)]
Disambiguate pledgedSrcSize == 0
- Modify ZSTD CLI to only set contentSizeFlag if it _knows_ the size
- Change pzstd to stop setting contentSizeFlag without accurate pledgedSrcSize
Sean Purcell [Wed, 8 Feb 2017 00:16:55 +0000 (16:16 -0800)]
ZSTD_decompress now handles multiple frames
Sean Purcell [Tue, 7 Feb 2017 21:50:09 +0000 (13:50 -0800)]
Decompressed size functions now handle multiframes and distinguish cases
- Add ZSTD_findDecompressedSize
- Traverses multiple frames to find total output size
- Add ZSTD_getFrameContentSize
- Gets the decompressed size of a single frame by reading header
- Deprecate ZSTD_getDecompressedSize
Przemyslaw Skibinski [Wed, 8 Feb 2017 20:11:18 +0000 (21:11 +0100)]
remove redundant "ifeq ($(HAVE_ZLIB), 1)"
Nick Terrell [Wed, 8 Feb 2017 20:00:21 +0000 (12:00 -0800)]
Remove test-longmatch from test target and only run it once
Przemyslaw Skibinski [Wed, 8 Feb 2017 13:32:49 +0000 (14:32 +0100)]
fixed zlib detection with MinGW
Przemyslaw Skibinski [Wed, 8 Feb 2017 12:58:04 +0000 (13:58 +0100)]
turn off test-pool for qemu-ppc64-static
Przemyslaw Skibinski [Wed, 8 Feb 2017 12:49:35 +0000 (13:49 +0100)]
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Przemyslaw Skibinski [Wed, 8 Feb 2017 12:49:06 +0000 (13:49 +0100)]
added description of "--priority=rt"
Yann Collet [Tue, 7 Feb 2017 23:34:33 +0000 (15:34 -0800)]
Merge pull request #526 from iburinoc/educational
Add educational decoder to /contrib
Sean Purcell [Tue, 7 Feb 2017 22:44:11 +0000 (14:44 -0800)]
Minor changes to educational decoder
Yann Collet [Tue, 7 Feb 2017 20:05:28 +0000 (12:05 -0800)]
completed NEWS for v1.1.3
Yann Collet [Tue, 7 Feb 2017 19:58:59 +0000 (11:58 -0800)]
Merge pull request #534 from terrelln/dev
Attribute cover dictionary code
Nick Terrell [Tue, 7 Feb 2017 19:35:07 +0000 (11:35 -0800)]
Attribute cover dictionary code
Przemyslaw Skibinski [Tue, 7 Feb 2017 19:12:59 +0000 (20:12 +0100)]
"--rt-prio" renamed to "--priority=rt"
Przemyslaw Skibinski [Tue, 7 Feb 2017 15:48:01 +0000 (16:48 +0100)]
added the "--rt-prio" option
Przemyslaw Skibinski [Tue, 7 Feb 2017 15:36:19 +0000 (16:36 +0100)]
SET_REALTIME_PRIORITY
Yann Collet [Tue, 7 Feb 2017 01:40:38 +0000 (17:40 -0800)]
Merge branch 'dev' of github.com:facebook/zstd into dev
Yann Collet [Tue, 7 Feb 2017 01:39:54 +0000 (17:39 -0800)]
corrected contributor's name
Yann Collet [Mon, 6 Feb 2017 22:44:23 +0000 (14:44 -0800)]
Merge pull request #532 from terrelln/zlib
Change zlib include to be a system include
Nick Terrell [Mon, 6 Feb 2017 19:54:31 +0000 (11:54 -0800)]
Ensure <zlib.h> can be included in HAVE_ZLIB test
Nick Terrell [Mon, 6 Feb 2017 19:32:13 +0000 (11:32 -0800)]
Change zlib include to be a system include
Przemyslaw Skibinski [Mon, 6 Feb 2017 18:43:13 +0000 (19:43 +0100)]
platform.h added to build_package.bat
Yann Collet [Mon, 6 Feb 2017 18:01:06 +0000 (10:01 -0800)]
changed download URL for github_users sample set
Przemyslaw Skibinski [Mon, 6 Feb 2017 10:21:51 +0000 (11:21 +0100)]
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Yann Collet [Mon, 6 Feb 2017 02:09:35 +0000 (18:09 -0800)]
updated NEWS for v1.1.3
Yann Collet [Mon, 6 Feb 2017 02:03:36 +0000 (18:03 -0800)]
Merge pull request #529 from facebook/gzstd_default
Gzstd default
Yann Collet [Sun, 5 Feb 2017 18:22:58 +0000 (10:22 -0800)]
fixed Mac OS-X specific directory in $(RM) list
these directories are now removed with -r command
cyan4973 [Sun, 5 Feb 2017 07:36:12 +0000 (23:36 -0800)]
Silence zlib detection routine
When it fails, $(CC) sends error message into stderr
redirected to /dev/null
cyan4973 [Sun, 5 Feb 2017 05:39:41 +0000 (21:39 -0800)]
removed gzstd test from travis
this target does no longer exist
Sean Purcell [Sat, 4 Feb 2017 02:04:00 +0000 (18:04 -0800)]
Inlined portions of specification for clarity
Yann Collet [Sat, 4 Feb 2017 01:10:06 +0000 (17:10 -0800)]
Merge pull request #530 from terrelln/cover-man
Add cover dictionary training to zstd.1
Nick Terrell [Sat, 4 Feb 2017 00:42:07 +0000 (16:42 -0800)]
Add cover dictionary training to zstd.1
Tested with `make install && man zstd` and visual inspection.
Sean Purcell [Thu, 2 Feb 2017 01:05:45 +0000 (17:05 -0800)]
Switch IO to go through streams
Yann Collet [Fri, 3 Feb 2017 22:35:42 +0000 (14:35 -0800)]
fixed zstdmt compilation under Windows minGW/MSYS2, by @inikep
Yann Collet [Fri, 3 Feb 2017 22:25:07 +0000 (14:25 -0800)]
remove zlib detection artefact
result of compilation test is sent to /dev/null
Yann Collet [Fri, 3 Feb 2017 16:43:06 +0000 (08:43 -0800)]
fixed pointer conversion warnings (C++)
in gz module
Przemyslaw Skibinski [Fri, 3 Feb 2017 10:03:00 +0000 (11:03 +0100)]
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Yann Collet [Fri, 3 Feb 2017 04:54:14 +0000 (20:54 -0800)]
release builds use less debug symbols and warnings
release build are triggered through either `make`,
or their specific target `make zstd-release` and `make lib-release`.
Yann Collet [Fri, 3 Feb 2017 01:12:50 +0000 (17:12 -0800)]
fixed silent conversion warnings in GZDECOMPRESS path
Yann Collet [Fri, 3 Feb 2017 00:49:34 +0000 (16:49 -0800)]
Changed Makefile to generate zstd with .gz support by default
.gz support is detected by a runtime test.
Yann Collet [Fri, 3 Feb 2017 00:19:09 +0000 (16:19 -0800)]
minor : fixed zstd-frugal
fixed a minor unused variable warning when compiling zstd-frugal target
Yann Collet [Thu, 2 Feb 2017 21:50:51 +0000 (13:50 -0800)]
removed unused chart
Yann Collet [Thu, 2 Feb 2017 21:50:07 +0000 (13:50 -0800)]
updated dictionary compression paragraph
Sean Purcell [Wed, 1 Feb 2017 18:41:04 +0000 (10:41 -0800)]
Minor security fixes
Yann Collet [Wed, 1 Feb 2017 00:36:46 +0000 (16:36 -0800)]
Merge pull request #527 from facebook/zstdmt
zstdmt refinements
Sean Purcell [Tue, 31 Jan 2017 23:57:18 +0000 (15:57 -0800)]
More const's and readability improvements
Sean Purcell [Tue, 31 Jan 2017 23:54:02 +0000 (15:54 -0800)]
Removed circular buffer, matches access destination buffer directly
Przemyslaw Skibinski [Tue, 31 Jan 2017 22:59:17 +0000 (23:59 +0100)]
Merge remote-tracking branch 'refs/remotes/facebook/dev' into dev11
Sean Purcell [Mon, 30 Jan 2017 22:57:02 +0000 (14:57 -0800)]
Minor fixes according to comments
- Add Facebook copyright notice
- Make max size macros more consistent
- Fix some unchecked malloc's
Sean Purcell [Mon, 30 Jan 2017 22:42:21 +0000 (14:42 -0800)]
Added ZSTD_get_decompressed_size
Since this implementation handles multiple concatenated frames,
to determine decompressed size we must traverse the entire input,
checking each frame's frame_content_size field
Yann Collet [Mon, 30 Jan 2017 22:54:46 +0000 (14:54 -0800)]
fixed overlapLog==0 => no overlap
Yann Collet [Mon, 30 Jan 2017 22:37:08 +0000 (14:37 -0800)]
fixed C constant restrictions
Yann Collet [Mon, 30 Jan 2017 21:35:45 +0000 (13:35 -0800)]
zstdmt : section size is set to be a minimum of overlapSize
the minimum size condition size is applied transparently (no warning, no error)
like previous minimum section size condition (1 KB) which still applies.
Yann Collet [Mon, 30 Jan 2017 21:07:24 +0000 (13:07 -0800)]
cli : added advanced parameter overlapLog
as a hidden (undocumented) parameter for now
Yann Collet [Mon, 30 Jan 2017 20:50:31 +0000 (12:50 -0800)]
zstreamtest uses random section sizes for fuzzing
Yann Collet [Mon, 30 Jan 2017 20:46:35 +0000 (12:46 -0800)]
zstreamtest uses random overlapLog for fuzzing
Sean Purcell [Mon, 30 Jan 2017 19:42:45 +0000 (11:42 -0800)]
Add educational decoder to /contrib
Yann Collet [Mon, 30 Jan 2017 19:17:26 +0000 (11:17 -0800)]
added command --block-size=
for Multi-threading only.
alias : -B#
Yann Collet [Mon, 30 Jan 2017 19:00:00 +0000 (11:00 -0800)]
changed advanced parameter overlapLog
Follows a positive logic (increasing value => increasing overlap)
which is easier to use
Yann Collet [Mon, 30 Jan 2017 18:45:58 +0000 (10:45 -0800)]
fixed : legacy decoders v04 and v05