]>
git.ipfire.org Git - thirdparty/zstd.git/log
Lzu Tao [Tue, 27 Nov 2018 18:04:41 +0000 (01:04 +0700)]
Update meson build system
NOTE: This commit only tested on Linux (Ubuntu 18.04). Windows
build may not work as expected.
* Use meson >= 0.47.0 cause we use install_man function
* Add three helper Python script:
* CopyFile.py: To copy file
* CreateSymlink.py: To make symlink (both Windows and Unix)
* GetZstdLibraryVersion.py: Parse lib/zstd.h to get zstd version
These help emulating equivalent functions in CMake and Makefile.
* Use subdir from meson to split meson.build
* Add contrib build
* Fix other build
* Add new build options
* build_programs: Enable programs build
* build_contrib: Enable contrib build
* build_tests: Enable tests build
* use_static_runtime: Link to static run-time libraries on MSVC
* zlib_support: Enable zlib support
* lzma_support: Enable lzma support
Lzu Tao [Tue, 27 Nov 2018 17:10:04 +0000 (00:10 +0700)]
Rename and update build instruction in README file to README.md
Lzu Tao [Mon, 26 Nov 2018 17:24:33 +0000 (00:24 +0700)]
Add meson build guide
Yann Collet [Tue, 27 Nov 2018 01:37:19 +0000 (17:37 -0800)]
Merge pull request #1419 from ryandesign/universal
Fix feature detection with multiple -arch flags
Nick Terrell [Fri, 16 Nov 2018 19:01:46 +0000 (11:01 -0800)]
Merge pull request #1418 from ryandesign/cpuid-i386
Fix i386 build failure "Junk character 13"
Ryan Schmidt [Fri, 16 Nov 2018 09:49:15 +0000 (03:49 -0600)]
Fix feature detection with multiple -arch flags
When multiple -arch flags are used, the compiler invokes itself once for
each architecture. Apparently, input on stdin is consumed by the
compilation of the first arch and is no longer available to the
compilation of the second arch, which results in a build failure and the
potentially incorrect determination that a feature is not available. So
write the feature detection source to a file instead of using stdin.
Ryan Schmidt [Fri, 16 Nov 2018 08:16:21 +0000 (02:16 -0600)]
Fix i386 build failure "Junk character 13"
Nick Terrell [Thu, 15 Nov 2018 02:43:12 +0000 (18:43 -0800)]
Merge pull request #1416 from terrelln/rsync
- Factor out LDM's hash function for reuse
- Add rsyncable mode to zstdmt and expose it via the advanced API
- Fix `-B`/`--block-size` to actually set the job size
- Add rsyncable tests to `zstreamtest` and `playTests.sh`
Tested by:
```
> cat A.100MB B.100MB C.100MB D.100MB E.100MB | zstd --rsyncable -fo src/file.zst
/*stdin*\ : 48.22% (
524288000 =>
252837782 bytes, src/file.zst)
> rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
Total bytes sent:
252868779
total size is
252837782 speedup is 1.00
> echo test > test
> cat test A.100MB test B.100MB test C.100MB test D.100MB test E.100MB | zstd --rsyncable -fo src/file.zst
/*stdin*\ : 48.23% (
524288025 =>
252838025 bytes, src/unicorn.tar.zst)
> rsync -rc --stats src devbigvm:/data/users/terrelln/rsync-test
Total bytes sent:
4605696
total size is
252838025 speedup is 53.60
```
Close #1155.
Nick Terrell [Wed, 14 Nov 2018 21:53:14 +0000 (13:53 -0800)]
[tests] Add rsyncable tests
Nick Terrell [Wed, 14 Nov 2018 20:59:01 +0000 (12:59 -0800)]
[cli] Set the job size
Nick Terrell [Tue, 13 Nov 2018 03:59:42 +0000 (19:59 -0800)]
[cli] Add rsyncable
Nick Terrell [Wed, 14 Nov 2018 20:15:20 +0000 (12:15 -0800)]
[lib] Add rsyncable mode
- Add rsyncable mode to multithreaded mode
- Factor out LDM's hash function for reuse
Yann Collet [Wed, 14 Nov 2018 21:00:08 +0000 (13:00 -0800)]
Merge pull request #1415 from facebook/benchfn
Separating benchfn and benchzstd
Yann Collet [Wed, 14 Nov 2018 00:04:35 +0000 (16:04 -0800)]
circumvent gcc-8 strict aliasing warning
because gcc considers that
`void* const *` and `void**` are 2 different types ...
Yann Collet [Tue, 13 Nov 2018 23:54:05 +0000 (15:54 -0800)]
Merge branch 'dev' of github.com:facebook/zstd into dev
Yann Collet [Tue, 13 Nov 2018 23:53:38 +0000 (15:53 -0800)]
bumped version number to v1.3.8
Yann Collet [Tue, 13 Nov 2018 22:15:12 +0000 (14:15 -0800)]
fixed decode-only test condition
Yann Collet [Tue, 13 Nov 2018 21:27:22 +0000 (13:27 -0800)]
fix debug trace
Yann Collet [Tue, 13 Nov 2018 21:22:16 +0000 (13:22 -0800)]
fixed VS2010+ build script
Yann Collet [Tue, 13 Nov 2018 21:16:41 +0000 (13:16 -0800)]
fixed cmake build script
Yann Collet [Tue, 13 Nov 2018 21:05:39 +0000 (13:05 -0800)]
changed benchfn api
to use structure for function parameters
as it expresses much clearer than a long list of parameters,
since each parameter can now be named.
Yann Collet [Tue, 13 Nov 2018 20:01:17 +0000 (12:01 -0800)]
benchfn can provided faulty return value
with BMK_extract_returnValue()
Yann Collet [Tue, 13 Nov 2018 19:01:59 +0000 (11:01 -0800)]
separated bench module into benchfn and benchzstd
it shall be possible to use benchfn
without any dependency on zstd.
Yann Collet [Tue, 13 Nov 2018 01:53:11 +0000 (17:53 -0800)]
Merge pull request #1413 from felixhandte/attach-dict-fix-unsigned-compare
Fix #1412: Perform Signed Comparison When Setting Attach Dict Param
Yann Collet [Tue, 13 Nov 2018 01:05:32 +0000 (17:05 -0800)]
changed PREFETCH() macro into PREFETCH_L2()
which is more accurate
W. Felix Handte [Mon, 12 Nov 2018 22:48:42 +0000 (14:48 -0800)]
Const
W. Felix Handte [Mon, 12 Nov 2018 21:01:14 +0000 (13:01 -0800)]
Fix Fuzz Range
W. Felix Handte [Mon, 12 Nov 2018 20:52:31 +0000 (12:52 -0800)]
Fix Types
W. Felix Handte [Mon, 12 Nov 2018 20:47:47 +0000 (12:47 -0800)]
Switch Enum to Only Non-Negative Values, Update Comments
W. Felix Handte [Mon, 12 Nov 2018 20:07:57 +0000 (12:07 -0800)]
Fix #1412: Perform Signed Comparison When Setting Attach Dict Param
Yann Collet [Fri, 9 Nov 2018 19:31:35 +0000 (11:31 -0800)]
Merge pull request #1411 from facebook/prefetch_dict
Improves decompression speed when using cold dictionary
Yann Collet [Fri, 9 Nov 2018 02:41:58 +0000 (18:41 -0800)]
Merge pull request #1410 from facebook/prefetch_dec
improve long-range decoder speed
Yann Collet [Fri, 9 Nov 2018 01:00:23 +0000 (17:00 -0800)]
Improves decompression speed when using cold dictionary
by triggering the prefetching decoder path
(which used to be dedicated to long-range offsets only).
Figures on my laptop :
no content prefetch : ~300 MB/s (for reference)
full content prefetch : ~325 MB/s (before this patch)
new prefetch path : ~375 MB/s (after this patch)
The benchmark speed is already significant,
but another side-effect is that this version
prefetch less data into memory,
since it only prefetches what's needed, instead of the full dictionary.
This is supposed to help highly active environments
such as active databases,
that can't be properly measured in benchmark environment (too clean).
Also :
fixed the largeNbDict test program
which was working improperly when setting nbBlocks > nbFiles.
Yann Collet [Thu, 8 Nov 2018 21:43:01 +0000 (13:43 -0800)]
Merge pull request #1409 from terrelln/tsan-dict
Add TSAN dictionary tests
Yann Collet [Thu, 8 Nov 2018 20:57:34 +0000 (12:57 -0800)]
reduced assertion strength
one limit case can apparently be generated during fuzzer tests
Yann Collet [Thu, 8 Nov 2018 20:51:42 +0000 (12:51 -0800)]
added a simple test case for decode-only benchmark mode
as it was broken and no one noticed
Yann Collet [Thu, 8 Nov 2018 20:47:46 +0000 (12:47 -0800)]
improve long-range decoder speed
on enwik9 at level 22 (which is almost a worst case scenario),
speed improves by +7% on my laptop (415 -> 445 MB/s)
Yann Collet [Thu, 8 Nov 2018 20:36:39 +0000 (12:36 -0800)]
fixed decompression-only benchmark
Nick Terrell [Thu, 8 Nov 2018 18:59:01 +0000 (10:59 -0800)]
Run TSAN fuzztest
Nick Terrell [Thu, 8 Nov 2018 18:58:51 +0000 (10:58 -0800)]
Add multithreaded dictbuilder tests to fuzzer.c
Nick Terrell [Thu, 8 Nov 2018 18:45:53 +0000 (10:45 -0800)]
Signal before unlocking in pool.c
Yann Collet [Thu, 8 Nov 2018 18:43:40 +0000 (10:43 -0800)]
Merge pull request #1408 from szreder/dev
Prevent deadlock on malloc() failure.
Bartosz Szreder [Thu, 8 Nov 2018 09:29:31 +0000 (10:29 +0100)]
Prevent deadlock on malloc() failure.
Yann Collet [Tue, 6 Nov 2018 21:58:50 +0000 (13:58 -0800)]
Merge pull request #1405 from pitrou/allow_cmake_debug_builds
Allow creating debug builds with CMake
Antoine Pitrou [Tue, 6 Nov 2018 20:24:29 +0000 (21:24 +0100)]
Allow creating debug builds with CMake
Only set the build type to Release if the user didn't select any.
See discussion in https://github.com/facebook/zstd/pull/1278#issuecomment-
436327923
and below.
Yann Collet [Tue, 6 Nov 2018 19:53:20 +0000 (11:53 -0800)]
Merge pull request #1404 from facebook/T36302429
fixed T36302471
Yann Collet [Tue, 6 Nov 2018 01:57:30 +0000 (17:57 -0800)]
Merge pull request #1403 from facebook/assert_opt
just add an assert() in ZSTD_insertBtAndGetAllMatches()
Yann Collet [Tue, 6 Nov 2018 01:50:30 +0000 (17:50 -0800)]
fixed T36302429
Yann Collet [Tue, 30 Oct 2018 01:07:07 +0000 (18:07 -0700)]
just add an assert() in ZSTD_insertBtAndGetAllMatches()
to express a condition on ll0 .
May help static analyzer as in #1397
Yann Collet [Mon, 29 Oct 2018 23:28:09 +0000 (16:28 -0700)]
Merge pull request #1395 from facebook/decompressblock
created zstd_decompress_block module
Yann Collet [Mon, 29 Oct 2018 23:00:26 +0000 (16:00 -0700)]
Merge pull request #1398 from bmwiedemann/pgo
Fix building zstd-pgo
Yann Collet [Mon, 29 Oct 2018 22:03:57 +0000 (15:03 -0700)]
fixed a second memset() on NULL
not sure why it only triggers now,
this code has been around for a while.
Introduced a new error code : dstBuffer_null,
I couldn't express anything even remotely similar with existing error codes set.
Yann Collet [Mon, 29 Oct 2018 20:57:37 +0000 (13:57 -0700)]
fixed memcpy() on NULL warning
memcpy(NULL, src, 0) is undefined behavior.
Yann Collet [Mon, 29 Oct 2018 20:06:45 +0000 (13:06 -0700)]
Merge pull request #1396 from facebook/huf_refactor
refactor HUF_compress_internal for clarity
Bernhard M. Wiedemann [Mon, 29 Oct 2018 16:20:03 +0000 (17:20 +0100)]
pgo: also optimize compression
and clean *.o so that linking does not fail from code
that is partially augmented with coverage generation instructions
Bernhard M. Wiedemann [Mon, 29 Oct 2018 14:11:28 +0000 (15:11 +0100)]
zstd-pgo: do not make clean and zstd in parallel
otherwise, there might not be a ./zstd to run
Yann Collet [Fri, 26 Oct 2018 23:38:23 +0000 (16:38 -0700)]
added flag -Wc++-compat
for library and cli
Yann Collet [Fri, 26 Oct 2018 22:57:13 +0000 (15:57 -0700)]
renamed NEWS as CHANGELOG
to conform fb publication rules.
Yann Collet [Fri, 26 Oct 2018 22:51:51 +0000 (15:51 -0700)]
minor clarifications of history update rules
Yann Collet [Fri, 26 Oct 2018 22:29:07 +0000 (15:29 -0700)]
Merge pull request #1392 from coetry/dev
provide consistent spacing to enum field
Yann Collet [Fri, 26 Oct 2018 22:28:44 +0000 (15:28 -0700)]
Merge pull request #1394 from terrelln/circleci-packaging
Publish uncompressed, zstd, and gzipped packages
Yann Collet [Fri, 26 Oct 2018 22:25:01 +0000 (15:25 -0700)]
Merge branch 'dev' into huf_Refactor
Yann Collet [Fri, 26 Oct 2018 22:21:52 +0000 (15:21 -0700)]
added a few comments for clarifications
Yann Collet [Fri, 26 Oct 2018 22:00:00 +0000 (15:00 -0700)]
Merge branch 'dev' into decompressblock
Yann Collet [Fri, 26 Oct 2018 21:43:16 +0000 (14:43 -0700)]
Merge pull request #1390 from facebook/nullAsOutput
support decompressing an empty frame into NULL
Allen Hai [Fri, 26 Oct 2018 21:16:06 +0000 (16:16 -0500)]
vertically align code comment
Yann Collet [Fri, 26 Oct 2018 20:21:37 +0000 (13:21 -0700)]
refactor HUF_compress_internal for clarity
changed workspace parameter convention
to always provide workspaceSize,
so that size can be explicitly checked.
Also, use more enum to make the meaning of some parameters more explicit.
Nick Terrell [Fri, 26 Oct 2018 20:06:53 +0000 (13:06 -0700)]
Publish uncompressed, zstd, and gzipped packages
Yann Collet [Fri, 26 Oct 2018 18:25:40 +0000 (11:25 -0700)]
fixed VS2010 project
Yann Collet [Fri, 26 Oct 2018 17:39:49 +0000 (10:39 -0700)]
fixed : extraneous function prototype in fullbench.c
now provided through zstd_internal.h
Allen Hai [Thu, 25 Oct 2018 23:45:20 +0000 (18:45 -0500)]
provide consistent spacing to enum field
Yann Collet [Thu, 25 Oct 2018 23:28:41 +0000 (16:28 -0700)]
created zstd_decompress_block module
isolate all logic associated with block decompression
into its own module.
zstd_decompress is still in charge
of context creation/destruction,
frames, headers, streaming, special blocks, etc.
Compressed blocks themselves are now handled within zstd_decompress_block .
Yann Collet [Thu, 25 Oct 2018 17:20:40 +0000 (10:20 -0700)]
Merge pull request #1388 from facebook/ddict
separate DDict logic into its own module
Yann Collet [Wed, 24 Oct 2018 23:50:03 +0000 (16:50 -0700)]
added comment on public ddict functions
Yann Collet [Wed, 24 Oct 2018 23:34:35 +0000 (16:34 -0700)]
support decompressing an empty frame into NULL
fix #1385
decompressing into NULL was an automatic error.
It is now allowed, as long as the content of the frame is empty.
Seems to simplify things for `arrow`.
Maybe some other projects rely on this behavior ?
Yann Collet [Wed, 24 Oct 2018 18:32:09 +0000 (11:32 -0700)]
fix and refactored libzstd_partial_build.sh
make the test intention clearer
(easier to maintain)
also resist the existence of zstd_ddict.o within libzstd.a
Yann Collet [Wed, 24 Oct 2018 17:36:06 +0000 (10:36 -0700)]
fixed warnings in testpools
Yann Collet [Wed, 24 Oct 2018 00:55:23 +0000 (17:55 -0700)]
added simple guard macros
in case of accidental multi-includes
Yann Collet [Wed, 24 Oct 2018 00:44:43 +0000 (17:44 -0700)]
fixed VS2010 build script
Yann Collet [Wed, 24 Oct 2018 00:40:15 +0000 (17:40 -0700)]
restored cmake build tests in Travis
for some reason, these tests were no longer played on Travis.
As a consequence, I discovered some left-over from the util.c change while testing.
Yann Collet [Wed, 24 Oct 2018 00:25:49 +0000 (17:25 -0700)]
separate DDict logic into its own module
created zstd_ddict.c within lib/decompress
Yann Collet [Tue, 23 Oct 2018 21:34:58 +0000 (14:34 -0700)]
Merge pull request #1384 from ulikunitz/dev
Clarify special case of offset history update
Ulrich Kunitz [Tue, 23 Oct 2018 06:43:19 +0000 (08:43 +0200)]
Reverted removal of a trailing space.
My editor removes trailing spaces while saving. Not confusing things I
reverted that change.
Ulrich Kunitz [Tue, 23 Oct 2018 06:36:50 +0000 (08:36 +0200)]
Fixed a typo
I fixed a typo in the last commit. Many thanks to @terrelin for pointing
that out.
Ulrich Kunitz [Mon, 22 Oct 2018 21:46:43 +0000 (23:46 +0200)]
Clarify special case of offset history update
If the current sequence has literal length of zero then an offset value
of three is handled in a special manner. While I implemented a golang
decoder I had to consult the educational decoder for clarification on
the update of the offset history in that case. This commit provides the
clarification that the offset value Repeated_Offset1-1 is handled as a
new offset is added to the offset history accordingly.
Yann Collet [Mon, 22 Oct 2018 21:04:15 +0000 (14:04 -0700)]
Merge pull request #1383 from facebook/uninstall
fix : removed zstdmt symlink on uninstall
Yann Collet [Mon, 22 Oct 2018 19:51:56 +0000 (12:51 -0700)]
fix : removed zstdmt symlink on uninstall
Yann Collet [Mon, 22 Oct 2018 18:57:12 +0000 (11:57 -0700)]
Merge pull request #1380 from facebook/decodecorpus
fix decodecorpus incorrect frame generation
Nick Terrell [Mon, 22 Oct 2018 17:48:56 +0000 (10:48 -0700)]
Merge pull request #1382 from jonesz/ticket-compiler-rt-fuzzer
Fix libFuzzer location in makefile.
Ethan Jones [Mon, 22 Oct 2018 16:19:13 +0000 (11:19 -0500)]
Fix libFuzzer location in makefile.
libFuzzer was moved into compiler-rt, update the repo location
accordingly.
Yann Collet [Sun, 21 Oct 2018 01:53:02 +0000 (18:53 -0700)]
fix decodecorpus incorrect frame generation
fix #1379
decodecorpus was generating one extraneous byte when `nbSeq==0`.
This is disallowed by the specification.
The reference decoder was just skipping the extraneous byte.
It is now stricter, and flag such situation as an error.
Yann Collet [Fri, 19 Oct 2018 23:31:10 +0000 (16:31 -0700)]
Merge pull request #1368 from rkjain89/test-branch
Moving Code To util.c
Rohit Jain [Wed, 17 Oct 2018 06:33:17 +0000 (23:33 -0700)]
Made the changes in zstd vcproj file for VS2008 as requested
Yann Collet [Wed, 17 Oct 2018 03:58:52 +0000 (20:58 -0700)]
minor NEWS update for v1.3.7
Yann Collet [Tue, 16 Oct 2018 17:09:33 +0000 (10:09 -0700)]
Merge pull request #1372 from samrussell/dev
Add man pages for zstdgrep and zstdless
Sam Russell [Mon, 15 Oct 2018 19:42:02 +0000 (08:42 +1300)]
fix up zgrep/zless references
Sam Russell [Mon, 15 Oct 2018 09:20:10 +0000 (22:20 +1300)]
Add man pages for zstdgrep and zstdless
Rohit Jain [Sat, 13 Oct 2018 02:06:58 +0000 (19:06 -0700)]
Changing tests/fuzz/Makefile to move util.o to FUZZ_SRC instead
Rohit Jain [Fri, 12 Oct 2018 21:22:05 +0000 (14:22 -0700)]
Adding util.c to zstd VS file
Rohit Jain [Fri, 12 Oct 2018 21:08:13 +0000 (14:08 -0700)]
Adding chagnes for cmake and VStudio