Eli Schwartz [Mon, 31 Jan 2022 03:38:04 +0000 (22:38 -0500)]
travis CI: update meson image to one with a python that isn't EOL
Currently the build errors out with:
```
ERROR: This script does not work on Python 3.6 The minimum supported Python version is 3.7. Please use https://bootstrap.pypa.io/pip/3.6/get-pip.py instead.
```
While in theory this advice could be followed to get a better pip on
xenial, Meson has now deprecated python 3.6 support too, and the next
(unreleased) version requires python 3.7
There are a couple solutions to this:
- hold the version of pip, allow pip to only install 3.6-compatible
versions of meson (effectively freezing meson going forward)
- install python 3.7 on xenial
- update to a 2-year-old image instead of a 4-year-old one
Eli Schwartz [Mon, 31 Jan 2022 02:45:19 +0000 (21:45 -0500)]
meson: valgrind wrapper should return correct errors
While trying to raise an exception on failures, it instead raised an
exception for misusing the exception. CalledProcessError is only
supposed to be used when given a return code and a command, and it
prints:
Command '{cmd}' returned non-zero exit status {ret}
Passing an error message string instead, just errored out with:
Instead use the subprocess module's base error which does accept string
messages. Everything that used to error out, still errors out, but now
they do so with a slightly prettier console message.
Yann Collet [Thu, 27 Jan 2022 13:13:45 +0000 (05:13 -0800)]
fix issue 44108
credit to oss-fuzz
In rare circumstances, the block-splitter might cut a block at the exact beginning of a repcode.
In which case, since litlength=0, if the repcode expected 1+ literals in front, its signification changes.
This scenario is controlled in ZSTD_seqStore_resolveOffCodes(),
and the repcode is transformed into a raw offset when its new meaning is incorrect.
In more complex scenarios, the previous block might be emitted as uncompressed after all,
thus modifying the expected repcode history.
In the case discovered by oss-fuzz, the first block is emitted as uncompressed,
so the repcode history remains at default values: 1,4,8.
But since the starting repcode is repcode3, and the literal length is == 0,
its meaning is : = repcode1 - 1.
Since repcode1==1, it results in an offset value of 0, which is invalid.
So that's what the `assert()` was verifying : the result of the repcode translation should be a valid offset.
But actually, it doesn't matter, because this result will then be compared to reality,
and since it's an invalid offset, it will necessarily be discarded if incorrect,
then the repcode will be replaced by a raw offset.
So the `assert()` is not useful.
Furthermore, it's incorrect, because it assumes this situation cannot happen, but it does, as described in above scenario.
Nick Terrell [Thu, 27 Jan 2022 01:55:26 +0000 (17:55 -0800)]
Print zlib/lz4/lzma library versions in verbose version output
Knowing the version of zlib/lz4/lzma we're linking against is very
useful for debugging issues with those libraries, so print it out in the
verbosity 4 version output.
Also print this information at the top of `playTests.sh`.
Yann Collet [Fri, 21 Jan 2022 05:24:33 +0000 (21:24 -0800)]
improved compression of literals in specific corner cases
In rare cases, the default huffman depth selector is a bit too harsh,
requiring brutal adaptations to the tree,
resulting is some loss of compression ratio.
This new heuristic avoids the worse cases, favoring compression ratio.
As an example, compression of a specific distribution of 771 literals
is now improved to 441 bytes, from 601 bytes before.
brailovich [Tue, 25 Jan 2022 01:42:21 +0000 (17:42 -0800)]
fix for error message in recursive mode for an empty folder
-r on empty directory resulted in zstd waiting input from stdin. now zstd exits without error and prints a warning message explaining why no processing happened (no files or directories to process).
Yonatan Komornik [Mon, 24 Jan 2022 22:43:02 +0000 (14:43 -0800)]
AsyncIO compression part 1 - refactor of existing asyncio code (#3021)
* Refactored fileio.c:
- Extracted asyncio code to fileio_asyncio.c/.h
- Moved type definitions to fileio_types.h
- Moved common macro definitions needed by both fileio.c and fileio_asyncio.c to fileio_common.h
Yonatan Komornik [Fri, 21 Jan 2022 21:55:41 +0000 (13:55 -0800)]
Async write for decompression (#2975)
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
W. Felix Handte [Thu, 20 Jan 2022 22:36:28 +0000 (17:36 -0500)]
Trigger Release Artifact Generation on Publish
We previously triggered release artifact generation on release creation. We
sometimes observed that the action failed to run. I hypothesized that we were
hitting rate limiting or something. I just stumbled across [this documentat-
ion](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release), which says:
> Note: Workflows are not triggered for the `created`, `edited`, or `deleted`
> activity types for draft releases. When you create your release through the
> GitHub browser UI, your release may automatically be saved as a draft.
This must have been what was happening. This commit therefore changes the
trigger to the `published` activity. This should be more reliable.
This does have the unfortunate side effect that artifacts won't be generated
or attached until *after* the release has been published, which is what I was
trying to avoid by using the `created` activity. Oh well.
Carl Woffenden [Wed, 19 Jan 2022 15:56:03 +0000 (16:56 +0100)]
Fixed bugs found in other projects
When testing amalgamating other projects it was found: invalid Unicode errors were tripping Python's text IO, and the header search order appears differs from the shell version.
requires that on Linux, all linker input files are marked as CET enabled
in .note.gnu.property section. For high-level language source codes,
.note.gnu.property section is added by compiler with the -fcf-protection
option. For assembly sources, include <cet.h> to add .note.gnu.property
section.
H.J. Lu [Tue, 11 Jan 2022 16:09:58 +0000 (08:09 -0800)]
x86-64: Hide internal assembly functions
Hide x86-64 internal assembly functions. Before
$ nm -D lib/libzstd.so.1 | grep usingDTable_internal_bmi2_asm_loop 00000000000c23c0 T _HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop 00000000000c23c0 T HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop 00000000000c283d T _HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop 00000000000c283d T HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop
$
Nick Terrell [Fri, 7 Jan 2022 23:09:56 +0000 (15:09 -0800)]
Fix stderr progress logging for decompression
When decompressing with `-q` and an output file, the progress bar was mistakenly printed. This is a minimal fix, with a larger refactor to be stacked on top of it.
projects / thirdparty / zstd.git / log
