Yann Collet [Fri, 21 Jan 2022 05:24:33 +0000 (21:24 -0800)]
improved compression of literals in specific corner cases
In rare cases, the default huffman depth selector is a bit too harsh,
requiring brutal adaptations to the tree,
resulting is some loss of compression ratio.
This new heuristic avoids the worse cases, favoring compression ratio.
As an example, compression of a specific distribution of 771 literals
is now improved to 441 bytes, from 601 bytes before.
Yonatan Komornik [Mon, 24 Jan 2022 22:43:02 +0000 (14:43 -0800)]
AsyncIO compression part 1 - refactor of existing asyncio code (#3021)
* Refactored fileio.c:
- Extracted asyncio code to fileio_asyncio.c/.h
- Moved type definitions to fileio_types.h
- Moved common macro definitions needed by both fileio.c and fileio_asyncio.c to fileio_common.h
Yonatan Komornik [Fri, 21 Jan 2022 21:55:41 +0000 (13:55 -0800)]
Async write for decompression (#2975)
* Async IO decompression:
- Added --[no-]asyncio flag for CLI decompression.
- Replaced dstBuffer in decompression with a pool of write jobs.
- Added an ability to execute write jobs in a separate thread.
- Added an ability to wait (join) on all jobs in a thread pool (queued and running).
W. Felix Handte [Thu, 20 Jan 2022 22:36:28 +0000 (17:36 -0500)]
Trigger Release Artifact Generation on Publish
We previously triggered release artifact generation on release creation. We
sometimes observed that the action failed to run. I hypothesized that we were
hitting rate limiting or something. I just stumbled across [this documentat-
ion](https://docs.github.com/en/actions/using-workflows/events-that-trigger-workflows#release), which says:
> Note: Workflows are not triggered for the `created`, `edited`, or `deleted`
> activity types for draft releases. When you create your release through the
> GitHub browser UI, your release may automatically be saved as a draft.
This must have been what was happening. This commit therefore changes the
trigger to the `published` activity. This should be more reliable.
This does have the unfortunate side effect that artifacts won't be generated
or attached until *after* the release has been published, which is what I was
trying to avoid by using the `created` activity. Oh well.
Carl Woffenden [Wed, 19 Jan 2022 15:56:03 +0000 (16:56 +0100)]
Fixed bugs found in other projects
When testing amalgamating other projects it was found: invalid Unicode errors were tripping Python's text IO, and the header search order appears differs from the shell version.
requires that on Linux, all linker input files are marked as CET enabled
in .note.gnu.property section. For high-level language source codes,
.note.gnu.property section is added by compiler with the -fcf-protection
option. For assembly sources, include <cet.h> to add .note.gnu.property
section.
H.J. Lu [Tue, 11 Jan 2022 16:09:58 +0000 (08:09 -0800)]
x86-64: Hide internal assembly functions
Hide x86-64 internal assembly functions. Before
$ nm -D lib/libzstd.so.1 | grep usingDTable_internal_bmi2_asm_loop 00000000000c23c0 T _HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop 00000000000c23c0 T HUF_decompress4X1_usingDTable_internal_bmi2_asm_loop 00000000000c283d T _HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop 00000000000c283d T HUF_decompress4X2_usingDTable_internal_bmi2_asm_loop
$
Nick Terrell [Fri, 7 Jan 2022 23:09:56 +0000 (15:09 -0800)]
Fix stderr progress logging for decompression
When decompressing with `-q` and an output file, the progress bar was mistakenly printed. This is a minimal fix, with a larger refactor to be stacked on top of it.
Nick Terrell [Fri, 7 Jan 2022 00:00:02 +0000 (16:00 -0800)]
[opt] Fix oss-fuzz bug in optimal parser
oss-fuzz uncovered a scenario where we're evaluating the cost of litLength = 131072,
which can't be represented in the zstd format, so we accessed 1 beyond LL_bits.
Fix the issue by making it cost 1 bit more than litLength = 131071.
There are still follow ups:
1. This happened because literals_cost[0] = 0, so the optimal parser chose 36 literals
over a match. Should we bound literals_cost[literal] > 0, unless the block truly only
has one literal value?
2. When no matches are found, the cost model isn't updated. In this case no matches were
found for an entire block. So the literals cost model wasn't updated at all. That made
the optimal parser think literals_cost[0] = 0, where it is actually quite high, since
the block was entirely random noise.
W. Felix Handte [Thu, 6 Jan 2022 05:20:49 +0000 (00:20 -0500)]
Avoid xxHash Dependency by Inlining
xxHash symbols are present in `libzstd.so`, but they are local and therefore
unavailable outside the lib. There are two possible solutions to the problem.
We could make those symbols global, or we could remove the dependency.
This commit chooses the latter approach. I suppose this comes at the cost of
code size / build time. I'm open to comments on whether this is a good thing
to do, especially since this will apply even when we are statically linking
everything.
W. Felix Handte [Thu, 23 Dec 2021 19:57:16 +0000 (14:57 -0500)]
Improve Module Map File
This commit makes several changes:
1. It adds modules for the dictionary builder and errors headers.
2. It captures all of the macros that are used to configure these headers.
When the headers are imported as modules and one of these macros is defined
the compiler issues a warning that it needs to be defined on the CLI.
3. It promotes the modulemap file into the root of the lib directory.
Experimentation shows that clang's `-fimplicit-module-maps` will find the
modulemap when placed here, but not when it's put in a subdirectory.
Nick Terrell [Wed, 5 Jan 2022 00:05:59 +0000 (16:05 -0800)]
[meson] Explicitly disable assembly for non clang/gcc copmilers
After merging #2951 I realized that we will want to explicitly disable
assembly when we aren't including the assembly source file. Otherwise,
if some non clang/gcc compiler does support assembly, it would fail to
build.
Yann Collet [Fri, 31 Dec 2021 21:03:12 +0000 (13:03 -0800)]
fix performance issue in scenario #2966 (part 1)
When re-using a compression state, across multiple successive compressions,
the state should minimize the amount of allocation and initialization required.
This mostly matters in situations where initialization is an overwhelming task
compared to compression itself.
This can happen when the amount to compress is small,
while the compression state was given the impression that it would be much larger,
aka, streaming mode without providing a srcSize hint.
This commit fixes it, making this scenario once again on par with v1.4.9.
Note that this does not completely fix #2966,
since another heavy initialization, specific to row mode,
is also happening (and was not present in v1.4.9).
This will be fixed in a separate commit.
W. Felix Handte [Thu, 30 Dec 2021 01:47:12 +0000 (17:47 -0800)]
Mark Huffman Decoder Assembly `noexecstack` on All Architectures
Apparently, even when the assembly file is empty (because
`ZSTD_ENABLE_ASM_X86_64_BMI2` is false), it still is marked as possibly
needing an executable stack and so the whole library is marked as such. This
commit applies a simple patch for this problem by moving the noexecstack
indication outside the macro guard.
projects / thirdparty / zstd.git / log
