Adam Stylinski [Fri, 3 Jul 2026 14:07:21 +0000 (10:07 -0400)]
Use the same unroll factor for the copying variant
We have to have narrower SIMD vectors for all the reasons why memcpy
is slower with 512 bit vectors. Memory bandwidth bound things in both
directions are generally a net loss with 512 bit wide operations for
AVX512. If it's one way, such as in the plain checksum, prefetching can
mostly save you from any penalty felt by downclocking or stalls due to
memory fetch latency.
Adam Stylinski [Fri, 3 Jul 2026 13:58:36 +0000 (09:58 -0400)]
Unroll by 4x instead of 2x on avx512vnni.
As it turns out, we weren't limited on execution ports here and to
hide the latency of these we needed to go wider with more separate
parallel accumulations.
This is based on an exploration in the avxvnni version.
Skip redundant hash slides for non-chain-walking strategies
deflate_quick reads only the head of each hash chain and never walks
prev, while Z_HUFFMAN_ONLY and Z_RLE read neither. Sliding the whole
hash on every window move is wasted work for them. Add a slide_hash_head
functable entry that slides only the head table, so deflate_quick slides
just the head and the huffman and rle strategies skip the slide entirely.
About 8% faster on the level-1 no-CRC deflate benchmark, output unchanged.
deflate_quick only looks at the head of each hash chain and never walks
prev, so writing the prev links is wasted work. Add head-only insert
helpers and use them in deflate_quick and the level-1 fill_window
catch-up inserts.
The leftover stale prev is safe if the level is later raised mid-stream,
since longest_match never records a match with cur_match >= strstart and
verifies every match by comparison.
Adam Stylinski [Sat, 4 Oct 2025 20:01:23 +0000 (16:01 -0400)]
Add AVXVNNI support
Intel, hopping on the ML all the things band wagon, decided that VNNI
was just _too_ good to not include on their lesser client CPUs without
AVX512. Given how many cycles it saves for the 32 bit widening
operation and how close we already were to something that supports this
sans AVX512, we may as well include support for it. To actually get
maximal ILP for this, this loop had to be unrolled a considerable
amount, but we're finally faster on both P cores and E cores for this
implementation. The gains are minor but measurable.
The copying variant is only faster on E-cores, so it's not being
compiled or used. This may need further testing on MSVC, they have no
definitive "arch" flag for it as far as I can tell.
deflateParams can leave the chains stale for the new strategy several ways:
the previous strategy didn't maintain them (stored, huffman, or rle), or
crossing the level-9 boundary switched the hash function from integer to
rolling. In each case the new strategy silently dropped match history until
the window refilled.
Replace the stored-only slide/clear fixup with a single staleness check that
rebuilds the chains from the current window with the new hash whenever the new
strategy reads it. The rebuild reconstructs the chains unconditionally, so
deflate_stored no longer needs to defer and count hash slides; drop that
counter.
CHUNK_MEMSET_RVV_IMPL read the broadcast value through a plain pointer
cast, but "from" can be at any byte offset in the output window, so the
read may be misaligned. Read the value with zng_memread_* like every
other chunkset implementation. Targets with fast unaligned loads
generate identical code; strict-align targets now get byte-wise loads
instead of a wide load that may trap.
Fixes #1670
Assisted-By: Claude Fable 5 <noreply@anthropic.com>
Replace the quick_ prefixed and unmarked insert_string helpers with
names based on the hash method used, insert_knuth and insert_roll,
marking deviations from the common single-position insert: _val for
a caller-supplied hash value and _batch for inserting a run of
positions. Rename UPDATE_HASH_INT to UPDATE_HASH_KNUTH to match.
This covers the callback type, dispatch pointers, and the
insert_string benchmark harness.
Add --benchmark_data_types flag to select data type variants
Register the data-type benchmark variants at runtime instead of
statically, gated by a new --benchmark_data_types flag taking a
comma-separated list of type names or "all". Only text is registered
by default, so the default benchmark count stays at the pre-existing
suite size. Non-text types use reduced size/level ladders (inflate:
16K/128K/1M, deflate: levels 3/6/9 at 128K/1M).
Literal runs mixed with medium self-referential matches, modeled on the
stream shape of compiled binaries: ~30% literal bytes with near-flat
codes, ~85% literal symbols, ratio ~2.3. Covers the region between
literals (no matches) and the match-dominated types.
Add literals data type to benchmark data generators
High-entropy bytes (~6.5 bits/byte) that deflate encodes as Huffman-coded
blocks with almost no matches, filling the gap between text (skewed codes,
root-table hits) and random (stored blocks, decode loop bypassed).
Add a 4-symbol (DNA base) alphabet generator. With only four symbols
nearly every 3-byte prefix collides, producing very long hash chains
while matches stay short, which stresses the longest_match short-match
path that the existing generators do not isolate. Register it for the
deflate and inflate benchmark variants.
Assisted-By: Claude Opus 4.8 <noreply@anthropic.com>
Cross-vary inflate and deflate benchmarks by input data type
inflate_bench and deflate_bench now register a separate benchmark per
(variant, data_type) pair so the data type appears in the display name.
inflate yields five benchmarks per size:
Both fixtures share a Dispatch() pattern with an empty SetUp; ->Name()
controls the "/"-separated display. Inflate sizes its compressed buffer
via deflateBound to handle incompressible inputs.
All data generators move to test/test_data_p.h (renamed from
compressible_data_p.h) behind a gen_test_data(test_data_type, bufsize)
dispatcher.
Dougall Johnson [Thu, 2 Jul 2026 10:08:56 +0000 (20:08 +1000)]
Read bits with zero refill latency
Look up the next length code before refilling the bit buffer, so the
refill's load latency overlaps with the table load instead of feeding
it. This is a small but confusing change discussed here:
https://dougallj.wordpress.com/2022/08/26/reading-bits-with-zero-refill-latency/
The loop-entry invariant becomes: hold contains at least the
MAX_LEN_ROOT_BITS (10) bits needed for the first-level length table
lookup. The refill before the loop establishes it, the literal-only
paths consume at most 35 of the 56+ bits from the in-loop refill, and
the distance path's conditional refill threshold is raised by
MAX_LEN_ROOT_BITS so that consuming the distance code and extra bits
(up to MAX_BITS + MAX_DIST_EXTRA_BITS) still leaves enough. The new
MAX_LEN_ROOT_BITS constant replaces the hardcoded 10 at the
state->lenbits assignments in inflate.c and infback.c so the invariant
cannot silently drift.
INFLATE_FAST_MIN_HAVE stays 15: there are still at most two 8-byte
refill reads per iteration, the first at the position validated by the
loop bound (advancing at most 7 bytes), and the pre-loop refill reads
at the entry position (avail_in >= 15) advancing exactly 7 bytes with
bits >= 56 left over, so the first iteration's in-loop refill advances
0 bytes before its read.
Inflate benchmarks on Apple Silicon (clang, Release):
When compare256_neon inlines into longest_match, neither clang nor gcc
keeps the two-stream compare loop post-indexed: both emit a base+offset
address plus a separate add per 16-byte step. On clang this is because
loop strength reduction cannot generate post-increment addressing at
all; its isLegalAddressingMode hook has no way to express it, a
long-standing "TODO: handle pre/postinc" [1][2]. Post-indexed loads are
left to the AArch64 load/store optimizer, a post-isel peephole that only
fires for simple loop shapes [3].
The short-match peel restructured the surrounding code enough to move the
loop out of the shape that peephole recognizes, turning a previously
post-indexed clang loop into the extra-add form and regressing the
striped_rgb benchmark. gcc never post-indexed this loop to begin with,
so it carried the extra add on both develop and the PR.
No source-level spelling avoids it; every C formulation funnels through
the same lowering. Pin the post-indexed form with inline asm: one
self-incrementing pointer reaches the other stream at a constant offset,
folding the +16 advance back into the load. The result is byte-identical
and speeds up striped_rgb on both compilers (M5: clang flips it from a
regression to a win, gcc -3% to -6%).
MSVC ARM64 has no inline asm and keeps the plain intrinsic loop.
Reject short-match candidates with a masked test instead of ctz
A candidate can only improve best_len when its first best_len+1 bytes
all match, so the rejection path does not need the exact mismatch
position. Compute a loop-invariant mask of those bytes and reject with a
single AND and zero-test, running ctz only on the rare accepting
candidate. This keeps the multi-cycle ctz off the hot rejection path,
which helps dense-chain inputs (low-alphabet data, slow cores) while
leaving compressed output unchanged.
Peel first candidate out of the short-match chain walk
Unroll the first candidate check and its chain advance ahead of the
loop so its match case is the fall-through into compare256, which the
fast levels hit on nearly every call, and single-candidate chains
(barely-compressible data) run straight-line with no loop-entry
overhead. Longer chains continue into a self-contained rejection loop,
costing one predictable branch per rejected candidate instead of three,
and the compiler lays out each path for its own hot case.
Improves level 9 by 1.5-2% on Apple M5 and trims the mid levels on
barely-compressible data. Compressed output is byte-identical.
Short-circuit longest_match on short candidates via XOR and ctz
When best_len is under 8, the whole candidate match fits in the
first 8 bytes, so a single load XORed against scan_start and run
through ctz gives the exact match length with no COMPARE256 call
and no separate tail sentinel. The best_len-to-offset mapping
collapses to a single subtract since the graduated sizeof(uint32_t)
and sizeof(uint64_t) brackets no longer cover anything distinct.
Benefits every deflate strategy that calls longest_match: level 3
gains 10-12% at 128KB+ inputs and levels 1, 6, and 7 each trim 1-3%.
Compressed output is byte-identical across corpora.
Prior art: zlib-rs first applied the same XOR+ctz technique in
trifectatechfoundation/zlib-rs#284.
Adam Stylinski [Sun, 5 Jul 2026 15:31:23 +0000 (11:31 -0400)]
Fix this compilation issue for non-windows machines
I've been manually fixing this up for a bit. We're on C++ so let's just
use STL's functions here, which should compile down to min and max
instructions.
Add Windows-only command-line flags to reduce benchmark variance.
--benchmark_cpu_affinity pins the process to a taskset-style CPU list,
--benchmark_no_power_throttling opts out of EcoQoS to keep full clock on
performance cores, and --benchmark_priority sets the process priority
class. Document the custom arguments in a table in the README.
dxbjavid [Tue, 23 Jun 2026 13:42:18 +0000 (19:12 +0530)]
cap gzbuffer request at 1 GiB in the api
move the size limit into gzbuffer itself: zng_gzbuffer rejects requests
over 1 GiB with -1 while the compat gzbuffer silently clamps to 1 GiB,
keeping the checks next to the api like the rest of zlib-ng. with want
bounded there the gz_buffer_alloc backstop is no longer needed, so drop
it and note the limit in the zlib-ng.h docs.
The compare256_*_static helpers are textually included into match_tpl.h
to specialize each longest_match variant, and compare256_rle_* serve the
RLE path the same way. At -O3 clang declines to inline the larger fully
unrolled bodies (scalar and NEON), emitting an out-of-line call with
register spills around it on the hot path. Mark them Z_FORCEINLINE so
each inlines into its consumer as intended. GCC already inlines them.
David Korth [Mon, 22 Jun 2026 22:36:55 +0000 (18:36 -0400)]
Extend the MSVC 2015 AVX2/AVX512 workaround to MSVC 2017.
The 32-bit MSVC 2015 debug build randomly crashes without this because
the stack ends up not being 16-byte aligned, and MSVC adds a vmovdqa
instruction.
MSVC 2022 and 2026 align the stack properly before using vmovdqa.
Introduce a dotprod-accelerated Adler-32 using vdotq_u32 for both the
byte sum (s1) and the position-weighted sum (s2). Four independent
accumulator sets break the dependency chains between successive dotprod
instructions, allowing the pipeline to fill without stalling on results
from the prior iteration.
On Apple M3, the dotprod path is roughly 49% faster than the standard
NEON implementation for inputs above 32 bytes.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Refactor Adler32 ARM NEON into a reusable template
Move the core adler32_copy_impl function from adler32_neon.c into
adler32_neon_tpl.h so that it may be included by variant-specific
source files compiled with different instruction set flags.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Use vaddvq_u32 for adler32 NEON horizontal reduction
Replace interleaved pairwise reduction with vaddvq_u32 to break the
dependency chain between s1 and s2 modulo computations. The original
code merged both accumulators through a shared addp, serializing the
subsequent umull/lsr/msub chains. Independent reductions allow them
to execute in parallel.
On AArch64 this maps to the ADDV instruction. A compatibility shim
in neon_intrins.h emulates this on 32-bit ARM using vadd and vpadd.
Add missing return after SkipWithError in benchmarks
SkipWithError marks the run skipped but does not return, so these
fixtures fell through into the wrapper that drives the benchmark loop.
The loop body is skipped once the error is set, so the change is
behavior neutral, but it matches the surrounding skip cases that
already return and Google Benchmark's documented contract that exiting
the scope is the caller's responsibility. It also keeps the skip
correct if a fixture later does work before the loop.
Mika Lindqvist [Wed, 17 Jun 2026 21:23:23 +0000 (00:23 +0300)]
[CI] Add coveralls as trusted formula for Homebrew under MacOS
* Homebrew 6.0.0 requires explicit trust when installing formulae from
third-party taps (repositories)
Cleanup deflate_quick variable reuse and call signatures
Drop the lc local: the four-byte window read is hoisted above the
match-search branch, and the literal-emit takes the low byte of
str_val directly so the short-lookahead arm no longer needs its own
load. Pass strstart into quick_start_block and quick_end_block so
they no longer reach into s for block_start. In the early-finish
arm, pass 1 to quick_start_block since last is known to be 1.
Hoist strstart and lookahead to locals in deflate strategies
Lift s->strstart and s->lookahead into function-local variables in
deflate_quick, deflate_huff, and deflate_rle, synced back to s before
external callouts that observe them (fill_window, longest_match,
FLUSH_BLOCK, returns) and reloaded after callouts that mutate them.
Use broadcast chunk store instead of memset for dist=1 on NEON/AVX2/AVX-512
Benchmarks show broadcasting the byte into a vector register and
falling through to the chunk store loop beats libc memset for the
short lengths common in inflate output (29-58% on AVX-512, 15-26%
on AVX2, 33% on NEON at len=8).
In 49a6bb5d, chunkmemset_1 was replaced with memset under the
assumption that libc memset would be at least as fast. According
to benchmarking, this isn't true in all cases.
When dist >= len the source bytes don't overlap the destination, so no pattern repeat is needed and CHUNKCOPY produces the same output. Extends the existing `dist >= sizeof(chunk_t)` shortcut so archs with chunk_t > 8 also catch dist in [chunk_t/2, chunk_t-1] with small lengths.
Skip CHUNKMEMSET loops when length fits in one chunk
When len <= sizeof(chunk_t) a single storechunk on chunk_load gives the same output as the two while-loops + tail. The over-write of chunk_t - len bytes lies in caller headroom.
Inline CHUNKMEMSET tail with bit-decomposed stores
Replace variable-length memcpy with bitmask-gated fixed-size copies (16 + 8 + 4 + 2 + 1) so the compiler can inline each as a direct store instead of a libc call.
Args span every dispatch arc in CHUNKMEMSET — dist=1 memset, dist>=chunk_t CHUNKCOPY, fast dist=2/4/8/16, GET_CHUNK_MAG, the unrolled loops, and the trailing remainder. Registered against chunkmemset_safe_{c,neon,sse2,ssse3,avx2,avx512,power8,rvv,lsx,lasx} where the symbol is built.
ptrdiff_t is defined in <stddef.h> since C89 and is guaranteed to
exist on any C99+ compiler. zlib-ng requires C99, so the build-time
check and the fallback typedef in zconf headers are unnecessary.
The AVX2 and AVX512 flags enable BMI2 but not BMI1, and TZCNT is a
BMI1 instruction. GCC emits the rep bsf encoding that executes as
TZCNT on BMI hardware regardless, but clang gates on the feature bit
and emits plain BSF, which is slower on AMD. Every CPU with AVX2 also
has BMI1, so the flag only affects code already behind AVX2 runtime
detection.
Add branch prediction hints to deflate strategy hot loops
The deflate strategies were inconsistent about branch-prediction hints
on their per-iteration lookahead checks. deflate_quick marked both the
window-refill path unlikely and the steady-state path likely;
deflate_slow marked only the steady-state path; deflate_fast and
deflate_medium marked neither, leaving the compiler to guess the inner
loop's hot/cold block placement.
Mark the window-refill and end-of-input paths unlikely and the
steady-state path likely across deflate_fast, deflate_slow,
deflate_medium, deflate_rle, and deflate_huff, following deflate_quick,
so the rarely-taken refill block stays off the hot fall-through path
and block placement is consistent across all strategies.
Remove dead unlink forward declaration in minigzip tests
Both files include zbuild.h which now unconditionally defines
_LARGEFILE64_SOURCE, making the !defined(_LARGEFILE64_SOURCE) clause
always false and the whole block unreachable. The upstream-zlib
purpose was forward-declaring unlink on platforms without unistd.h
that aren't Windows, but zlib-ng's supported platforms either have
unistd.h or define _WIN32.
zlib-ng never calls fseeko; gzlib.c was rewritten to use
lseek / lseek64 / _lseeki64 directly. The NO_FSEEKO define is not
referenced by any source file, so the CMake check_function_exists
and matching configure probe are dead code.
Drop dead glibc feature-macro juggling from gzguts.h
_LARGEFILE_SOURCE enables fseeko/ftello, which zlib-ng never calls
(gzlib.c uses lseek/lseek64/_lseeki64 directly). The _FILE_OFFSET_BITS
and _TIME_BITS undefs were defensive against a consumer-provided
-D_FILE_OFFSET_BITS=64 leaking into library internals, but they sat
after zbuild.h had already included <stdio.h> so they never affected
system-header sizing. The zlib.h gzopen->gzopen64 remap they were
guarding against is already blocked by the Z_INTERNAL gate for
library builds.
zbuild.h defines _LARGEFILE64_SOURCE, but the macro only takes effect
if seen before any system header. The tools and fuzz/test files that
included <stdio.h> ahead of zbuild.h processed <features.h> without it,
so _LFS64_LARGEFILE never got set and z_off64_t expanded to an unknown
off64_t once LFS64 detection moved out of the build system.
Reorder so zbuild.h precedes any system header, and drop the redundant
<stdio.h>/<stdlib.h>/<string.h> includes that zbuild.h already pulls in.
Move LFS64 detection from build system to C preprocessor
zbuild.h now defines _LARGEFILE64_SOURCE before any system header, so
glibc exposes off64_t and lseek64 where available. The _LFS64_LARGEFILE
check already in zconf.h handles per-platform gating. Drops the
redundant -D__USE_LARGEFILE64 (internal glibc macro, set automatically)
and the unused __off64_t probe that only wrote to HAVE___OFF64_T.
The configure script's _off64_t probe was log-only and the off64_t=yes
path assumed fseeko exists without testing; both are removed in favor
of the standalone fseeko check.
projects / thirdparty / zlib-ng.git / log
