Look up the next length code before refilling the bit buffer, so the
refill's load latency overlaps with the table load instead of feeding
it. This is a small but confusing change discussed here:
https://dougallj.wordpress.com/2022/08/26/reading-bits-with-zero-refill-latency/
The loop-entry invariant becomes: hold contains at least the
MAX_LEN_ROOT_BITS (10) bits needed for the first-level length table
lookup. The refill before the loop establishes it, the literal-only
paths consume at most 35 of the 56+ bits from the in-loop refill, and
the distance path's conditional refill threshold is raised by
MAX_LEN_ROOT_BITS so that consuming the distance code and extra bits
(up to MAX_BITS + MAX_DIST_EXTRA_BITS) still leaves enough. The new
MAX_LEN_ROOT_BITS constant replaces the hardcoded 10 at the
state->lenbits assignments in inflate.c and infback.c so the invariant
cannot silently drift.
INFLATE_FAST_MIN_HAVE stays 15: there are still at most two 8-byte
refill reads per iteration, the first at the position validated by the
loop bound (advancing at most 7 bytes), and the pre-loop refill reads
at the entry position (avail_in >= 15) advancing exactly 7 bytes with
bits >= 56 left over, so the first iteration's in-loop refill advances
0 bytes before its read.
Inflate benchmarks on Apple Silicon (clang, Release):
benchmark_zlib inflate_nocrc/
1048576: 139 us -> 121 us (-13.1%)
benchmark_zlib small_output_buf/16384: 49 us -> 42 us (-12.9%)
zlib-ng sources, level 9: 919 MB/s -> 1042 MB/s (+13.4%)
dict/words, level 6: 804 MB/s -> 916 MB/s (+14.0%)
static library, level 6: 709 MB/s -> 794 MB/s (+12.0%)
Assisted-By: Claude Fable 5
}
/* build code tables -- note: do not change the lenbits or distbits
- values here (10 and 9) without reading the comments in inftrees.h
- concerning the ENOUGH constants, which depend on those values */
+ values here (MAX_LEN_ROOT_BITS and 9) without reading the comments
+ in inftrees.h concerning the ENOUGH constants, which depend on
+ those values, and the refill comments in inffast_tpl.h, which
+ depend on lenbits never exceeding MAX_LEN_ROOT_BITS */
state->next = state->codes;
state->lencode = (const code *)(state->next);
- state->lenbits = 10;
+ state->lenbits = MAX_LEN_ROOT_BITS;
ret = zng_inflate_table(LENS, state->lens, state->nlen, &(state->next), &(state->lenbits), state->work);
if (ret) {
SET_BAD("invalid literal/lengths set");
lmask = (1U << state->lenbits) - 1;
dmask = (1U << state->distbits) - 1;
+ /* Ensure enough bits for the first length table lookup. Every refill after
+ this one is issued behind a table lookup that only needs bits already in
+ hold, so the refill's load latency overlaps with the table load instead
+ of feeding it. */
+ REFILL();
+
/* decode literals and length/distances until end-of-block or not enough
input data or output space */
do {
- REFILL();
+ /* This lookup needs bits >= MAX_LEN_ROOT_BITS, which every path here
+ guarantees: each refill leaves bits >= 56, the literal-only paths
+ loop after consuming at most 35 bits, and the distance path refills
+ to at least MAX_BITS + MAX_DIST_EXTRA_BITS + MAX_LEN_ROOT_BITS
+ before consuming at most MAX_BITS + MAX_DIST_EXTRA_BITS. */
here = lcode[hold & lmask];
Z_TOUCH(here);
+ /* No-op on the first iteration: the pre-loop refill already filled hold
+ and the lookup above consumes nothing, so bits is still >= 56. Tops
+ hold back up on every later iteration. */
+ REFILL();
old = hold;
DROPBITS(here.bits);
if (LIKELY(here.op == 0)) {
TRACE_LENGTH(len);
here = dcode[hold & dmask];
Z_TOUCH(here);
- if (UNLIKELY(bits < MAX_BITS + MAX_DIST_EXTRA_BITS)) {
+ /* Reserve bits for the distance code and its extra bits, plus the
+ next iteration's length table lookup, which happens before the
+ next refill. */
+ if (UNLIKELY(bits < MAX_BITS + MAX_DIST_EXTRA_BITS + MAX_LEN_ROOT_BITS)) {
REFILL();
}
dodist:
}
/* build code tables -- note: do not change the lenbits or distbits
- values here (10 and 9) without reading the comments in inftrees.h
- concerning the ENOUGH constants, which depend on those values */
+ values here (MAX_LEN_ROOT_BITS and 9) without reading the comments
+ in inftrees.h concerning the ENOUGH constants, which depend on
+ those values, and the refill comments in inffast_tpl.h, which
+ depend on lenbits never exceeding MAX_LEN_ROOT_BITS */
state->next = state->codes;
state->lencode = (const code *)(state->next);
- state->lenbits = 10;
+ state->lenbits = MAX_LEN_ROOT_BITS;
ret = zng_inflate_table(LENS, state->lens, state->nlen, &(state->next), &(state->lenbits), state->work);
if (ret) {
SET_BAD("invalid literal/lengths set");
#define TRACE_END_OF_BLOCK() \
Tracevv((stderr, "inflate: end of block\n"))
-#define INFLATE_FAST_MIN_HAVE 15 /* max input bits per length/distance pair */
+/* inflate_fast() executes at most two 8-byte refill reads per iteration: the
+ first at the input position validated by the loop bound (advancing at most
+ 7 bytes), the second therefore ending at most 15 bytes past that position.
+ The refill before the loop reads at the entry position and advances exactly
+ 7 bytes (entry guarantees bits < 8), so the first iteration stays within
+ the same bound. */
+#define INFLATE_FAST_MIN_HAVE 15
#define INFLATE_FAST_MIN_LEFT 260 /* max output per token (258) + 2 */
#define INFLATE_FAST_MIN_SAFE 3 /* max unchecked literal writes per iteration */
/* all codes must not exceed MAX_BITS bits */
#define MAX_DIST_EXTRA_BITS 13
/* maximum number of extra distance bits */
+#define MAX_LEN_ROOT_BITS 10
+/* maximum number of index bits for the first-level length code table
+ (the largest value assigned to state->lenbits) */
#if MAX_MEM_LEVEL >= 8
# define DEF_MEM_LEVEL 8
projects / thirdparty / zlib-ng.git / commitdiff
