IMPORT: slz/uslz: widen crc_flush, it overflowed on large stored blocks
crc_flush counts the bytes decoded but not checksummed yet. It was
declared as a (signed) short, while a single stored (uncompressed) block
copy can add up to a whole output buffer in one call, making it overflow:
crc_flush=-32768 after +=32768 (index=32768)
That happens with any incompressible payload, which the encoders emit as
stored blocks, as soon as the input is fed in chunks large enough for the
copy to advance by more than 32767 bytes at once. The subsequent call to
uslz_update_crc() then gets a negative length. The checksum functions
happen not to touch memory in that case (their loops simply do not run),
so the visible symptom is a wrong checksum and a valid stream rejected
with USLZ_DECODE_E_BAD_CRC -- but the invariant crc_flush <= index is
broken from then on, and (out_base + index - crc_flush) can point before
the output buffer, which would read out of bounds.
Here we perform two changes:
- crc_flush becomes an int. It has to be larger than the output
buffer, not smaller.
- the CRC_BLOCK batch flush becomes a while loop instead of an if.
It only ever flushed one CRC_BLOCK per emitted chunk, so a large
copy left the rest pending and defeated the point of batching the
checksum in blocks small enough to stay in L1. With the loop,
crc_flush is back below CRC_BLOCK after every emission, which also
bounds it tightly.
It's easy to reproduce with a 40kB incompressible file (e.g. urandom):
gzip -9 -c < rand40k > r.gz9
./uslztest 32768
1000000 < r.gz9 # was: error (5), now: crc=
195cf5ee
tests/uslztest.sh goes from 1488/1514 to 1510/1514. The four remaining
failures are for future patches.
This is libslz upstream commit
96b80bb978e676db75a631bf63aaf5d32c22f015