IMPORT: slz/uslz: decode all the members of a multi-member gzip stream
A gzip file is a series of members (rfc1952), which is what "gzip -c a b",
"cat a.gz b.gz" and most log rotators produce. uslz stopped after the first
one and returned USLZ_DECODE_SUCCESS, so the caller silently got truncated
content with no way to notice:
cat a.gz b.gz | gzip -dc | wc -c -> 151636
cat a.gz b.gz | ./zdec | wc -c -> 76799 and rc=0
This is not commonly used with our targetted use cases but can sometimes
be seen in incremental backups for example where extra inputs will be
ignored.
The required change is not that big but is not obvious:
- First, the gzip trailer is 8 bytes, crc32 followed by isize, and only
the crc32 was consumed. The four isize bytes were left in the stream,
so nothing could have recognised the next member's magic behind them.
Both halves are now accumulated before anything is compared, which
also makes the trailer read resumable without having to remember how
far into it we got; isize is consumed but not verified.
- Second, a new function, uslz_next_member() detects a following member
and resets the per-member state (checksum, flags, state machine, bit
accumulator) while preserving the output ring, the total decoded size
and the drain offset, so that the members' contents are simply
concatenated. It is called both right after a member completes, so
that a single call decodes as many members as its input holds, and at
the start of a call, so that a member boundary falling between two
calls works too.
The two magic bytes may be split across calls, and the header buffer
cannot hold them in the meantime because it shares storage with the
distance table which the member just decoded has overwritten. The bit
accumulator isn't used at that point and survives across calls, so it
is reused here to store the previous bytes and the confirmed magic is
then passed to the format detection, which already knows how to
accumulate the rest of a header across calls.
- Third, when a member ends exactly at the end of the input, there is no
way to tell whether another one follows without more data. Success is
reported, which is what a caller with nothing left to send needs, and
the check is retried on the next call for a caller which has more.
This is now stated in uslz_decode()'s documentation: success means
complete as far as the data provided goes, and a caller with input
left must call again anyway. Anything after the last member which is
not a gzip magic is ignored as trailing garbage, as gzip(1) does.
With all this done, concatenating two silesia archives and passing them
to zdec properly now reports twice the uncompressed size.
This was the last failure of tests/uslztest.sh which now shows 1514/1514.
This is libslz upstream commit
76b983eccb8220fdd3083bd6d826a5ea07f45af3