]> git.ipfire.org Git - thirdparty/zstd.git/commit
[lib] Allow compression dictionaries with missing symbols 2202/head
authorNick Terrell <terrelln@fb.com>
Sat, 13 Jun 2020 00:48:39 +0000 (17:48 -0700)
committerNick Terrell <terrelln@fb.com>
Sat, 13 Jun 2020 00:57:19 +0000 (17:57 -0700)
commit08981d26384617d2b0233978e7d44f8f95ec6eb1
tree7a195c73f2a247f4553cb7f2593adfe35da8fa50
parente8127b3f159e9321d2cebbd65a098c32e6916b19
[lib] Allow compression dictionaries with missing symbols

Allow compression to use dictionaries with missing symbols in their
entropy tables. We set the FSE repeat mode to check when there are
missing symbols, and set the FSE repeat mode to valid when all symbols
are present.

Note that when not all symbols are present, the heuristics which favor
dictionary tables for lower compression levels won't activate.

Tested by manually creating a dictionary with missing symbols of every
type, and validing that the compressor rejects it before this change,
and accepts it after this change. Also, I ran the `dictionary_loader`
fuzzer for >1 hour of CPU time without running into cases where
compression succeeds, but decompression fails.

Fixes #2174.
lib/compress/zstd_compress.c
lib/compress/zstd_compress_internal.h
lib/dictBuilder/zdict.c
tests/golden-compression/http [new file with mode: 0644]
tests/golden-dictionaries/http-dict-missing-symbols [new file with mode: 0644]
tests/playTests.sh