From: Nick Terrell <terrelln@fb.com>
Date: Fri, 13 Oct 2017 20:23:57 +0000 (-0700)
Subject: Don't allow default tables to repeat
X-Git-Tag: v1.3.3^2~53^2
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=refs%2Fpull%2F890%2Fhead;p=thirdparty%2Fzstd.git

Don't allow default tables to repeat

It isn't useful in any case to repeat default tables.
Saves a few bytes on Silesia, since we don't trigger the dictionary
heuristic.

Before: 211988480 => 73651998 bytes
After:  211988480 => 73651721 bytes
---

diff --git a/lib/compress/zstd_compress.c b/lib/compress/zstd_compress.c
index 188b8e717..43e5f5461 100644
--- a/lib/compress/zstd_compress.c
+++ b/lib/compress/zstd_compress.c
@@ -1298,7 +1298,13 @@ symbolEncodingType_e ZSTD_selectEncodingType(
     if ( isDefaultAllowed
       && ((nbSeq < MIN_SEQ_FOR_DYNAMIC_FSE) || (mostFrequent < (nbSeq >> (defaultNormLog-1)))) ) {
         DEBUGLOG(5, "Selected set_basic");
-        *repeatMode = FSE_repeat_valid;
+        /* The format allows default tables to be repeated, but it isn't useful.
+         * When using simple heuristics to select encoding type, we don't want
+         * to confuse these tables with dictionaries. When running more careful
+         * analysis, we don't need to waste time checking both repeating tables
+         * and default tables.
+         */
+        *repeatMode = FSE_repeat_none;
         return set_basic;
     }
     DEBUGLOG(5, "Selected set_compressed");