From: Yann Collet Date: Sat, 10 Feb 2018 19:54:49 +0000 (-0800) Subject: Merge branch 'dev' into flexibleLevel X-Git-Tag: v1.3.4~1^2~60^2 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=9945e60ac429bbdd879f516ccd57009b81b858d1;p=thirdparty%2Fzstd.git Merge branch 'dev' into flexibleLevel --- 9945e60ac429bbdd879f516ccd57009b81b858d1 diff --cc doc/zstd_manual.html index 748568237,1dea249d4..9b743c34b --- a/doc/zstd_manual.html +++ b/doc/zstd_manual.html @@@ -800,11 -800,18 +800,13 @@@ size_t ZSTD_decodingBufferSize_min(unsi /* multi-threading parameters */ /* These parameters are only useful if multi-threading is enabled (ZSTD_MULTITHREAD). * They return an error otherwise. */ - ZSTD_p_nbThreads=400, /* Select how many threads a compression job can spawn (default:1) - * More threads improve speed, but also increase memory usage. - * Can only receive a value > 1 if ZSTD_MULTITHREAD is enabled. - * Special: value 0 means "do not change nbThreads" */ - ZSTD_p_nonBlockingMode, /* Single thread mode is by default "blocking" : - * it finishes its job as much as possible, and only then gives back control to caller. - * In contrast, multi-thread is by default "non-blocking" : - * it takes some input, flush some output if available, and immediately gives back control to caller. - * Compression work is performed in parallel, within worker threads. - * (note : a strong exception to this rule is when first job is called with ZSTD_e_end : it becomes blocking) - * Setting this parameter to 1 will enforce non-blocking mode even when only 1 thread is selected. - * It allows the caller to do other tasks while the worker thread compresses in parallel. */ + ZSTD_p_nbWorkers=400, /* Select how many threads will be spawned to compress in parallel. - * Triggers asynchronous mode, even with nbWorkers = 1. - * Can only be set to a value >= 1 if ZSTD_MULTITHREAD is enabled. - * More threads improve speed, but also increase memory usage. - * Default value is `0`, aka "blocking mode" : no worker is spawned, compression is performed inside Caller's thread */ ++ * When nbWorkers >= 1, triggers asynchronous mode : ++ * ZSTD_compress_generic() consumes some input, flush some output if possible, and immediately gives back control to caller, ++ * while compression work is performed in parallel, within worker threads. ++ * (note : a strong exception to this rule is when first invocation sets ZSTD_e_end : it becomes a blocking call). ++ * More workers improve speed, but also increase memory usage. ++ * Default value is `0`, aka "single-threaded mode" : no worker is spawned, compression is performed inside Caller's thread, all invocations are blocking */ ZSTD_p_jobSize, /* Size of a compression job. This value is only enforced in streaming (non-blocking) mode. * Each compression job is completed in parallel, so indirectly controls the nb of active threads. * 0 means default, which is dynamically determined based on compression parameters. @@@ -854,8 -857,8 +856,10 @@@ 
size_t ZSTD_CCtx_setParameter(ZSTD_CCtx* cctx, ZSTD_cParameter param, unsigned value);
  
  Set one compression parameter, selected by enum ZSTD_cParameter.
++  Setting a parameter is generally only possible during frame initialization (before starting compression),
++  except for a few exceptions which can be updated during compression: compressionLevel, hashLog, chainLog, searchLog, minMatch, targetLength and strategy.
    Note : when `value` is an enum, cast it to unsigned for proper type checking.
--  @result : informational value (typically, the one being set, possibly corrected),
++  @result : informational value (typically, value being set clamped correctly),
              or an error code (which can be tested with ZSTD_isError()).

@@@ -1025,9 -1028,9 +1029,10 @@@ size_t ZSTD_CCtx_refPrefix_advanced(ZST 
size_t ZSTD_CCtx_setParametersUsingCCtxParams(
          ZSTD_CCtx* cctx, const ZSTD_CCtx_params* params);
  
  Apply a set of ZSTD_CCtx_params to the compression context.
--  This must be done before the dictionary is loaded.
--  The pledgedSrcSize is treated as unknown.
-   Multithreading parameters are applied only if nbWorkers >= 1.
 -  Multithreading parameters are applied only if nbThreads > 1.
++  This can be done even after compression is started,
++    if nbWorkers==0, this will have no impact until a new compression is started.
++    if nbWorkers>=1, new parameters will be picked up at next job,
++       with a few restrictions (windowLog, pledgedSrcSize, nbWorkers, jobSize, and overlapLog are not updated).

diff --cc lib/compress/zstd_compress_internal.h index 22213c225,35b153fa3..9da2cbfeb --- a/lib/compress/zstd_compress_internal.h +++ b/lib/compress/zstd_compress_internal.h @@@ -30,9 -30,14 +30,15 @@@ extern "C" /*-************************************* * Constants ***************************************/ -static const U32 g_searchStrength = 8; -#define HASH_READ_SIZE 8 +#define kSearchStrength 8 +#define HASH_READ_SIZE 8 +#define ZSTD_CLEVEL_CUSTOM 999 + #define ZSTD_DUBT_UNSORTED_MARK 1 /* For btlazy2 strategy, index 1 now means "unsorted". + It could be confused for a real successor at index "1", if sorted as larger than its predecessor. + It's not a big deal though : candidate will just be sorted again. + Additionnally, candidate position 1 will be lost. + But candidate 1 cannot hide a large tree of candidates, so it's a minimal loss. + The benefit is that ZSTD_DUBT_UNSORTED_MARK cannot be misdhandled after table re-use with a different strategy */ /*-*************************************