From: Yann Collet Date: Fri, 8 May 2020 18:59:45 +0000 (-0700) Subject: Merge branch 'dev' into clevel_doc X-Git-Tag: v1.4.5^2~27^2~1 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=4f2cb944a8cd97d7402e65d8acb468270c524f37;p=thirdparty%2Fzstd.git Merge branch 'dev' into clevel_doc --- 4f2cb944a8cd97d7402e65d8acb468270c524f37 diff --cc programs/README.md index a097cdac7,7ae7822d4..2f212f68a --- a/programs/README.md +++ b/programs/README.md @@@ -141,50 -141,58 +141,61 @@@ Advanced arguments -v : verbose mode; specify multiple times to increase verbosity -q : suppress warnings; specify twice to suppress errors too -c : force write to standard output, even if it is the console - -l : print information about zstd compressed files ---exclude-compressed: only compress files that are not previously compressed + -l : print information about zstd compressed files ++--exclude-compressed: only compress files that are not previously compressed --ultra : enable levels beyond 19, up to 22 (requires more memory) - --long : enable long distance matching (requires more memory) + --long[=#]: enable long distance matching with given window log (default: 27) + --fast[=#]: switch to very fast compression levels (default: 1) ---adapt : dynamically adapt compression level to I/O conditions ---stream-size=# : optimize compression parameters for streaming input of given number of bytes ++--adapt : dynamically adapt compression level to I/O conditions ++--stream-size=# : optimize compression parameters for streaming input of given number of bytes + --size-hint=# optimize compression parameters for streaming input of approximately this size ---target-compressed-block-size=# : make compressed block near targeted size - -T# : spawns # compression threads (default: 1, 0==# cores) - -B# : select size of each job (default: 0==automatic) ---rsyncable : compress using a rsync-friendly method (-B sets block size) ++--target-compressed-block-size=# : make compressed block near targeted size ++ -T# : spawns # compression threads (default: 1, 0==# cores) ++ -B# : select size of each job (default: 0==automatic) ++--rsyncable : compress using a rsync-friendly method (-B sets block size) --no-dictID : don't write dictID into header (dictionary compression) ---[no-]check : integrity check (default: enabled) ---[no-]compress-literals : force (un)compressed literals - -r : operate recursively on directories ---output-dir-flat[=directory]: all resulting files stored into `directory`. ---format=zstd : compress files to the .zst format (default) ---format=gzip : compress files to the .gz format ---test : test compressed file integrity +--[no-]check : integrity check (default: enabled) ++--[no-]compress-literals : force (un)compressed literals + -r : operate recursively on directories ++--output-dir-flat[=directory]: all resulting files stored into `directory`. ++--format=zstd : compress files to the .zst format (default) +--format=gzip : compress files to the .gz format - --format=xz : compress files to the .xz format - --format=lzma : compress files to the .lzma format +--test : test compressed file integrity --[no-]sparse : sparse mode (default: disabled) - -M# : Set a memory usage limit for decompression ---no-progress : do not display the progress bar --- : All arguments after "--" are treated as files + -M# : Set a memory usage limit for decompression ++--no-progress : do not display the progress bar +-- : All arguments after "--" are treated as files -Dictionary builder : ---train ## : create a dictionary from a training set of files +Dictionary builder : +--train ## : create a dictionary from a training set of files --train-cover[=k=#,d=#,steps=#,split=#,shrink[=#]] : use the cover algorithm with optional args - --train-fastcover[=k=#,d=#,f=#,steps=#,split=#,shrink[=#],accel=#] : use the fastcover algorithm with optional args + --train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#,shrink[=#]] : use the fast cover algorithm with optional args --train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9) - -o file : `file` is dictionary name (default: dictionary) ---maxdict=# : limit dictionary to specified size (default: 112640) + -o file : `file` is dictionary name (default: dictionary) +--maxdict=# : limit dictionary to specified size (default: 112640) --dictID=# : force dictionary ID to specified value (default: random) -Benchmark arguments : - -b# : benchmark file(s), using # compression level (default: 3) +Benchmark arguments : + -b# : benchmark file(s), using # compression level (default: 3) -e# : test all compression levels from -bX to # (default: 1) - -i# : minimum evaluation time in seconds (default: 3s) + -i# : minimum evaluation time in seconds (default: 3s) -B# : cut file into independent blocks of size # (default: no block) ---priority=rt : set process priority to real-time +--priority=rt : set process priority to real-time ``` -#### Restricted usage of Environment Variables -Using environment variables to set parameters has security implications. -Therefore, this avenue is intentionally restricted. -Only `ZSTD_CLEVEL` is supported currently, for setting compression level. -`ZSTD_CLEVEL` can be used to set the level between 1 and 19 (the "normal" range). -If the value of `ZSTD_CLEVEL` is not a valid integer, it will be ignored with a warning message. -`ZSTD_CLEVEL` just replaces the default compression level (`3`). -It can be overridden by corresponding command line arguments. +### Passing parameters through Environment Variables +`ZSTD_CLEVEL` can be used to modify the default compression level of `zstd` +(usually set to `3`) to another value between 1 and 19 (the "normal" range). +This can be useful when `zstd` CLI is invoked in a way that doesn't allow passing arguments. +One such scenario is `tar --zstd`. +As `ZSTD_CLEVEL` only replaces the default compression level, +it can then be overridden by corresponding command line arguments. -#### Long distance matching mode +There is no "generic" way to pass "any kind of parameter" to `zstd` in a pass-through manner. +Using environment variables for this purpose has security implications. +Therefore, this avenue is intentionally restricted and only supports `ZSTD_CLEVEL`. + +### Long distance matching mode The long distance matching mode, enabled with `--long`, is designed to improve the compression ratio for files with long matches at a large distance (up to the maximum window size, `128 MiB`) while still maintaining compression speed. @@@ -208,12 -216,12 +219,12 @@@ Compression Speed vs Ratio | Decompress | Method | Compression ratio | Compression speed | Decompression speed | |:-------|------------------:|-------------------------:|---------------------------:| --| `zstd -1` | `5.065` | `284.8 MB/s` | `759.3 MB/s` | ++| `zstd -1` | `5.065` | `284.8 MB/s` | `759.3 MB/s` | | `zstd -5` | `5.826` | `124.9 MB/s` | `674.0 MB/s` | | `zstd -10` | `6.504` | `29.5 MB/s` | `771.3 MB/s` | | `zstd -1 --long` | `17.426` | `220.6 MB/s` | `1638.4 MB/s` | --| `zstd -5 --long` | `19.661` | `165.5 MB/s` | `1530.6 MB/s`| --| `zstd -10 --long`| `21.949` | `75.6 MB/s` | `1632.6 MB/s`| ++| `zstd -5 --long` | `19.661` | `165.5 MB/s` | `1530.6 MB/s` | ++| `zstd -10 --long`| `21.949` | `75.6 MB/s` | `1632.6 MB/s` | On this file, the compression ratio improves significantly with minimal impact on compression speed, and the decompression speed doubles.