From: Yann Collet Date: Thu, 22 Oct 2020 19:31:23 +0000 (-0700) Subject: updated build documentation X-Git-Tag: v1.4.7~46^2~2 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=ce6cd07c3344df80e9830f619f950329bf29a388;p=thirdparty%2Fzstd.git updated build documentation --- diff --git a/lib/README.md b/lib/README.md index ff62b668a..db9170a75 100644 --- a/lib/README.md +++ b/lib/README.md @@ -168,6 +168,26 @@ file it should be linked with `dll\libzstd.dll`. For example: The compiled executable will require ZSTD DLL which is available at `dll\libzstd.dll`. +#### Advanced Build options + +The build system requires a hash function in order to +separate object files created with different compilation flags. +By default, it tries to use `md5sum` or equivalent. +The hash function can be manually switched by setting the `HASH` variable. +For example : `make HASH=xxhsum` +The hash function needs to generate at least 64-bit using hexadecimal format. +When no hash function is found, +the Makefile just generates all object files into the same default directory, +irrespective of compilation flags. +This functionality only matters if `libzstd` is compiled multiple times +with different build flags. + +The build directory, where object files are stored +can also be manually controlled using variable `BUILD_DIR`, +for example `make BUILD_DIR=objectDir/v1`. +In which case, the hash function doesn't matter. + + #### Deprecated API Obsolete API on their way out are stored in directory `lib/deprecated`. diff --git a/programs/README.md b/programs/README.md index 53f716bda..cf7f5ba46 100644 --- a/programs/README.md +++ b/programs/README.md @@ -3,7 +3,7 @@ Command Line Interface for Zstandard library Command Line Interface (CLI) can be created using the `make` command without any additional parameters. There are however other Makefile targets that create different variations of CLI: -- `zstd` : default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and support for decompression of legacy zstd formats +- `zstd` : default CLI supporting gzip-like arguments; includes dictionary builder, benchmark, and supports decompression of legacy zstd formats - `zstd_nolegacy` : Same as `zstd` but without support for legacy zstd formats - `zstd-small` : CLI optimized for minimal size; no dictionary builder, no benchmark, and no support for legacy zstd formats - `zstd-compress` : version of CLI which can only compress into zstd format @@ -147,76 +147,91 @@ FILE : a filename Arguments : -# : # compression level (1-19, default: 3) -d : decompression - -D file: use `file` as Dictionary - -o file: result stored into `file` (only if 1 input file) - -f : overwrite output without prompting and (de)compress links + -D DICT: use DICT as Dictionary for compression or decompression + -o file: result stored into `file` (only 1 output file) + -f : overwrite output without prompting, also (de)compress links --rm : remove source file(s) after successful de/compression -k : preserve source file(s) (default) -h/-H : display help/long help and exit Advanced arguments : -V : display Version number and exit + -c : force write to standard output, even if it is the console -v : verbose mode; specify multiple times to increase verbosity -q : suppress warnings; specify twice to suppress errors too - -c : force write to standard output, even if it is the console - -l : print information about zstd compressed files ---exclude-compressed: only compress files that are not previously compressed +--no-progress : do not display the progress counter + -r : operate recursively on directories +--filelist FILE : read list of files to operate upon from FILE +--output-dir-flat DIR : processed files are stored into DIR +--output-dir-mirror DIR : processed files are stored into DIR respecting original directory structure +--[no-]check : during compression, add XXH64 integrity checksum to frame (default: enabled). If specified with -d, decompressor will ignore/validate checksums in compressed frame (default: validate). +-- : All arguments after "--" are treated as files + +Advanced compression arguments : --ultra : enable levels beyond 19, up to 22 (requires more memory) --long[=#]: enable long distance matching with given window log (default: 27) --fast[=#]: switch to very fast compression levels (default: 1) --adapt : dynamically adapt compression level to I/O conditions ---stream-size=# : optimize compression parameters for streaming input of given number of bytes ---size-hint=# optimize compression parameters for streaming input of approximately this size ---target-compressed-block-size=# : make compressed block near targeted size -T# : spawns # compression threads (default: 1, 0==# cores) -B# : select size of each job (default: 0==automatic) +--single-thread : use a single thread for both I/O and compression (result slightly different than -T1) --rsyncable : compress using a rsync-friendly method (-B sets block size) ---no-dictID : don't write dictID into header (dictionary compression) ---[no-]check : integrity check (default: enabled) +--exclude-compressed: only compress files that are not already compressed +--stream-size=# : specify size of streaming input from `stdin` +--size-hint=# optimize compression parameters for streaming input of approximately this size +--target-compressed-block-size=# : generate compressed block of approximately targeted size +--no-dictID : don't write dictID into header (dictionary compression only) --[no-]compress-literals : force (un)compressed literals - -r : operate recursively on directories ---output-dir-flat[=directory]: all resulting files stored into `directory`. --format=zstd : compress files to the .zst format (default) --format=gzip : compress files to the .gz format +--format=xz : compress files to the .xz format +--format=lzma : compress files to the .lzma format +--format=lz4 : compress files to the .lz4 format + +Advanced decompression arguments : + -l : print information about zstd compressed files --test : test compressed file integrity ---[no-]sparse : sparse mode (default: disabled) -M# : Set a memory usage limit for decompression ---no-progress : do not display the progress bar --- : All arguments after "--" are treated as files +--[no-]sparse : sparse mode (default: disabled) Dictionary builder : --train ## : create a dictionary from a training set of files --train-cover[=k=#,d=#,steps=#,split=#,shrink[=#]] : use the cover algorithm with optional args --train-fastcover[=k=#,d=#,f=#,steps=#,split=#,accel=#,shrink[=#]] : use the fast cover algorithm with optional args --train-legacy[=s=#] : use the legacy algorithm with selectivity (default: 9) - -o file : `file` is dictionary name (default: dictionary) + -o DICT : DICT is dictionary name (default: dictionary) --maxdict=# : limit dictionary to specified size (default: 112640) --dictID=# : force dictionary ID to specified value (default: random) Benchmark arguments : -b# : benchmark file(s), using # compression level (default: 3) - -e# : test all compression levels from -bX to # (default: 1) + -e# : test all compression levels successively from -b# to -e# (default: 1) -i# : minimum evaluation time in seconds (default: 3s) -B# : cut file into independent blocks of size # (default: no block) + -S : output one benchmark result per input file (default: consolidated result) --priority=rt : set process priority to real-time ``` ### Passing parameters through Environment Variables +There is no "generic" way to pass "any kind of parameter" to `zstd` in a pass-through manner. +Using environment variables for this purpose has security implications. +Therefore, this avenue is intentionally restricted and only supports `ZSTD_CLEVEL` and `ZSTD_NBTHREADS`. + `ZSTD_CLEVEL` can be used to modify the default compression level of `zstd` (usually set to `3`) to another value between 1 and 19 (the "normal" range). -`ZSTD_NBTHREADS` can be used to specify number of threads that `zstd` will use during compression, which by default is `1`. + +`ZSTD_NBTHREADS` can be used to specify a number of threads +that `zstd` will use for compression, which by default is `1`. This functionality only exists when `zstd` is compiled with multithread support. +`0` means "use as many threads as detected cpu cores on local system". The max # of threads is capped at: `ZSTDMT_NBWORKERS_MAX==200`. This functionality can be useful when `zstd` CLI is invoked in a way that doesn't allow passing arguments. One such scenario is `tar --zstd`. As `ZSTD_CLEVEL` and `ZSTD_NBTHREADS` only replace the default compression level -and number of threads, respectively, they can both be overridden by corresponding command line arguments: +and number of threads respectively, they can both be overridden by corresponding command line arguments: `-#` for compression level and `-T#` for number of threads. -There is no "generic" way to pass "any kind of parameter" to `zstd` in a pass-through manner. -Using environment variables for this purpose has security implications. -Therefore, this avenue is intentionally restricted and only supports `ZSTD_CLEVEL` and `ZSTD_NBTHREADS`. ### Long distance matching mode The long distance matching mode, enabled with `--long`, is designed to improve