**Desktop (please complete the following information):**
- OS: [e.g. Mac]
- Version [e.g. 22]
- - Compiler [e.g gcc]
- - Flags [e.g O2]
+ - Compiler [e.g. gcc]
+ - Flags [e.g. O2]
- Other relevant hardware specs [e.g. Dual-core]
- Build system [e.g. Makefile]
or in their own feature branch.
When they are deemed ready for a release, they are merged into "release".
-As a consequences, all contributions must stage first through "dev"
+As a consequence, all contributions must stage first through "dev"
or their own feature branch.
## Pull Requests
- Static analyzers are full of false positive. The signal to noise ratio is actually pretty low.
- A good CI policy is "zero-warning tolerance". That means that all issues must be solved, including false positives. This quickly becomes a tedious workload.
- Multiple static analyzers will feature multiple kind of false positives, sometimes applying to the same code but in different ways leading to :
- + torteous code, trying to please multiple constraints, hurting readability and therefore maintenance. Sometimes, such complexity introduce other more subtle bugs, that are just out of scope of the analyzers.
+ + tortuous code, trying to please multiple constraints, hurting readability and therefore maintenance. Sometimes, such complexity introduce other more subtle bugs, that are just out of scope of the analyzers.
+ sometimes, these constraints are mutually exclusive : if one try to solve one, the other static analyzer will complain, they can't be both happy at the same time.
- As if that was not enough, the list of false positives change with each version. It's hard enough to follow one static analyzer, but multiple ones with their own update agenda, this quickly becomes a massive velocity reducer.
-This is different from running a static analyzer once in a while, looking at the output, and __cherry picking__ a few warnings that seem helpful, either because they detected a genuine risk of bug, or because it helps expressing the code in a way which is more readable or more difficult to misuse. These kind of reports can be useful, and are accepted.
+This is different from running a static analyzer once in a while, looking at the output, and __cherry picking__ a few warnings that seem helpful, either because they detected a genuine risk of bug, or because it helps expressing the code in a way which is more readable or more difficult to misuse. These kinds of reports can be useful, and are accepted.
## Continuous Integration
CI tests run every time a pull request (PR) is created or updated. The exact tests
take the time to run extensive, long-duration, and potentially cross-(os, platform, process, etc)
benchmarks on your end before submitting a PR. Of course, you will not be able to benchmark
your changes on every single processor and os out there (and neither will we) but do that best
-you can:) We've adding some things to think about when benchmarking below in the Benchmarking
+you can:) We've added some things to think about when benchmarking below in the Benchmarking
Performance section which might be helpful for you.
3. Optimizing performance for a certain OS, processor vendor, compiler, or network system is a perfectly
legitimate thing to do as long as it does not harm the overall performance health of Zstd.
benchmarking script works, you can check out programs/benchzstd.c
For example: say you have made a change that you believe improves the speed of zstd level 1. The
-very first thing you should use to asses whether you actually achieved any sort of improvement
+very first thing you should use to assess whether you actually achieved any sort of improvement
is `zstd -b`. You might try to do something like this. Note: you can use the `-i` option to
specify a running time for your benchmark in seconds (default is 3 seconds).
Usually, the longer the running time, the more stable your results will be.
### Profiling
There are a number of great profilers out there. We're going to briefly mention how you can
profile your code using `instruments` on mac, `perf` on linux and `visual studio profiler`
-on windows.
+on Windows.
Say you have an idea for a change that you think will provide some good performance gains
for level 1 compression on Zstd. Typically this means, you have identified a section of
Most profilers (including the profilers discussed below) will generate a call graph of
functions for you. Your goal will be to find your function of interest in this call graph
-and then inspect the time spent inside of it. You might also want to to look at the
-annotated assembly which most profilers will provide you with.
+and then inspect the time spent inside of it. You might also want to look at the annotated
+assembly which most profilers will provide you with.
#### Instruments
We will once again consider the scenario where you think you've identified a piece of code
* You will want a benchmark that runs for at least a few seconds (5 seconds will
usually be long enough). This way the profiler will have something to work with
and you will have ample time to attach your profiler to this process:)
- * I will just use benchzstd as my bencharmking script for this example:
+ * I will just use benchzstd as my benchmarmking script for this example:
```
$ zstd -b1 -i5 <my-data> # this will run for 5 seconds
```
Any variable that can be `const` (aka. read-only) **must** be `const`.
Any pointer which content will not be modified must be `const`.
This property is then controlled at compiler level.
- `const` variables are an important signal to readers that this variable isn’t modified.
+ `const` variables are an important signal to readers that this variable isn't modified.
Conversely, non-const variables are a signal to readers to watch out for modifications later on in the function.
* If a function must be inlined, mention it explicitly,
using project's own portable macros, such as `FORCE_INLINE_ATTR`,
# Cmake contributions
Contributions to the cmake build configurations are welcome. Please
-use case sensitivity that matches modern (ie. cmake version 2.6 and above)
+use case sensitivity that matches modern (i.e. cmake version 2.6 and above)
conventions of using lower-case for commands, and upper-case for
variables.
## How to build
-As cmake doesn't support command like `cmake clean`, it's recommended to perform a "out of source build".
+As cmake doesn't support command like `cmake clean`, it's recommended to perform an "out of source build".
To do this, you can create a new directory and build in it:
```sh
cd build/cmake
# Emscripten build using emcc.
emscripten_emcc_build() {
- # Compile the the same example as above
+ # Compile the same example as above
CC_FLAGS="-Wall -Wextra -Wshadow -Werror -Os -g0 -flto"
emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM $IN_FILES
# Did compilation work?
# Emscripten build using emcc.
emscripten_emcc_build() {
- # Compile the the same example as above
+ # Compile the same example as above
CC_FLAGS="-Wall -Wextra -Wshadow -Werror -Os -g0 -flto"
emcc $CC_FLAGS -s WASM=1 -I. -o $OUT_WASM $IN_FILES
# Did compilation work?
return False
# Finds 'file'. First the list of 'root' paths are searched, followed by the
-# the currently processing file's 'parent' path, returning a valid Path in
+# currently processing file's 'parent' path, returning a valid Path in
# canonical form. If no match is found None is returned.
#
def resolve_include(file: str, parent: Optional[Path] = None) -> Optional[Path]:
* Note: MEM_MODULE stops xxhash redefining BYTE, U16, etc., which are also
* defined in mem.h (breaking C99 compatibility).
*
- * Note: the undefs for xxHash allow Zstd's implementation to coincide with with
+ * Note: the undefs for xxHash allow Zstd's implementation to coincide with
* standalone xxHash usage (with global defines).
*
* Note: if you enable ZSTD_LEGACY_SUPPORT the combine.py script will need
* Note: MEM_MODULE stops xxhash redefining BYTE, U16, etc., which are also
* defined in mem.h (breaking C99 compatibility).
*
- * Note: the undefs for xxHash allow Zstd's implementation to coincide with with
+ * Note: the undefs for xxHash allow Zstd's implementation to coincide with
* standalone xxHash usage (with global defines).
*
* Note: if you enable ZSTD_LEGACY_SUPPORT the combine.py script will need
* files would be small relative to the size of the file.
*
* Various 'diffing' algorithms utilize this notion of edit distance and
- * the corrensponding concept of a minimal edit script between two
+ * the corresponding concept of a minimal edit script between two
* sequences to identify the regions within two files where they differ.
* The core algorithm used in this match finder is described in:
*
*
* Note: after some experimentation, this approach proved to not provide enough
* utility to justify the additional CPU used in finding matches. The one area
- * where this approach consistenly outperforms Zstandard even on level 19 is
- * when compressing small files (<10 KB) using a equally small dictionary that
+ * where this approach consistently outperforms Zstandard even on level 19 is
+ * when compressing small files (<10 KB) using an equally small dictionary that
* is very similar to the source file. For the use case that this was intended,
* (large similar files) this approach by itself took 5-10X longer than zstd-19 and
* generally resulted in 2-3X larger files. The core advantage that zstd-19 has
- * over this appraoch for match finding is the overlapping matches. This approach
+ * over this approach for match finding is the overlapping matches. This approach
* cannot find any.
*
* I'm leaving this in the contrib section in case this ever becomes interesting
* Dependencies
***************************************/
-/* Currently relies on qsort when combining contiguous matches. This can probabily
+/* Currently relies on qsort when combining contiguous matches. This can probably
* be avoided but would require changes to the algorithm. The qsort is far from
* the bottleneck in this algorithm even for medium sized files so it's probably
* not worth trying to address */
* Constants
***************************************/
-/* Just a sential for the entires of the diagnomal matrix */
+/* Just a sential for the entires of the diagonal matrix */
#define ZSTD_EDIST_DIAG_MAX (S32)(1 << 30)
/* How large should a snake be to be considered a 'big' snake.
#define ZSTD_EDIST_SNAKE_ITER_THRESH 200
/* After how many iterations should be just give up and take
- * the best availabe edit script for this round */
+ * the best available edit script for this round */
#define ZSTD_EDIST_EXPENSIVE_THRESH 1024
/*-*************************************
if (!useHeuristics)
continue;
- /* Everything under this point is a heuritic. Using these will
+ /* Everything under this point is a heuristic. Using these will
* substantially speed up the match finding. In some cases, taking
* the total match finding time from several minutes to seconds.
* Of course, the caveat is that the edit script found may no longer
}
} else if (srcLow == srcHigh) {
while (dictLow < dictHigh) {
- /* Reaching this point means deleteing dict[dictLow] from
- * the current positino of dict */
+ /* Reaching this point means deleting dict[dictLow] from
+ * the current position of dict */
dictLow++;
}
} else {
}
/* The matches from the approach above will all be of the form
- * (dictIdx, srcIdx, 1). this method combines contiguous matches
+ * (dictIdx, srcIdx, 1). This method combines contiguous matches
* of length MINMATCH or greater. Matches less than MINMATCH
* are discarded */
static void ZSTD_eDist_combineMatches(ZSTD_eDist_state* state)
* files would be small relative to the size of the file.
*
* Various 'diffing' algorithms utilize this notion of edit distance and
- * the corrensponding concept of a minimal edit script between two
+ * the corresponding concept of a minimal edit script between two
* sequences to identify the regions within two files where they differ.
* The core algorithm used in this match finder is described in:
*
*
* Note: after some experimentation, this approach proved to not provide enough
* utility to justify the additional CPU used in finding matches. The one area
- * where this approach consistenly outperforms Zstandard even on level 19 is
- * when compressing small files (<10 KB) using a equally small dictionary that
+ * where this approach consistently outperforms Zstandard even on level 19 is
+ * when compressing small files (<10 KB) using an equally small dictionary that
* is very similar to the source file. For the use case that this was intended,
* (large similar files) this approach by itself took 5-10X longer than zstd-19 and
* generally resulted in 2-3X larger files. The core advantage that zstd-19 has
- * over this appraoch for match finding is the overlapping matches. This approach
+ * over this approach for match finding is the overlapping matches. This approach
* cannot find any.
*
* I'm leaving this in the contrib section in case this ever becomes interesting
/**
* Dismissable scope guard.
* `Function` must be callable and take no parameters.
- * Unless `dissmiss()` is called, the callable is executed upon destruction of
+ * Unless `dismiss()` is called, the callable is executed upon destruction of
* `ScopeGuard`.
*
* Example:
It can be noted that a skippable frame
can be used to watermark a stream of concatenated frames
-embedding any kind of tracking information (even just an UUID).
+embedding any kind of tracking information (even just a UUID).
Users wary of such possibility should scan the stream of concatenated frames
in an attempt to detect such frame for analysis or removal.
</p></pre><BR>
<pre><b>unsigned ZSTD_getDictID_fromFrame(const void* src, size_t srcSize);
-</b><p> Provides the dictID required to decompressed the frame stored within `src`.
+</b><p> Provides the dictID required to decompress the frame stored within `src`.
If @return == 0, the dictID could not be decoded.
This could for one of the following reasons :
- The frame does not require a dictionary to be decoded (most common case).
- - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden information.
+ - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden piece of information.
Note : this use case also happens when using a non-conformant dictionary.
- `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`).
- This is not a Zstandard frame.
<pre><b>ZSTDLIB_STATIC_API size_t ZSTD_compressSequences(ZSTD_CCtx* const cctx, void* dst, size_t dstSize,
const ZSTD_Sequence* inSeqs, size_t inSeqsSize,
const void* src, size_t srcSize);
-</b><p> Compress an array of ZSTD_Sequence, associted with @src buffer, into dst.
+</b><p> Compress an array of ZSTD_Sequence, associated with @src buffer, into dst.
@src contains the entire input (not just the literals).
If @srcSize > sum(sequence.length), the remaining bytes are considered all literals
If a dictionary is included, then the cctx should reference the dict. (see: ZSTD_CCtx_refCDict(), ZSTD_CCtx_loadDictionary(), etc.)
`ZSTD_LIB_COMPRESSION, ZSTD_LIB_DECOMPRESSION`, `ZSTD_LIB_DICTBUILDER`,
and `ZSTD_LIB_DEPRECATED` as `0` to forgo compilation of the
corresponding features. This will also disable compilation of all
- dependencies (eg. `ZSTD_LIB_COMPRESSION=0` will also disable
+ dependencies (e.g. `ZSTD_LIB_COMPRESSION=0` will also disable
dictBuilder).
- There are a number of options that can help minimize the binary size of
}
/*! BIT_readBitsFast() :
- * unsafe version; only works only if nbBits >= 1 */
+ * unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, unsigned nbBits)
{
size_t const value = BIT_lookBitsFast(bitD, nbBits);
/* FSE_getMaxNbBits() :
* Approximate maximum cost of a symbol, in bits.
- * Fractional get rounded up (i.e : a symbol with a normalized frequency of 3 gives the same result as a frequency of 2)
+ * Fractional get rounded up (i.e. a symbol with a normalized frequency of 3 gives the same result as a frequency of 2)
* note 1 : assume symbolValue is valid (<= maxSymbolValue)
* note 2 : if freq[symbolValue]==0, @return a fake cost of tableLog+1 bits */
MEM_STATIC U32 FSE_getMaxNbBits(const void* symbolTTPtr, U32 symbolValue)
}
}
/* Now we spread those positions across the table.
- * The benefit of doing it in two stages is that we avoid the the
+ * The benefit of doing it in two stages is that we avoid the
* variable size inner loop, which caused lots of branch misses.
* Now we can run through all the positions without any branch misses.
- * We unroll the loop twice, since that is what emperically worked best.
+ * We unroll the loop twice, since that is what empirically worked best.
*/
{
size_t position = 0;
* @brief Used to prevent unwanted optimizations for @p var.
*
* It uses an empty GCC inline assembly statement with a register constraint
- * which forces @p var into a general purpose register (eg eax, ebx, ecx
+ * which forces @p var into a general purpose register (e.g. eax, ebx, ecx
* on x86) and marks it as modified.
*
* This is used in a few places to avoid unwanted autovectorization (e.g.
* @ingroup tuning
* @brief Selects the minimum alignment for XXH3's accumulators.
*
- * When using SIMD, this should match the alignment reqired for said vector
+ * When using SIMD, this should match the alignment required for said vector
* type, so, for example, 32 for AVX2.
*
* Default: Auto detected.
const BYTE* const iend = ip + srcSize;
int const loadLdmDict = params->ldmParams.enableLdm == ZSTD_ps_enable && ls != NULL;
- /* Assert that we the ms params match the params we're being given */
+ /* Assert that the ms params match the params we're being given */
ZSTD_assertEqualCParams(params->cParams, ms->cParams);
if (srcSize > ZSTD_CHUNKSIZE_MAX) {
size_t posInSrc, U32 windowLog, size_t dictSize)
{
U32 const windowSize = 1 << windowLog;
- /* posInSrc represents the amount of data the the decoder would decode up to this point.
+ /* posInSrc represents the amount of data the decoder would decode up to this point.
* As long as the amount of data decoded is less than or equal to window size, offsets may be
* larger than the total length of output decoded in order to reference the dict, even larger than
* window size. After output surpasses windowSize, we're limited to windowSize offsets again.
{
/* Heuristic: This should cover most blocks <= 16K and
* start to fade out after 16K to about 32K depending on
- * comprssibility.
+ * compressibility.
*/
return nbSeq >= 2048;
}
if (mostFrequent == nbSeq) {
*repeatMode = FSE_repeat_none;
if (isDefaultAllowed && nbSeq <= 2) {
- /* Prefer set_basic over set_rle when there are 2 or less symbols,
+ /* Prefer set_basic over set_rle when there are 2 or fewer symbols,
* since RLE uses 1 byte, but set_basic uses 5-6 bits per symbol.
* If basic encoding isn't possible, always choose RLE.
*/
* If it is set_compressed, first sub-block's literals section will be Treeless_Literals_Block
* and the following sub-blocks' literals sections will be Treeless_Literals_Block.
* @return : compressed size of literals section of a sub-block
- * Or 0 if it unable to compress.
+ * Or 0 if unable to compress.
* Or error code */
static size_t
ZSTD_compressSubBlock_literal(const HUF_CElt* hufTable,
const U32 mls = MIN(ms->cParams.minMatch, 6 /* mls caps out at 6 */);
DEBUGLOG(5, "ZSTD_row_update(), rowLog=%u", rowLog);
- ZSTD_row_update_internal(ms, ip, mls, rowLog, rowMask, 0 /* dont use cache */);
+ ZSTD_row_update_internal(ms, ip, mls, rowLog, rowMask, 0 /* don't use cache */);
}
#if defined(ZSTD_ARCH_X86_SSE2)
* the window through early invalidation.
* TODO: * Test the chunk size.
* * Try invalidation after the sequence generation and test the
- * the offset against maxDist directly.
+ * offset against maxDist directly.
*
* NOTE: Because of dictionaries + sequence splitting we MUST make sure
* that any offset used is valid at the END of the sequence, since it may
}
} else {
/* We have enough bytes buffered to initialize the hash,
- * and are have processed enough bytes to find a sync point.
+ * and have processed enough bytes to find a sync point.
* Start scanning at the beginning of the input.
*/
assert(mtctx->inBuff.filled >= RSYNC_MIN_BLOCK_SIZE);
size_t ZSTD_nextSrcSizeToDecompress(ZSTD_DCtx* dctx) { return dctx->expected; }
/**
- * Similar to ZSTD_nextSrcSizeToDecompress(), but when when a block input can be streamed,
- * we allow taking a partial block as the input. Currently only raw uncompressed blocks can
+ * Similar to ZSTD_nextSrcSizeToDecompress(), but when a block input can be streamed, we
+ * allow taking a partial block as the input. Currently only raw uncompressed blocks can
* be streamed.
*
* For blocks that can be streamed, this allows us to reduce the latency until we produce
default:
assert(0); /* impossible */
- RETURN_ERROR(GENERIC, "impossible to reach"); /* some compiler require default to do something */
+ RETURN_ERROR(GENERIC, "impossible to reach"); /* some compilers require default to do something */
}
}
* This could for one of the following reasons :
* - The frame does not require a dictionary (most common case).
* - The frame was built with dictID intentionally removed.
- * Needed dictionary is a hidden information.
+ * Needed dictionary is a hidden piece of information.
* Note : this use case also happens when using a non-conformant dictionary.
* - `srcSize` is too small, and as a result, frame header could not be decoded.
* Note : possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`.
default:
assert(0); /* impossible */
- RETURN_ERROR(GENERIC, "impossible to reach"); /* some compiler require default to do something */
+ RETURN_ERROR(GENERIC, "impossible to reach"); /* some compilers require default to do something */
} }
/* result */
dctx->litBufferEnd = dctx->litBuffer + litSize - ZSTD_LITBUFFEREXTRASIZE;
}
else {
- /* initially this will be stored entirely in dst during huffman decoding, it will partially shifted to litExtraBuffer after */
+ /* initially this will be stored entirely in dst during huffman decoding, it will partially be shifted to litExtraBuffer after */
dctx->litBuffer = (BYTE*)dst + expectedWriteSize - litSize;
dctx->litBufferEnd = (BYTE*)dst + expectedWriteSize;
}
}
}
/* Now we spread those positions across the table.
- * The benefit of doing it in two stages is that we avoid the the
+ * The benefit of doing it in two stages is that we avoid the
* variable size inner loop, which caused lots of branch misses.
* Now we can run through all the positions without any branch misses.
- * We unroll the loop twice, since that is what emperically worked best.
+ * We unroll the loop twice, since that is what empirically worked best.
*/
{
size_t position = 0;
U32 const ofnbBits = ofDInfo->nbBits;
/*
* As gcc has better branch and block analyzers, sometimes it is only
- * valuable to mark likelyness for clang, it gives around 3-4% of
+ * valuable to mark likeliness for clang, it gives around 3-4% of
* performance.
*/
/**
* Prepare a context for dictionary building.
- * The context is only dependent on the parameter `d` and can used multiple
+ * The context is only dependent on the parameter `d` and can be used multiple
* times.
* Returns 0 on success or error code on error.
* The context must be destroyed with `COVER_ctx_destroy()`.
/**
* Prepare a context for dictionary building.
- * The context is only dependent on the parameter `d` and can used multiple
+ * The context is only dependent on the parameter `d` and can be used multiple
* times.
* Returns 0 on success or error code on error.
* The context must be destroyed with `FASTCOVER_ctx_destroy()`.
Open `example\fullbench-dll.sln` to compile `fullbench-dll` that uses a
dynamic ZSTD library from the `dll` directory. The solution works with Visual C++
2010 or newer. When one will open the solution with Visual C++ newer than 2010
-then the solution will upgraded to the current version.
+then the solution will be upgraded to the current version.
## Using ZSTD DLL with Visual C++
}
/*! BIT_lookBitsFast :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BIT_lookBitsFast(BIT_DStream_t* bitD, U32 nbBits)
{
const U32 bitMask = sizeof(bitD->bitContainer)*8 - 1;
}
/*!BIT_readBitsFast :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, U32 nbBits)
{
size_t value = BIT_lookBitsFast(bitD, nbBits);
}
/*! BIT_lookBitsFast :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BIT_lookBitsFast(BIT_DStream_t* bitD, U32 nbBits)
{
const U32 bitMask = sizeof(bitD->bitContainer)*8 - 1;
}
/*!BIT_readBitsFast :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, U32 nbBits)
{
size_t value = BIT_lookBitsFast(bitD, nbBits);
}
/*! BIT_lookBitsFast :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BIT_lookBitsFast(BIT_DStream_t* bitD, U32 nbBits)
{
const U32 bitMask = sizeof(bitD->bitContainer)*8 - 1;
}
/*!BIT_readBitsFast :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BIT_readBitsFast(BIT_DStream_t* bitD, U32 nbBits)
{
size_t value = BIT_lookBitsFast(bitD, nbBits);
}
/*! BITv05_lookBitsFast :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BITv05_lookBitsFast(BITv05_DStream_t* bitD, U32 nbBits)
{
const U32 bitMask = sizeof(bitD->bitContainer)*8 - 1;
}
/*!BITv05_readBitsFast :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BITv05_readBitsFast(BITv05_DStream_t* bitD, unsigned nbBits)
{
size_t value = BITv05_lookBitsFast(bitD, nbBits);
}
/*! BITv06_lookBitsFast() :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BITv06_lookBitsFast(const BITv06_DStream_t* bitD, U32 nbBits)
{
U32 const bitMask = sizeof(bitD->bitContainer)*8 - 1;
}
/*! BITv06_readBitsFast() :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BITv06_readBitsFast(BITv06_DStream_t* bitD, U32 nbBits)
{
size_t const value = BITv06_lookBitsFast(bitD, nbBits);
}
/*! BITv07_lookBitsFast() :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BITv07_lookBitsFast(const BITv07_DStream_t* bitD, U32 nbBits)
{
U32 const bitMask = sizeof(bitD->bitContainer)*8 - 1;
}
/*! BITv07_readBitsFast() :
-* unsafe version; only works only if nbBits >= 1 */
+* unsafe version; only works if nbBits >= 1 */
MEM_STATIC size_t BITv07_readBitsFast(BITv07_DStream_t* bitD, U32 nbBits)
{
size_t const value = BITv07_lookBitsFast(bitD, nbBits);
* The zstd CLI defaults to a 110KB dictionary. You likely don't need a
* dictionary larger than that. But, most use cases can get away with a
* smaller dictionary. The advanced dictionary builders can automatically
- * shrink the dictionary for you, and select a the smallest size that
- * doesn't hurt compression ratio too much. See the `shrinkDict` parameter.
+ * shrink the dictionary for you, and select the smallest size that doesn't
+ * hurt compression ratio too much. See the `shrinkDict` parameter.
* A smaller dictionary can save memory, and potentially speed up
* compression.
*
* If @return == 0, the dictID could not be decoded.
* This could for one of the following reasons :
* - The frame does not require a dictionary to be decoded (most common case).
- * - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden information.
+ * - The frame was built with dictID intentionally removed. Whatever dictionary is necessary is a hidden piece of information.
* Note : this use case also happens when using a non-conformant dictionary.
* - `srcSize` is too small, and as a result, the frame header could not be decoded (only possible if `srcSize < ZSTD_FRAMEHEADERSIZE_MAX`).
* - This is not a Zstandard frame.
ZSTDLIB_STATIC_API size_t ZSTD_mergeBlockDelimiters(ZSTD_Sequence* sequences, size_t seqsSize);
/*! ZSTD_compressSequences() :
- * Compress an array of ZSTD_Sequence, associted with @src buffer, into dst.
+ * Compress an array of ZSTD_Sequence, associated with @src buffer, into dst.
* @src contains the entire input (not just the literals).
* If @srcSize > sum(sequence.length), the remaining bytes are considered all literals
* If a dictionary is included, then the cctx should reference the dict. (see: ZSTD_CCtx_refCDict(), ZSTD_CCtx_loadDictionary(), etc.)
* Without validation, providing a sequence that does not conform to the zstd spec will cause
* undefined behavior, and may produce a corrupted block.
*
- * With validation enabled, a if sequence is invalid (see doc/zstd_compression_format.md for
+ * With validation enabled, if sequence is invalid (see doc/zstd_compression_format.md for
* specifics regarding offset/matchlength requirements) then the function will bail out and
* return an error.
*
* in the range [dst, dst + pos) MUST not be modified during decompression
* or you will get data corruption.
*
- * When this flags is enabled zstd won't allocate an output buffer, because
+ * When this flag is enabled zstd won't allocate an output buffer, because
* it can write directly to the ZSTD_outBuffer, but it will still allocate
* an input buffer large enough to fit any compressed block. This will also
* avoid the memcpy() from the internal output buffer to the ZSTD_outBuffer.
/* when benchmark failed, it means one invocation of `benchFn` failed.
* The failure was detected by `errorFn`, operating on return values of `benchFn`.
* Returns the faulty return value.
- * note : this function will abort() program execution if benchmark did not failed.
+ * note : this function will abort() program execution if benchmark did not fail.
* always check if benchmark failed first !
*/
size_t BMK_extract_errorResult(BMK_runOutcome_t outcome);
if (cParams.strategy >= ZSTD_btopt) {
DISPLAYLEVEL(1, "[Optimal parser notes] Consider the following to improve patch size at the cost of speed:\n");
DISPLAYLEVEL(1, "- Use --single-thread mode in the zstd cli\n");
- DISPLAYLEVEL(1, "- Set a larger targetLength (eg. --zstd=targetLength=4096)\n");
- DISPLAYLEVEL(1, "- Set a larger chainLog (eg. --zstd=chainLog=%u)\n", ZSTD_CHAINLOG_MAX);
+ DISPLAYLEVEL(1, "- Set a larger targetLength (e.g. --zstd=targetLength=4096)\n");
+ DISPLAYLEVEL(1, "- Set a larger chainLog (e.g. --zstd=chainLog=%u)\n", ZSTD_CHAINLOG_MAX);
DISPLAYLEVEL(1, "Also consider playing around with searchLog and hashLog\n");
}
}
trimPath(currDirName)))
uniqueDirNr++;
- /* we need maintain original src dir name instead of trimmed
+ /* we need to maintain original src dir name instead of trimmed
* dir, so we can retrieve the original src dir's mode_t */
uniqueDirNames[uniqueDirNr - 1] = currDirName;
}
\fB\-\-no\-progress\fR: do not display the progress bar, but keep all other messages\.
.
.IP "\(bu" 4
-\fB\-\-show\-default\-cparams\fR: Shows the default compression parameters that will be used for a particular src file\. If the provided src file is not a regular file (eg\. named pipe), the cli will just output the default parameters\. That is, the parameters that are used when the src size is unknown\.
+\fB\-\-show\-default\-cparams\fR: Shows the default compression parameters that will be used for a particular src file\. If the provided src file is not a regular file (e\.g\. named pipe), the cli will just output the default parameters\. That is, the parameters that are used when the src size is unknown\.
.
.IP "\(bu" 4
\fB\-\-\fR: All arguments after \fB\-\-\fR are treated as files
Specify the maximum number of bits for a hash table\.
.
.IP
-Bigger hash tables cause less collisions which usually makes compression faster, but requires more memory during compression\.
+Bigger hash tables cause fewer collisions which usually makes compression faster, but requires more memory during compression\.
.
.IP
The minimum \fIhlog\fR is 6 (64 B) and the maximum is 30 (1 GiB)\.
Note: `--long` mode will be automatically activated if chainLog < fileLog
(fileLog being the windowLog required to cover the whole file). You
can also manually force it.
- Node: for all levels, you can use --patch-from in --single-thread mode
- to improve compression ratio at the cost of speed
+ Node: for all levels, you can use --patch-from in --single-thread mode
+ to improve compression ratio at the cost of speed
Note: for level 19, you can get increased compression ratio at the cost
of speed by specifying `--zstd=targetLength=` to be something large
- (i.e 4096), and by setting a large `--zstd=chainLog=`
+ (i.e. 4096), and by setting a large `--zstd=chainLog=`
* `--rsyncable` :
`zstd` will periodically synchronize the compression state to make the
compressed file more rsync-friendly. There is a negligible impact to
* `-M#`, `--memory=#`:
Set a memory usage limit. By default, Zstandard uses 128 MB for decompression
as the maximum amount of memory the decompressor is allowed to use, but you can
- override this manually if need be in either direction (ie. you can increase or
+ override this manually if need be in either direction (i.e. you can increase or
decrease it).
This is also used during compression when using with --patch-from=. In this case,
* `--show-default-cparams`:
Shows the default compression parameters that will be used for a
particular src file. If the provided src file is not a regular file
- (eg. named pipe), the cli will just output the default parameters.
+ (e.g. named pipe), the cli will just output the default parameters.
That is, the parameters that are used when the src size is unknown.
* `--`:
All arguments after `--` are treated as files
- `hashLog`=_hlog_, `hlog`=_hlog_:
Specify the maximum number of bits for a hash table.
- Bigger hash tables cause less collisions which usually makes compression
+ Bigger hash tables cause fewer collisions which usually makes compression
faster, but requires more memory during compression.
The minimum _hlog_ is 6 (64 B) and the maximum is 30 (1 GiB).
if (showDefaultCParams) {
if (operation == zom_decompress) {
- DISPLAY("error : can't use --show-default-cparams in decomrpession mode \n");
+ DISPLAY("error : can't use --show-default-cparams in decompression mode \n");
CLEAN_RETURN(1);
}
}
-h, --help show this help message and exit
--directory DIRECTORY
directory with files to benchmark
- --levels LEVELS levels to test eg ('1,2,3')
+ --levels LEVELS levels to test e.g. ('1,2,3')
--iterations ITERATIONS
number of benchmark iterations to run
--emails EMAILS email addresses of people who will be alerted upon
If a new commit is found it is compiled and a speed benchmark for this commit is performed.
The results of the speed benchmark are compared to the previous results.
If compression or decompression speed for one of zstd levels is lower than `lowerLimit` (an optional parameter, default 0.98) the speed benchmark is restarted.
-If second results are also lower than `lowerLimit` the warning e-mail is send to recipients from the list (the `emails` parameter).
+If second results are also lower than `lowerLimit` the warning e-mail is sent to recipients from the list (the `emails` parameter).
Additional remarks:
- To be sure that speed results are accurate the script should be run on a "stable" target system with no other jobs running in parallel
can use all --zstd parameter names and 'cParams' as a shorthand for all parameters used in ZSTD_compressionParameters
(Default: display all params available)
-P# : generated sample compressibility (when no file is provided)
- -t# : Caps runtime of operation in seconds (default : 99999 seconds (about 27 hours ))
+ -t# : Caps runtime of operation in seconds (default: 99999 seconds (about 27 hours))
-v : Prints Benchmarking output
-D : Next argument dictionary file
-s : Benchmark all files separately
parser = argparse.ArgumentParser()
parser.add_argument("--directory", help="directory with files to benchmark", default="golden-compression")
- parser.add_argument("--levels", help="levels to test eg ('1,2,3')", default="1")
+ parser.add_argument("--levels", help="levels to test e.g. ('1,2,3')", default="1")
parser.add_argument("--iterations", help="number of benchmark iterations to run", default="1")
parser.add_argument("--emails", help="email addresses of people who will be alerted upon regression. Only for continuous mode", default=None)
parser.add_argument("--frequency", help="specifies the number of seconds to wait before each successive check for new PRs in continuous mode", default=DEFAULT_MAX_API_CALL_FREQUENCY_SEC)
try:
description = """
Runs one or more regression tests.
- The fuzzer should have been built with with
+ The fuzzer should have been built with
LIB_FUZZING_ENGINE='libregression.a'.
Takes input from CORPORA.
"""
FUZZ_ASSERT(dictBuffer);
dictBuffer = generatePseudoRandomString(dictBuffer, dictSize);
}
- /* Generate window log first so we dont generate offsets too large */
+ /* Generate window log first so we don't generate offsets too large */
wLog = FUZZ_dataProducer_uint32Range(producer, ZSTD_WINDOWLOG_MIN, ZSTD_WINDOWLOG_MAX_32);
cLevel = FUZZ_dataProducer_int32Range(producer, -3, 22);
mode = (ZSTD_sequenceFormat_e)FUZZ_dataProducer_int32Range(producer, 0, 1);
*/
/**
- * This fuzz target attempts to comprss the fuzzed data with the simple
+ * This fuzz target attempts to compress the fuzzed data with the simple
* compression function with an output buffer that may be too small to
* ensure that the compressor never crashes.
*/
void* recon = (void*)malloc(size);
size_t refPrefixCompressedSize = 0;
- size_t refPrefixLdmComrpessedSize = 0;
+ size_t refPrefixLdmCompressedSize = 0;
size_t reconSize = 0;
ZSTD_CCtx* const cctx = ZSTD_createCCtx();
/* compress on level 1 using refPrefix and ldm */
ZSTD_CCtx_refPrefix(cctx, dict, size);;
CHECK_Z(ZSTD_CCtx_setParameter(cctx, ZSTD_c_enableLongDistanceMatching, 1))
- refPrefixLdmComrpessedSize = ZSTD_compress2(cctx, dst, dstSize, src, size);
- assert(!ZSTD_isError(refPrefixLdmComrpessedSize));
+ refPrefixLdmCompressedSize = ZSTD_compress2(cctx, dst, dstSize, src, size);
+ assert(!ZSTD_isError(refPrefixLdmCompressedSize));
/* test round trip refPrefix + ldm*/
ZSTD_DCtx_refPrefix(dctx, dict, size);
- reconSize = ZSTD_decompressDCtx(dctx, recon, size, dst, refPrefixLdmComrpessedSize);
+ reconSize = ZSTD_decompressDCtx(dctx, recon, size, dst, refPrefixLdmCompressedSize);
assert(!ZSTD_isError(reconSize));
assert(reconSize == size);
assert(!memcmp(recon, src, size));
/* make sure that refPrefixCompressedSize is significantly greater */
- assert(refPrefixCompressedSize > 10 * refPrefixLdmComrpessedSize);
- /* make sure the ldm comrpessed size is less than 1% of original */
- assert((double)refPrefixLdmComrpessedSize / (double)size < 0.01);
+ assert(refPrefixCompressedSize > 10 * refPrefixLdmCompressedSize);
+ /* make sure the ldm compressed size is less than 1% of original */
+ assert((double)refPrefixLdmCompressedSize / (double)size < 0.01);
ZSTD_freeDCtx(dctx);
ZSTD_freeCCtx(cctx);
}
/* hill climbing value for part 1 */
-/* Scoring here is a linear reward for all set constraints normalized between 0 to 1
+/* Scoring here is a linear reward for all set constraints normalized between 0 and 1
* (with 0 at 0 and 1 being fully fulfilling the constraint), summed with a logarithmic
* bonus to exceeding the constraint value. We also give linear ratio for compression ratio.
* The constant factors are experimental.
#### Example
-We have take the file `test/example.c` from [the zlib library distribution](http://zlib.net/) and copied it to [zlibWrapper/examples/example.c](examples/example.c).
+We have taken the file `test/example.c` from [the zlib library distribution](http://zlib.net/) and copied it to [zlibWrapper/examples/example.c](examples/example.c).
After compilation and execution it shows the following results:
```
zlib version 1.2.8 = 0x1280, compile flags = 0x65
projects / thirdparty / zstd.git / commitdiff
