Nick Terrell [Tue, 26 Mar 2024 17:01:19 +0000 (10:01 -0700)]
[fuzz] Turn off -Werror by default
This was causing OSS-Fuzz errors, due to compiler differences.
* Fix the issue
* Also turn off -Werror so we don't fail fuzzer builds for warnings
* Turn on -Werror in our CI
Nick Terrell [Tue, 19 Mar 2024 19:37:55 +0000 (12:37 -0700)]
Fix & fuzz ZSTD_generateSequences
This function was seriously flawed:
* It didn't do output bounds checks
* It produced invalid sequences when an uncompressed or RLE block was emitted
* It produced invalid sequences when the block splitter was enabled
* It produced invalid sequences when ZSTD_c_targetCBlockSize was enabled
I've attempted to fix these issues, but this function is just a bad idea,
so I've marked it as deprecated and unsafe. We should replace it with
`ZSTD_extractSequences()` which operates on a compressed frame.
Yonatan Komornik [Mon, 18 Mar 2024 22:25:22 +0000 (15:25 -0700)]
Fail on errors when building fuzzers
Fails on errors when building fuzzers with `fuzz.py` (adds `Werror`).
Currently allows `declaration-after-statement`, `c++-compat` and
`deprecated` as they are abundant in code (some fixes to
`declaration-after-statement` are presented in this commit).
Yonatan Komornik [Mon, 18 Mar 2024 22:36:40 +0000 (15:36 -0700)]
Fix bugs in simple decompression fuzzer (#3978)
Fixes 2 issue in `simple_decompress.c`:
1. Wrong type used for storing the results of `ZSTD_findDecompressedSize` resulting in never matching to `ZSTD_CONTENTSIZE_ERROR` or `ZSTD_CONTENTSIZE_UNKNOWN`.
2. Experimental API is used (`ZSTD_findDecompressedSize`) without defining `ZSTD_STATIC_LINKING_ONLY`.
Nick Terrell [Thu, 14 Mar 2024 19:12:55 +0000 (12:12 -0700)]
[cmake] Fix up PR #3716
* Make a variable `PublicHeaders` for Zstd's public headers
* Add `PublicHeaders` to `Headers`, which was missing
* Only export `${LIBRARY_DIR}` publicly, not `common/`
* Switch the `target_include_directories()` to `INTERFACE` because zstd uses relative includes internally, so doesn't need any include directories to build
* Switch installation to use the `PublicHeaders` variable, and test that the right headers are installed
Nick Terrell [Thu, 14 Mar 2024 15:47:04 +0000 (08:47 -0700)]
[cmake] Always create libzstd target
If both `ZSTD_BUILD_SHARED` and `ZSTD_BUILD_STATIC` are set, then cmake exports the libraries `libzstd_shared` and `libzstd_static` only.
It does not export `libzstd`, which is only exported when exactly one of `ZSTD_BUILD_SHARED` and `ZSTD_BUILD_STATIC` is set.
This PR exports `libzstd` in that case, based on the value of the standard CMake variable [`BUILD_SHARED_LIBS`](https://cmake.org/cmake/help/latest/variable/BUILD_SHARED_LIBS.html).
This ensures that `libzstd` can always be used to refer to the exported zstd library, since the build errors if neither `ZSTD_BUILD_SHARED` nor `ZSTD_BUILD_STATIC` are set.
I tested all the possible combinations of `ZSTD_BUILD_SHARED`, `ZSTD_BUILD_STATIC`, and `BUILD_SHARED_LIBS` and they always worked as expected:
* If only exactly one of `ZSTD_BUILD_SHARED` and `ZSTD_BUILD_STATIC` is set, that is used as `libzstd`.
* Otherwise, libzstd is set based on `BUILD_SHARED_LIBS`.
W. Felix Handte [Wed, 13 Mar 2024 20:56:54 +0000 (16:56 -0400)]
Promote `ZSTD_c_targetCBlockSize` Parameter to Stable API
This feature has demonstrated itself to be useful in web compression and we
want to encourage other folks to use it. But we currently make it difficult
to do so since it's locked away in the experimental API.
The API itself is really straightforward and I think it's fine to commit to
maintaining support / compatibility for this API even if in the future the
underlying implementation may continue to evolve.
Note that this commit changes its enum name and also its numeric value. Users
who respected the instructions of using the experimental API should be fine
with both of these changes since they should only have referred to it by the.
Conceivably someone could have done bad feature detection of this capability
by doing `#ifdef ZSTD_c_targetCBlockSize` which will now return false since
it's no longer a macro... but I think that's an acceptable hypothetical
breakage.
W. Felix Handte [Tue, 12 Mar 2024 21:01:17 +0000 (14:01 -0700)]
Use `utimensat()` on FreeBSD
FreeBSD only claims to support POSIX 2001 [0]. But they do in fact support
`utimensat()`. This adds a specific check to opt them in to using it. This
value was selected by consulting [1].
Yonatan Komornik [Mon, 11 Mar 2024 23:28:32 +0000 (16:28 -0700)]
Fix AsyncIO reading seed queueing (#3940)
Fixes a bug in AsyncIO where we queue reads after opening a file so our queue will always be saturated (or as saturated as possible).
Previous code was looping up to `availableJobsCount` not realizing `availableJobsCount` was also decreasing in each iteration, so instead of queueing 10 jobs we'd queue 5 (and instead of 2 we'd queue 1).
This PR fixes the loop to queue as long as `availableJobsCount` is not 0.
Yann Collet [Mon, 11 Mar 2024 18:38:55 +0000 (11:38 -0700)]
fix #3719
only disable `--rm` at end of command line parsing,
so that `-c` only disables `--rm` if it's effectively selected,
and not if it's overriden by a later `-o FILE` command.
Yann Collet [Fri, 8 Mar 2024 22:55:38 +0000 (14:55 -0800)]
new method to deal with offset==0
in this new method, when an `offset==0` is detected,
it's converted into (size_t)(-1), instead of 1.
The logic is that (size_t)(-1) is effectively an extremely large positive number,
which will not pass the offset distance test at next stage (`execSequence()`).
Checked the source code, and offset is always checked (as it should),
using a formula which is not vulnerable to arithmetic overflow:
```
RETURN_ERROR_IF(sequence.offset > (size_t)(oLitEnd - virtualStart),
```
The benefit is that such a case (offset==0) is always detected as corrupted data
as opposed to relying on the checksum to detect the error.
Yann Collet [Mon, 4 Mar 2024 08:16:01 +0000 (00:16 -0800)]
fix LLU->ULL
LLU is a correct prefix according to C99 & C11 standards (but not C90).
However, older versions of Visual Studio do not work with it.
Replace by ULL, which doesn't have this issue.