Ilya Leoshkevich [Thu, 20 Oct 2022 00:28:47 +0000 (02:28 +0200)]
IBM zSystems DFLTCC: Support inflate with small window
There is no hardware control for DFLTCC window size, and because of
that supporting small windows for deflate is not trivial: one has to
make sure that DFLTCC does not emit large distances, which most likely
entails somehow trimming the window and/or input in order to make sure
that whave + avail_in <= wsize.
But inflate is much easier: one only has to allocate enough space. Do
that in dfltcc_alloc_window(), and also introduce ZCOPY_WINDOW() in
order to copy everything, not just what the software implementation
cares about.
After this change, software and hardware window formats no longer
match: the software will use wbits and wsize, and the hardware will use
HB_BITS and HB_SIZE. Unlike deflate, inflate does not switch between
software and hardware implementations mid-stream, which leaves only
inflateSetDictionary() and inflateGetDictionary() interesting.
Use CMake detection for Emscripten due to a bug in vcpkg.
When installing for wasm32 with vcpkg, CMake's try_run does not end up running
with the correct compiler. This issue has been mention in a few existing vcpkg
portfiles.
Alternatively, we could check the VCPKG_TARGET_ARCHITECTURE variable for wasm32
but instead we check for EMSCRIPTEN CMake variable.
This patch adds the ability to run zlib-ng test suite against the
original zlib as follows:
cmake -DZLIB_COMPAT=ON -DZLIBNG_ENABLE_TESTS=OFF .
make
LD_LIBRARY_PATH=/usr/lib/x86_64-linux-gnu make test
The benefit of this is that modifications to the original zlib can be
tested with a more extensive zlib-ng's testsuite, and the assumptions
that the zlib-ng tests make can be validated against the original zlib.
In addition to a number of tests that exercise purely zlib-ng specific
API, there are a few that expect zlib-ng specific behavior from the
original zlib API:
- deflate() (obviously) emits different streams
- zlib-ng's deflatePrime() can take more than 16 bits
- zVersion() returns a different string
Adjust or disable the respective tests for ZLIBNG_ENABLE_TESTS=OFF.
Ilya Leoshkevich [Thu, 20 Oct 2022 04:28:44 +0000 (06:28 +0200)]
Fix deflate() with Z_BEST_COMPRESSION ignoring the dictionary
deflate_slow() uses s->quick_insert_string(), while
deflateSetDictionary() uses functable.insert_string(). These functions
use different hashing algorithms, which leads to deflate_slow()
ignoring the dictionary.
Fix by using s->insert_string() instead of functable.insert_string(),
which is set by lm_set_level() and matches what deflate_*() uses
(suggested by Mika Lindqvist).
Mika Lindqvist [Wed, 5 Oct 2022 10:58:28 +0000 (13:58 +0300)]
Fix memcpy() overflow in adler32_fold.c and crc32_fold.c
* On 32-bit platforms, last parameter of memcpy() is limited to SSIZE_MAX, but is likely to overlap if used
IBM Z DFLTCC: Fix updating strm.adler with inflate()
inflate() does not update strm.adler with DFLTCC.
Add a missing assignment to dfltcc_inflate() to fix this.
Note that deflate() is not affected.
Also add a test to prevent regressions.
Mika Lindqvist [Sun, 11 Sep 2022 13:15:10 +0000 (16:15 +0300)]
[Compat] Don't use uint32_t for z_crc_t
* We don't include stdint.h as it must be included before stdarg.h and other headers might include stdarg.h before us
Dougall Johnson [Mon, 22 Aug 2022 00:57:39 +0000 (10:57 +1000)]
Inflate: Increase max root table sizes to 10 and 9
This increases the size of the `codes` array by 1920 bytes (33%), but
improves performance a little. Root table size is still limited by the
maximum code length in use, so tiny files typically see no change to
table-building time, as they don't use longer codes.
Mika Lindqvist [Fri, 19 Aug 2022 12:00:21 +0000 (15:00 +0300)]
If the extra field was larger than the space the user provided with
inflateGetHeader(), and if multiple calls of inflate() delivered
the extra header data, then there could be a buffer overflow of the
provided space. This commit assures that provided space is not
exceeded.
Mosè Giordano [Wed, 24 Aug 2022 19:39:03 +0000 (20:39 +0100)]
Actually run `configure` CI on macOS with GCC
`gcc` is an alias for Apple Clang on macOS. See for example https://github.com/zlib-ng/zlib-ng/runs/7963904948?check_suite_focus=true
```
gcc -O2 -std=c11 -Wall -fPIC -DNDEBUG -DHAVE_POSIX_MEMALIGN -DWITH_GZFILEOP -fno-semantic-interposition -DHAVE_VISIBILITY_HIDDEN -DHAVE_VISIBILITY_INTERNAL -DHAVE_BUILTIN_CTZ -DHAVE_BUILTIN_CTZLL -DX86_FEATURES -DX86_AVX2 -DX86_AVX2_ADLER32 -DX86_AVX_CHUNKSET -DX86_AVX512 -DX86_AVX512_ADLER32 -DX86_MASK_INTRIN -DX86_AVX512VNNI -DX86_AVX512VNNI_ADLER32 -DX86_SSE41 -DX86_SSE42_CRC_HASH -DX86_SSE42_ADLER32 -DX86_SSE42_CRC_INTRIN -DX86_SSE2 -DX86_SSE2_CHUNKSET -DX86_SSSE3 -DX86_SSSE3_ADLER32 -DX86_PCLMULQDQ_CRC -DPIC -I/Users/runner/work/zlib-ng/zlib-ng -c -o adler32.lo /Users/runner/work/zlib-ng/zlib-ng/adler32.c
clang: warning: argument unused during compilation: '-fno-semantic-interposition' [-Wunused-command-line-argument]
```
In order to use the real GCC you have to call `gcc-9`, `gcc-10`, or `gcc-11`: https://github.com/actions/runner-images/blob/06dd4c14e4aa8c14febdd8d6cf123b8d770b4e4a/images/macos/macos-11-Readme.md#language-and-runtime.
Fixed content already populated error in CMake scripts. #1327
Should only need to use either FetchContent_MakeAvailable or
FetchContent_GetProperties and FetchContent_Populate but not both methods. We
use the later for CMake compatibility with lower versions.
Viktor Szakats [Sun, 17 Jul 2022 19:33:01 +0000 (19:33 +0000)]
cmake: respect custom `RC` flags and delete `GCC_WINDRES`
Before this patch, `zlib.rc` was compiled using a manual command [1] when
using the MinGW (and MSYS/Cygwin) toolchains. This method ignores
`CMAKE_RC_FLAGS` and offers no other way to pass a custom flag, breaking
the build in cases where a custom `windres` option is required. E.g.
`--target=` or `-I` on some platforms and configuration, in particular
with `llvm-windres`.
This patch deletes the special case for these toolchains and lets CMake
compile the `.rc` file the default way used for all Windows targets.
I'm not entirely sure why this special case was added back in 2011. The
need to pass `-DGCC_WINDRES` is my suspect. We can resolve this much
simpler by adding this line for the targets that require it:
set(CMAKE_RC_FLAGS "${CMAKE_RC_FLAGS} -DGCC_WINDRES")
But, the `.rc` line protected by `GCC_WINDRES`, these days work just fine
with `windres`. Moreover, that protected line are oboslete flags from the
16-bit era, which for a long time have no effect, as documented here:
<https://docs.microsoft.com/windows/win32/menurc/common-resource-attributes>
So, this patch deletes `GCC_WINDRES` from the project entirely.
* Moves cmake scripts for testing into test/cmake.
* Separates out related add_tests into separate cmake scripts.
* Moves building test binaries into CMakeLists.txt in test directory.
Fixed functions declared without a prototype warning in tools.
tools/maketrees.c:101:29: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
static void gen_trees_header()
tools/makecrct.c:65:27: warning: a function declaration without a prototype is deprecated in all versions of C [-Wstrict-prototypes]
static void make_crc_table()
Ilya Leoshkevich [Mon, 27 Jun 2022 18:16:54 +0000 (20:16 +0200)]
Uninstall strawberryperl
strawberryperl installs /c/Strawberry/c/bin/libstdc++-6.dll, which is
incompatible with the mingw64 one. zlib-ng does not need perl, so
simply remove it.
Ilya Leoshkevich [Tue, 28 Jun 2022 08:51:39 +0000 (10:51 +0200)]
Bump _POSIX_C_SOURCE to 200809 for strdup()
Google Test uses strdup(), which makes building tests fail on a fresh
MSYS2 setup:
In file included from zlib-ng/_deps/googletest-src/googletest/include/gtest/internal/gtest-internal.h:40,
from zlib-ng/_deps/googletest-src/googletest/include/gtest/gtest.h:62,
from zlib-ng/test/test_compress.cc:17:
zlib-ng/_deps/googletest-src/googletest/include/gtest/internal/gtest-port.h: In function ‘char* testing::internal::posix::StrDup(const char*)’:
zlib-ng/_deps/googletest-src/googletest/include/gtest/internal/gtest-port.h:2046:47: error: ‘strdup’ was not declared in this scope; did you mean ‘StrDup’?
2046 | inline char* StrDup(const char* src) { return strdup(src); }
| ^~~~~~
| StrDup
Bump _POSIX_C_SOURCE to enable this function. An alternative solution
would be to define _POSIX_C_SOURCE in test/CMakeLists.txt, but having a
bigger value for zlib-ng itself should not hurt.
Include zbuild.h earlier in minideflate.c in order to make the new
setting take effect for this file.
Adam Stylinski [Thu, 2 Jun 2022 22:46:56 +0000 (18:46 -0400)]
Improve the swizzle of the memory magazine fed in a chunk copy for neon
Like the x86 variant, we can leverage the same tables to load a vector
register worth of values. This shows a vast improvement in places where
very large run length encodes can be found in the lz runs.