Sebastian Pop [Tue, 26 Mar 2019 16:59:45 +0000 (11:59 -0500)]
fix oss-fuzz/13863
The oss fuzzers started failing with the following assert
```
ASSERT: 0 == memcmp(data + offset, buf, len)
```
after the following patch has been pulled in the tree:
define and use chunkmemset instead of byte_memset for INFFAST_CHUNKSIZE
```
The function chunkcopysafe is assuming that the input `len` is less than 16 bytes:
```
if ((safe - out) < (ptrdiff_t)INFFAST_CHUNKSIZE) {
```
but we were called with `len = 22` because `safe` was defined too small:
```
- safe = out + (strm->avail_out - INFFAST_CHUNKSIZE);
```
and the difference `safe - out` was 16 bytes smaller than the actual `len`.
The patch fixes the initialization of `safe` to:
```
+ safe = out + strm->avail_out;
```
Update x86 and x86_64 arch checks to use the recommended
define names, resulting in improved compiler support.
Based on the overviews from several sites, such as:
http://nadeausoftware.com/articles/2012/02/c_c_tip_how_detect_processor_type_using_compiler_predefined_macros
Default sse2 to be available on x86_64 arch.
Allow x86 arch to force sse2 availability.
It still depends on sse2 intrinsics being detected and optimizations enabled.
Sebastian Pop [Fri, 25 Jan 2019 17:44:46 +0000 (11:44 -0600)]
ARM: check cpu feature once at init time
This makes the checks for arm cpu features as inexpensive as on the x86 side
by calling the runtime feature detection once in deflate/inflate init and then
storing the result in a global variable.
Sebastian Pop [Wed, 23 Jan 2019 20:17:49 +0000 (14:17 -0600)]
ARM: enable neon and acle when available
this patch changes the default for cmake and configure to enable neon and acle
when no flags have been specified. This mimics the default for x86. The flags
--neon and --acle are changed to the opposite --without-neon and --without-acle
allowing the user to disable detection of neon and acle.
Hans Johnson [Wed, 16 Jan 2019 16:07:14 +0000 (10:07 -0600)]
ENH: Conistently read version information from one place
Apply "one-definition-rule" getting the version information.
The zlib.h file is the definitive source for version
information. Add code to extract version information
for both ZLIB and ZLIBNG variants from the zlib.h file.
Hans Johnson [Wed, 16 Jan 2019 15:48:04 +0000 (09:48 -0600)]
ENH: Use modern (cmake 3.0+) project signature
Set a name, version, and enable languages for the entire project.
This signature uses a common paradigm for propogating defaults
for setting version information in several targets.
Hans Johnson [Wed, 16 Jan 2019 15:41:53 +0000 (09:41 -0600)]
ENH: Allow setting the C_STANDARD version from command line
Allow for both C99 and C11 standards to be used. By default,
do not support compiler extensions, but allow this setting to
be overridden by the developer.
The setting of the language standard needs to be performed before
the "project" directive.
Hans Johnson [Wed, 16 Jan 2019 15:13:36 +0000 (09:13 -0600)]
BUG: CMake 2.8.4 does not support required features
CMake Error at CMakeLists.txt:682 (target_include_directories):
Unknown CMake command "target_include_directories".
target_include_directories was introduced in cmake 3
C_STANDARD 99 support was added in cmake 3.1
Ubuntu 16.04 (Xenial) was distributed with cmake 3.5.1 by default.
CMake versions > 3.3 allow simplified implementations of modern
cmake compilation support.
===
For newer versions of cmake (upto a maximum validated version [3.13.2 in this case]),
use newer cmake policies available. The newer policies often provide better
diagnostics for subtle build related issues.
Sebastian Pop [Tue, 22 Jan 2019 19:46:34 +0000 (13:46 -0600)]
fix bug #289: use strcpy instead of strncpy
to avoid a GCC 8 warning:
test/example.c:465:48: warning: argument to ‘sizeof’ in ‘strncpy’ call is the same expression as the source; did you mean to use the size of the destination? [-Wsizeof-pointer-memaccess]
strncpy((char*)uncompr, garbage_str, sizeof(garbage_str));
Sebastian Pop [Thu, 17 Jan 2019 19:07:16 +0000 (13:07 -0600)]
fix bug #208: let the compiler generate code for unaligned stores
to avoid this error:
zlib-ng/arch/x86/deflate_quick.c:154:5: runtime error: store to misaligned address 0x631000014801 for type 'unsigned int', which requires 4 byte alignment
0x631000014801: note: pointer points here
00 80 02 d3 07 00 00 be be be be be be be be be be be be be be be be be be be be be be be be be
^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior zlib-ng/arch/x86/deflate_quick.c:154:5 in
It looks like it is a known problem:
https://gcc.gnu.org/ml/gcc-help/2018-05/msg00053.html This patch checks whether
the compiler has the flag -print-multiarch in which case it uses it instead of
-dumpmachine to print the compiler target:
There were a few places that used an explicit test for i686 that are now also
checking for i386 as this is the value set in ARCH for gcc and clang when
configuring zlib-ng with --32.
Hans Johnson [Wed, 16 Jan 2019 16:37:30 +0000 (10:37 -0600)]
COMP: Fix missing header unistd.h
zlib-ng/gzlib.c:196:9: warning: implicit declaration of function 'lseek' is invalid in C99 [-Wimplicit-function-declaration]
LSEEK(state->fd, 0, SEEK_END); /* so gzoffset() is correct */
^
zlib-ng/gzlib.c:17:17: note: expanded from macro 'LSEEK'
^
[ 61%] Building C object CMakeFiles/zlibstatic.dir/gzread.c.o
zlib-ng/gzread.c:27:15: warning: implicit declaration of function 'read' is invalid in C99 [-Wimplicit-function-declaration]
ret = read(state->fd, buf + *have, len - *have);
^
zlib-ng/gzread.c:596:11: warning: implicit declaration of function 'close' is invalid in C99 [-Wimplicit-function-declaration]
ret = close(state->fd);
^
[ 62%] Building C object CMakeFiles/zlibstatic.dir/gzwrite.c.o
zlib-ng/gzwrite.c:84:15: warning: implicit declaration of function 'write' is invalid in C99 [-Wimplicit-function-declaration]
got = write(state->fd, strm->next_in, strm->avail_in);
^
zlib-ng/gzwrite.c:100:33: warning: implicit declaration of function 'write' is invalid in C99 [-Wimplicit-function-declaration]
if (have && ((got = write(state->fd, state->x.next, (unsigned long)have)) < 0 || (unsigned)got != have)) {
^
zlib-ng/gzwrite.c:512:9: warning: implicit declaration of function 'close' is invalid in C99 [-Wimplicit-function-declaration]
if (close(state->fd) == -1)"
Replace the UNROLL_LESS define with UNROLL_MORE, making UNROLL_LESS the default.
Performance benchmarks have so far not shown that any platform benefits from UNROLL_MORE,
although this might be beneficial on older compilers/cpus or for compiling without optimizations.
The extra UNROLL_MORE code should be considered for removal since it is never enabled by us
and will likely only serve to confuse and contribute to bitrot.
Sebastian Pop [Thu, 17 Jan 2019 16:32:25 +0000 (10:32 -0600)]
fix warning with configure --debug
arch/x86/deflate_quick.c:184:21: warning: format ‘%u’ expects argument of type ‘unsigned int’, but argument 5 has type ‘size_t {aka long unsigned int}’ [-Wformat=]
Hans Johnson [Wed, 16 Jan 2019 14:46:08 +0000 (08:46 -0600)]
ENH: Remove superflous setting for cmake 2.6.0+
As of CMake 2.6.0 the ELSE() and ENDIF() constructs can be empty. The
same is true for closing constructs on ENDMACRO(), ENDFUNCTION(), and
ENDFOREACH().
If you require 2.4.x compatibility, CMake 2.4.3 or greater
recognizes the CMAKE_ALLOW_LOOSE_LOOP_CONSTRUCTS option (which is
superfluous in 2.6.0)
Fix win32 makefiles to no longer attempt to build match.obj.
Both of these makefiles should get their dependency maps
regenerated, since they seem to be very outdated.
Sebastian Pop [Tue, 15 Jan 2019 21:51:06 +0000 (15:51 -0600)]
mark longest_match static inline
x86_64-linux compiled with clang and gcc at -O3 shows that inlining
longest_match() brings good speedup overall with speedups of up to 8%.
clang https://gist.github.com/sebpop/17d9c340cfaf16c1854017efefce6fc3
gcc https://gist.github.com/sebpop/451160356fc4e73541a1b8eef7dceb97
Sebastian Pop [Thu, 20 Dec 2018 20:08:16 +0000 (14:08 -0600)]
fix bug #263: hoist invariant loads
This patch speeds up longest_match when compiled with clang for x86_64 to match
the performance of zlib.org. The performance of zlib-ng does not change much
when compiled with gcc for x86_64, or when zlib-ng is cross compiled by gcc or
llvm for aarch64.
Mark Adler [Sat, 21 Jan 2017 09:50:26 +0000 (01:50 -0800)]
Limit hash table inserts after switch from stored deflate.
This limits hash table inserts to the available data in the window
and to the sliding window size in deflate_stored(). The hash table
inserts are deferred until deflateParams() switches to a non-zero
compression level.
Sebastian Pop [Fri, 21 Dec 2018 19:46:46 +0000 (13:46 -0600)]
fix configure for arm cross compilation
configure used to end with ARCH=x86_64 even when using a cross compiler
targeting arm. When using a compiler targeting aarch64 there was no problem
detecting a correct ARCH.
Mark Adler [Mon, 16 Jan 2017 17:49:35 +0000 (09:49 -0800)]
Permit a deflateParams() parameter change as soon as possible.
This commit allows a parameter change even if the input data has
not all been compressed and copied to the application output
buffer, so long as all of the input data has been compressed to
the internal pending output buffer. This also allows an immediate
deflateParams change so long as there have been no deflate calls
since initialization or reset.
Mark Adler [Sun, 5 Feb 2017 07:58:37 +0000 (23:58 -0800)]
Avoid a conversion error in gzseek when off_t type too small.
This is a problem in the odd case that the second argument of
LSEEK is a larger type than off_t. Apparently MinGW defines off_t
to be 32 bits, but _lseeki64 has a 64-bit second argument.
Also undo a previous commit to permit MinGW to use _lseeki64.
Sebastian Pop [Wed, 19 Dec 2018 17:30:25 +0000 (11:30 -0600)]
avoid double definitions for LITTLE_ENDIAN and BYTE_ORDER
When compiling with `cmake; make` the compiler used to warn about double
definitions:
../gzendian.h:11:0: warning: "LITTLE_ENDIAN" redefined
# define LITTLE_ENDIAN __ORDER_LITTLE_ENDIAN__
^
In file included from /usr/include/x86_64-linux-gnu/bits/string2.h:51:0,
from /usr/include/string.h:630,
from ../zutil.h:24,
from ../deflate.h:15,
from ../functable.h:9,
from ../functable.c:7:
/usr/include/endian.h:45:0: note: this is the location of the previous definition
# define LITTLE_ENDIAN __LITTLE_ENDIAN
^
In file included from ../deflate.h:16:0,
from ../functable.h:9,
from ../functable.c:7:
../gzendian.h:12:0: warning: "BYTE_ORDER" redefined
# define BYTE_ORDER LITTLE_ENDIAN
^
In file included from /usr/include/x86_64-linux-gnu/bits/string2.h:51:0,
from /usr/include/string.h:630,
from ../zutil.h:24,
from ../deflate.h:15,
from ../functable.h:9,
from ../functable.c:7:
/usr/include/endian.h:48:0: note: this is the location of the previous definition
# define BYTE_ORDER __BYTE_ORDER
^
This patch makes unzipping of files up to 1.2x faster on x86_64. The other part
(1.3x speedup) of the patch by Nigel Tao is unsafe as discussed in the review of
that pull request. zlib-ng already has a different way to optimize the memcpy
for that missing part.
The original patch was enabled only on little-endian machines. This patch adapts
the loading of 64 bits at a time to big endian machines.
Benchmarking notes from Hans Kristian Rosbach:
https://github.com/zlib-ng/zlib-ng/pull/224#issuecomment-444837182
Benchmark runs: 7, tested levels: 0-7, testfile 100M
So I see about a 5.4% speedup on my x86_64 machine, not quite the 1.2x speedup
but a nice speedup nevertheless. This benchmark measures the total execution
time of minigzip, so that might have caused some inefficiencies.
Mark Adler [Sun, 16 Apr 2017 15:35:33 +0000 (08:35 -0700)]
Handle case where inflateSync used when header never processed.
If zlib and/or gzip header processing was requested, but a header
was never provided and inflateSync was used successfully, then the
inflate state would be inconsistent, trying to compute a check
value but with no flags set. This commit sets the inflate mode to
raw in this case, since there is no other assumption that can be
made if a header was requested but never seen.
Sebastian Pop [Wed, 12 Dec 2018 15:35:43 +0000 (09:35 -0600)]
Zlib patch: prevent uninitialized use of state->check
This CL fixes a security bug in zlib. It was reported upstream long ago
and the testcase was shared upstream but it's yet unsolved. As a fix,
state->check is set to the same value as the adler32 of an empty string.
Upstream bug: madler/zlib#245
Bug: chromium:697481 https://crbug.com/697481
Reviewed-on: https://chromium-review.googlesource.com/601193 Reviewed-by: Tom Sepez <tsepez@chromium.org> Reviewed-by: Adam Langley <agl@chromium.org>
Commit-Queue: Nicolás Peña <npm@chromium.org>
Sebastian Pop [Thu, 6 Dec 2018 19:23:17 +0000 (13:23 -0600)]
return an index for hash map collisions in insert_string
The current version of insert_string_c and variations for sse2, arm, and aarch64
in zlib-ng has changed semantics from the original code of INSERT_STRING macro
in zlib:
The code of INSERT_STRING assigns match_head with the content of s->head[s->ins_h].
In zlib-ng, the assignment to match_head happens in the caller of insert_string().
zlib-ng's insert_string_*() functions return 0 instead of str+idx in case of
collision, i.e., when if (s->head[s->ins_h] == str+idx).
The effect of returning 0 instead of the content of s->head[s->ins_h] is that
the search for a longest_match through s->prev[] chains will be cut short when
arriving at 0. This leads to a shorter compression time at the expense of a
worse compression rate: returning 0 cuts out the search space.
With this patch:
Performance counter stats for './minigzip -9 llvm.tar':
Sebastian Pop [Wed, 26 Sep 2018 02:49:06 +0000 (21:49 -0500)]
fix bug #210: split statement in two to avoid substract overflow
make check used to fail with:
trees.c:482:53: runtime error: unsigned integer overflow: 6 - 7 cannot be represented in type 'unsigned int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior trees.c:482:53 in
Mika Lindqvist [Tue, 13 Mar 2018 09:26:19 +0000 (11:26 +0200)]
[ARM/AArch64] Add run-time detection of ACLE and NEON instructions under Linux.
* Use getauxval() to check support for ACLE CRC32 instructions
* Allow disabling CRC32 instruction check