]> git.ipfire.org Git - thirdparty/zlib-ng.git/log
thirdparty/zlib-ng.git
6 years agocleanup arm/adler32_neon.c code
Sebastian Pop [Mon, 1 Apr 2019 20:05:53 +0000 (15:05 -0500)] 
cleanup arm/adler32_neon.c code

6 years agoonly call NEON adler32 for more than 16 bytes
Sebastian Pop [Mon, 28 Jan 2019 22:05:50 +0000 (16:05 -0600)] 
only call NEON adler32 for more than 16 bytes

improves performance of inflate by up to 6% on an A-73 Hikey running at 2.36 GHz
when executing the chromium benchmark on the snappy data set.  In a few cases
inflate is slower by up to 0.8%.  Overall performance of inflate is better by
about 0.3%.

6 years agofix configure output when compiler is clang
Sebastian Pop [Mon, 1 Apr 2019 18:21:18 +0000 (13:21 -0500)] 
fix configure output when compiler is clang

The patch cleans up the expected errors that clutter the output of configure:

$ CC=clang ./configure
Checking for compiler... clang-6.0
clang: error: unknown argument: '-print-multiarch'
clang: error: no input files

With this patch the last two lines disappear.

6 years agofix configure --help text to match parsed options
Sebastian Pop [Wed, 3 Apr 2019 14:50:02 +0000 (09:50 -0500)] 
fix configure --help text to match parsed options

6 years agofix warning when compiling for AArch64
Sebastian Pop [Wed, 3 Apr 2019 15:00:08 +0000 (10:00 -0500)] 
fix warning when compiling for AArch64

gcc stopped warning about an unused function when I declared `arm_has_neon` as
`static inline`.  However clang still warns about an unused function:

```
arch/arm/armfeature.c:20:19: warning: unused function 'arm_has_neon' [-Wunused-function]
static inline int arm_has_neon()

                  ^
1 warning generated.
```

This patch moves the function under a preprocessor `#if !defined(__aarch64)`
which makes clang stop warn about it.

6 years agoFix building with gcc 8.2.1 and -Wall -Wextra -pedantic -Werror
Ilya Leoshkevich [Tue, 26 Mar 2019 09:52:02 +0000 (10:52 +0100)] 
Fix building with gcc 8.2.1 and -Wall -Wextra -pedantic -Werror

* ptrdiff_t check always failed because of unused parameter
* sizeof(void *) check always failed because of double semicolon
* Sign issue in nice_match assignment
* dist parameter of set_bytes may be unused
* Parameters of main may be unused in test/example.c
* snprintf requires a _POSIX_C_SOURCE #define in test/minigzip.c,
  because a _POSIX_SOURCE #define is present

6 years agoIncrease verbosity required to warn about bit length overflow.
Mark Adler [Sun, 20 Nov 2016 19:36:15 +0000 (11:36 -0800)] 
Increase verbosity required to warn about bit length overflow.

When debugging the Huffman coding would warn about resulting codes
greater than 15 bits in length. This is handled properly, and is
not uncommon. This increases the verbosity of the warning by one,
so that it is not displayed by default.

6 years agoFix endianness detection in memcopy.h
Ilya Leoshkevich [Tue, 26 Mar 2019 12:06:36 +0000 (13:06 +0100)] 
Fix endianness detection in memcopy.h

When memcopy.h is included into inffast.c, endianness-related
preperocessor defines are not set, which leads to
BYTE_ORDER == LITTLE_ENDIAN condition being always true. This breaks
decompression at least on s390x.

6 years agofix oss-fuzz/13863
Sebastian Pop [Tue, 26 Mar 2019 16:59:45 +0000 (11:59 -0500)] 
fix oss-fuzz/13863

The oss fuzzers started failing with the following assert
```
ASSERT: 0 == memcmp(data + offset, buf, len)
```
after the following patch has been pulled in the tree:

```
commit 20ca64fa5d2d8a7421ed86b68709ef971dcfbddf
Author: Sebastian Pop <s.pop@samsung.com>
Date:   Wed Mar 6 14:16:20 2019 -0600

    define and use chunkmemset instead of byte_memset for INFFAST_CHUNKSIZE
```

The function chunkcopysafe is assuming that the input `len` is less than 16 bytes:
```
    if ((safe - out) < (ptrdiff_t)INFFAST_CHUNKSIZE) {
```
but we were called with `len = 22` because `safe` was defined too small:

```
-    safe = out + (strm->avail_out - INFFAST_CHUNKSIZE);
```
and the difference `safe - out` was 16 bytes smaller than the actual `len`.
The patch fixes the initialization of `safe` to:
```
+    safe = out + strm->avail_out;
```

6 years agouse sizeof(type) instead of magic numbers
Sebastian Pop [Tue, 5 Mar 2019 19:36:25 +0000 (19:36 +0000)] 
use sizeof(type) instead of magic numbers

6 years agoautomatically update makefile dependences
Sebastian Pop [Sat, 26 Jan 2019 21:54:59 +0000 (15:54 -0600)] 
automatically update makefile dependences

6 years agofix all ASan errors on arm/aarch64
Sebastian Pop [Tue, 5 Mar 2019 19:15:17 +0000 (19:15 +0000)] 
fix all ASan errors on arm/aarch64

6 years agofactor out code in arch/{arm,aarch64}
Sebastian Pop [Tue, 5 Mar 2019 15:57:05 +0000 (09:57 -0600)] 
factor out code in arch/{arm,aarch64}

6 years agofix UBSan warnings of unsigned overflow
Sebastian Pop [Wed, 6 Mar 2019 20:33:49 +0000 (14:33 -0600)] 
fix UBSan warnings of unsigned overflow

6 years agochunk_memcpy is not needed under INFFAST_CHUNKSIZE
Sebastian Pop [Wed, 6 Mar 2019 20:32:14 +0000 (14:32 -0600)] 
chunk_memcpy is not needed under INFFAST_CHUNKSIZE

6 years agouse chunkcopysafe and chunkmemsetsafe under INFFAST_CHUNKSIZE from inflate
Sebastian Pop [Wed, 6 Mar 2019 20:20:04 +0000 (14:20 -0600)] 
use chunkcopysafe and chunkmemsetsafe under INFFAST_CHUNKSIZE from inflate

6 years agodefine and use chunkmemset instead of byte_memset for INFFAST_CHUNKSIZE
Sebastian Pop [Wed, 6 Mar 2019 20:16:20 +0000 (14:16 -0600)] 
define and use chunkmemset instead of byte_memset for INFFAST_CHUNKSIZE

6 years agofactor out code
Sebastian Pop [Wed, 6 Mar 2019 20:00:30 +0000 (14:00 -0600)] 
factor out code

6 years agoenable INFFAST_CHUNKSIZE on x86
Sebastian Pop [Fri, 1 Mar 2019 15:48:58 +0000 (09:48 -0600)] 
enable INFFAST_CHUNKSIZE on x86

6 years agofix indent for pre-processor
Sebastian Pop [Wed, 6 Mar 2019 19:37:18 +0000 (13:37 -0600)] 
fix indent for pre-processor

6 years agomove INFFAST_CHUCKSIZE code to memcopy.h
Sebastian Pop [Fri, 1 Mar 2019 15:45:33 +0000 (09:45 -0600)] 
move INFFAST_CHUCKSIZE code to memcopy.h

6 years agofix typo: deflate is compression
Sebastian Pop [Wed, 6 Mar 2019 19:16:06 +0000 (13:16 -0600)] 
fix typo: deflate is compression

6 years agoUpdate x86 and x86_64 arch checks to use the recommended
Hans Kristian Rosbach [Wed, 6 Mar 2019 09:59:06 +0000 (10:59 +0100)] 
Update x86 and x86_64 arch checks to use the recommended
define names, resulting in improved compiler support.
Based on the overviews from several sites, such as:
http://nadeausoftware.com/articles/2012/02/c_c_tip_how_detect_processor_type_using_compiler_predefined_macros

6 years agoDefault sse2 to be available on x86_64 arch.
Hans Kristian Rosbach [Wed, 6 Mar 2019 09:39:32 +0000 (10:39 +0100)] 
Default sse2 to be available on x86_64 arch.
Allow x86 arch to force sse2 availability.
It still depends on sse2 intrinsics being detected and optimizations enabled.

6 years agoremove MEMSET, replace with memset
Sebastian Pop [Wed, 6 Mar 2019 15:19:43 +0000 (15:19 +0000)] 
remove MEMSET, replace with memset

6 years agoremove MEMCPY, replace with memcpy
Sebastian Pop [Tue, 5 Mar 2019 18:27:05 +0000 (18:27 +0000)] 
remove MEMCPY, replace with memcpy

6 years agounify uses of __ARM_FEATURE_CRC32
Sebastian Pop [Tue, 5 Mar 2019 15:28:27 +0000 (09:28 -0600)] 
unify uses of __ARM_FEATURE_CRC32

6 years agounify uses of __ARM_NEON__ and __ARM_NEON
Sebastian Pop [Tue, 5 Mar 2019 15:47:03 +0000 (09:47 -0600)] 
unify uses of __ARM_NEON__ and __ARM_NEON

6 years agofactor out common cmake code for x86_64 and i386
Sebastian Pop [Tue, 5 Mar 2019 15:08:02 +0000 (09:08 -0600)] 
factor out common cmake code for x86_64 and i386

6 years agorename X86_SSE2_FILL_WINDOW to X86_SSE2
Sebastian Pop [Fri, 1 Mar 2019 14:48:38 +0000 (08:48 -0600)] 
rename X86_SSE2_FILL_WINDOW to X86_SSE2

6 years agoremove unused variable NM
Sebastian Pop [Fri, 8 Feb 2019 20:46:48 +0000 (14:46 -0600)] 
remove unused variable NM

6 years agoFixed arithmetic overflow warnings on Windows (#302)
Nathan Moinvaziri [Fri, 1 Mar 2019 10:44:37 +0000 (02:44 -0800)] 
Fixed arithmetic overflow warnings on Windows (#302)

Fixed arithmetic overflow warnings in MSVC.
Fixed uint64_t to uint32_t casting warning.
Added assert to check if bits is greater than 32 before cast.

6 years agowin32: update makefile dependences
Sebastian Pop [Sat, 26 Jan 2019 20:37:07 +0000 (14:37 -0600)] 
win32: update makefile dependences

6 years agowin32: add missing makefile dependence
Sebastian Pop [Sat, 26 Jan 2019 20:06:04 +0000 (14:06 -0600)] 
win32: add missing makefile dependence

6 years agoARM: check cpu feature once at init time
Sebastian Pop [Fri, 25 Jan 2019 17:44:46 +0000 (11:44 -0600)] 
ARM: check cpu feature once at init time

This makes the checks for arm cpu features as inexpensive as on the x86 side
by calling the runtime feature detection once in deflate/inflate init and then
storing the result in a global variable.

6 years agofix auto-detection of cross compilation for bare-metal linaro compilers
Sebastian Pop [Fri, 8 Feb 2019 20:41:07 +0000 (14:41 -0600)] 
fix auto-detection of cross compilation for bare-metal linaro compilers

6 years agoARM: enable neon and acle when available
Sebastian Pop [Wed, 23 Jan 2019 20:17:49 +0000 (14:17 -0600)] 
ARM: enable neon and acle when available

this patch changes the default for cmake and configure to enable neon and acle
when no flags have been specified.  This mimics the default for x86.  The flags
--neon and --acle are changed to the opposite --without-neon and --without-acle
allowing the user to disable detection of neon and acle.

6 years agocleanup: remove unused file
Sebastian Pop [Thu, 24 Jan 2019 18:26:55 +0000 (12:26 -0600)] 
cleanup: remove unused file

6 years agocleanup: move code from arch/x86/crc_pclmulqdq.c to crc32.c
Sebastian Pop [Wed, 23 Jan 2019 19:15:33 +0000 (13:15 -0600)] 
cleanup: move code from arch/x86/crc_pclmulqdq.c to crc32.c

6 years agocleanup: remove partial do { } while constructs from macros
Sebastian Pop [Wed, 23 Jan 2019 17:30:05 +0000 (11:30 -0600)] 
cleanup: remove partial do { } while constructs from macros

also remove dead code that was there only to close the open braces of the
do-while construct.

6 years agoENH: Use modern cmake variable conventions
Hans Johnson [Wed, 16 Jan 2019 16:11:00 +0000 (10:11 -0600)] 
ENH: Use modern cmake variable conventions

Use recommended naming conventions for variables

See https://cmake.org/cmake/help/v3.0/command/project.html?highlight=project for
the common conventions.

The project() command stores the version number and its components in variables

PROJECT_VERSION, <PROJECT-NAME>_VERSION
PROJECT_VERSION_MAJOR, <PROJECT-NAME>_VERSION_MAJOR
PROJECT_VERSION_MINOR, <PROJECT-NAME>_VERSION_MINOR
PROJECT_VERSION_PATCH, <PROJECT-NAME>_VERSION_PATCH
PROJECT_VERSION_TWEAK, <PROJECT-NAME>_VERSION_TWEAK

6 years agoENH: Conistently read version information from one place
Hans Johnson [Wed, 16 Jan 2019 16:07:14 +0000 (10:07 -0600)] 
ENH: Conistently read version information from one place

Apply "one-definition-rule" getting the version information.

The zlib.h file is the definitive source for version
information.  Add code to extract version information
for both ZLIB and ZLIBNG variants from the zlib.h file.

6 years agoENH: Use modern (cmake 3.0+) project signature
Hans Johnson [Wed, 16 Jan 2019 15:48:04 +0000 (09:48 -0600)] 
ENH: Use modern (cmake 3.0+) project signature

Set a name, version, and enable languages for the entire project.
This signature uses a common paradigm for propogating defaults
for setting version information in several targets.

6 years agoENH: Allow setting the C_STANDARD version from command line
Hans Johnson [Wed, 16 Jan 2019 15:41:53 +0000 (09:41 -0600)] 
ENH: Allow setting the C_STANDARD version from command line

Allow for both C99 and C11 standards to be used. By default,
do not support compiler extensions, but allow this setting to
be overridden by the developer.

The setting of the language standard needs to be performed before
the "project" directive.

6 years agoBUG: CMake 2.8.4 does not support required features
Hans Johnson [Wed, 16 Jan 2019 15:13:36 +0000 (09:13 -0600)] 
BUG:  CMake 2.8.4 does not support required features

CMake Error at CMakeLists.txt:682 (target_include_directories):
  Unknown CMake command "target_include_directories".

target_include_directories was introduced in cmake 3

C_STANDARD 99 support was added in cmake 3.1

Ubuntu 16.04 (Xenial) was distributed with cmake 3.5.1 by default.

CMake versions > 3.3 allow simplified implementations of modern
cmake compilation support.

===
For newer versions of cmake (upto a maximum validated version [3.13.2 in this case]),
use newer cmake policies available.  The newer policies often provide better
diagnostics for subtle build related issues.

6 years agofix bug #289: use strcpy instead of strncpy 294/head
Sebastian Pop [Tue, 22 Jan 2019 19:46:34 +0000 (13:46 -0600)] 
fix bug #289: use strcpy instead of strncpy

to avoid a GCC 8 warning:

test/example.c:465:48: warning: argument to ‘sizeof’ in ‘strncpy’ call is the same expression as the source; did you mean to use the size of the destination? [-Wsizeof-pointer-memaccess]
     strncpy((char*)uncompr, garbage_str, sizeof(garbage_str));

6 years agofix bug #208: let the compiler generate code for unaligned stores
Sebastian Pop [Thu, 17 Jan 2019 19:07:16 +0000 (13:07 -0600)] 
fix bug #208: let the compiler generate code for unaligned stores

to avoid this error:
zlib-ng/arch/x86/deflate_quick.c:154:5: runtime error: store to misaligned address 0x631000014801 for type 'unsigned int', which requires 4 byte alignment
0x631000014801: note: pointer points here
00 80 02  d3 07 00 00 be be be be  be be be be be be be be  be be be be be be be be  be be be be be
               ^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior zlib-ng/arch/x86/deflate_quick.c:154:5 in

6 years agofix bug #225: account bits sent
Sebastian Pop [Thu, 17 Jan 2019 18:26:30 +0000 (12:26 -0600)] 
fix bug #225: account bits sent

6 years agoclean up logic of configure and make for sanitizer flags
Sebastian Pop [Wed, 16 Jan 2019 21:37:10 +0000 (15:37 -0600)] 
clean up logic of configure and make for sanitizer flags

6 years agocleanup configure output for checking the compiler
Sebastian Pop [Tue, 22 Jan 2019 19:05:53 +0000 (13:05 -0600)] 
cleanup configure output for checking the compiler

6 years agocleanup configure output by using echo -n
Sebastian Pop [Tue, 22 Jan 2019 18:32:40 +0000 (12:32 -0600)] 
cleanup configure output by using echo -n

6 years agoonly check for -mfpu=neon on ARM processors
Sebastian Pop [Tue, 22 Jan 2019 18:24:30 +0000 (12:24 -0600)] 
only check for -mfpu=neon on ARM processors

6 years agocleanup multiple test -o expressions by using case
Sebastian Pop [Tue, 22 Jan 2019 18:18:56 +0000 (12:18 -0600)] 
cleanup multiple test -o expressions by using case

6 years agoThis patch fixes a problem with detecting the target that gcc compiles for when
Sebastian Pop [Wed, 16 Jan 2019 16:24:37 +0000 (10:24 -0600)] 
This patch fixes a problem with detecting the target that gcc compiles for when
using -m32.  The error is in the way we detect GCC_ARCH in configure:

GCC_ARCH=`$CC $CFLAGS -dumpmachine | sed 's/-.*//g'`

Here is the output of gcc and clang:

$ gcc  -m32 -std=c99 -dumpmachine
x86_64-linux-gnu

$ clang  -m32 -std=c99 -dumpmachine
i386-pc-linux-gnu

It looks like it is a known problem:
https://gcc.gnu.org/ml/gcc-help/2018-05/msg00053.html This patch checks whether
the compiler has the flag -print-multiarch in which case it uses it instead of
-dumpmachine to print the compiler target:

$ gcc -print-multiarch
x86_64-linux-gnu
$ gcc -print-multiarch -m32
i386-linux-gnu
$ clang -print-multiarch
clang: error: unknown argument: '-print-multiarch'

There were a few places that used an explicit test for i686 that are now also
checking for i386 as this is the value set in ARCH for gcc and clang when
configuring zlib-ng with --32.

6 years agoremove -DX86
Sebastian Pop [Wed, 2 Jan 2019 21:12:57 +0000 (15:12 -0600)] 
remove -DX86

6 years agoremove unused compilation flag -DX86_64
Sebastian Pop [Wed, 2 Jan 2019 20:33:49 +0000 (14:33 -0600)] 
remove unused compilation flag -DX86_64

6 years agobug #266: select std2 for gcc/clang ARM 32-bit builds
Brian Rzycki [Thu, 17 Jan 2019 22:19:27 +0000 (16:19 -0600)] 
bug #266: select std2 for gcc/clang ARM 32-bit builds

Running the Chromium zlib benchmark tool on a Hikey 970 device
showed uplifts for both gcc and clang when using std2 (larger is
better):

arm_h02_std3_gcc_0ca47588bd2e38/score.out  =  7.964881
arm_h02_std3_llvm_2e8efbb084ae87/score.out =  9.522024
arm_h02_std1_gcc_0ca47588bd2e38/score.out  =  9.641071
arm_h02_std2_gcc_0ca47588bd2e38/score.out  =  9.919345
arm_h02_std1_llvm_2e8efbb084ae87/score.out =  9.925893
arm_h02_std2_llvm_2e8efbb084ae87/score.out = 10.049107

6 years agoremove unused variable
Sebastian Pop [Thu, 17 Jan 2019 16:25:40 +0000 (10:25 -0600)] 
remove unused variable

6 years agoremove unused variables
Sebastian Pop [Thu, 17 Jan 2019 16:25:19 +0000 (10:25 -0600)] 
remove unused variables

6 years agoCOMP: Fix missing header unistd.h
Hans Johnson [Wed, 16 Jan 2019 16:37:30 +0000 (10:37 -0600)] 
COMP: Fix missing header unistd.h

zlib-ng/gzlib.c:196:9: warning: implicit declaration of function 'lseek' is invalid in C99 [-Wimplicit-function-declaration]
        LSEEK(state->fd, 0, SEEK_END);  /* so gzoffset() is correct */
        ^
zlib-ng/gzlib.c:17:17: note: expanded from macro 'LSEEK'
                ^
[ 61%] Building C object CMakeFiles/zlibstatic.dir/gzread.c.o
zlib-ng/gzread.c:27:15: warning: implicit declaration of function 'read' is invalid in C99 [-Wimplicit-function-declaration]
        ret = read(state->fd, buf + *have, len - *have);
              ^
zlib-ng/gzread.c:596:11: warning: implicit declaration of function 'close' is invalid in C99 [-Wimplicit-function-declaration]
    ret = close(state->fd);
          ^
[ 62%] Building C object CMakeFiles/zlibstatic.dir/gzwrite.c.o
zlib-ng/gzwrite.c:84:15: warning: implicit declaration of function 'write' is invalid in C99 [-Wimplicit-function-declaration]
        got = write(state->fd, strm->next_in, strm->avail_in);
              ^
zlib-ng/gzwrite.c:100:33: warning: implicit declaration of function 'write' is invalid in C99 [-Wimplicit-function-declaration]
            if (have && ((got = write(state->fd, state->x.next, (unsigned long)have)) < 0 || (unsigned)got != have)) {
                                ^
zlib-ng/gzwrite.c:512:9: warning: implicit declaration of function 'close' is invalid in C99 [-Wimplicit-function-declaration]
    if (close(state->fd) == -1)"

6 years agoLet deflate_medium be enabled by default.
Hans Kristian Rosbach [Wed, 16 Jan 2019 11:35:34 +0000 (12:35 +0100)] 
Let deflate_medium be enabled by default.

6 years agoReplace the UNROLL_LESS define with UNROLL_MORE, making UNROLL_LESS the default.
Hans Kristian Rosbach [Wed, 16 Jan 2019 10:49:29 +0000 (11:49 +0100)] 
Replace the UNROLL_LESS define with UNROLL_MORE, making UNROLL_LESS the default.
Performance benchmarks have so far not shown that any platform benefits from UNROLL_MORE,
although this might be beneficial on older compilers/cpus or for compiling without optimizations.

The extra UNROLL_MORE code should be considered for removal since it is never enabled by us
and will likely only serve to confuse and contribute to bitrot.

6 years agofix warning with configure --debug
Sebastian Pop [Thu, 17 Jan 2019 16:32:25 +0000 (10:32 -0600)] 
fix warning with configure --debug

arch/x86/deflate_quick.c:184:21: warning: format ‘%u’ expects argument of type ‘unsigned int’, but argument 5 has type ‘size_t {aka long unsigned int}’ [-Wformat=]

6 years agoENH: Remove superflous setting for cmake 2.6.0+
Hans Johnson [Wed, 16 Jan 2019 14:46:08 +0000 (08:46 -0600)] 
ENH: Remove superflous setting for cmake 2.6.0+

As of CMake 2.6.0 the ELSE() and ENDIF() constructs can be empty. The
same is true for closing constructs on ENDMACRO(), ENDFUNCTION(), and
ENDFOREACH().

If you require 2.4.x compatibility, CMake 2.4.3 or greater
recognizes the CMAKE_ALLOW_LOOSE_LOOP_CONSTRUCTS option (which is
superfluous in 2.6.0)

6 years agotypedef ptrdiff_t when stddef.h does not provide it
Sebastian Pop [Thu, 13 Dec 2018 15:58:08 +0000 (09:58 -0600)] 
typedef ptrdiff_t when stddef.h does not provide it

6 years agoFix win32 makefiles to no longer attempt to build match.obj.
Hans Kristian Rosbach [Wed, 16 Jan 2019 11:00:38 +0000 (12:00 +0100)] 
Fix win32 makefiles to no longer attempt to build match.obj.
Both of these makefiles should get their dependency maps
regenerated, since they seem to be very outdated.

6 years agomark longest_match static inline
Sebastian Pop [Tue, 15 Jan 2019 21:51:06 +0000 (15:51 -0600)] 
mark longest_match static inline

x86_64-linux compiled with clang and gcc at -O3 shows that inlining
longest_match() brings good speedup overall with speedups of up to 8%.
clang https://gist.github.com/sebpop/17d9c340cfaf16c1854017efefce6fc3
gcc https://gist.github.com/sebpop/451160356fc4e73541a1b8eef7dceb97

performance is also better on aarch64-linux with speedups up to 3.5%.
clang https://gist.github.com/sebpop/aec3e068a61c1daddf9c72ab4bc33281
gcc https://gist.github.com/sebpop/f0d4e6bfd58d95b313c955b5d747a74f

6 years agomove match.c to match_p.h and remove match.h
Sebastian Pop [Tue, 15 Jan 2019 21:49:57 +0000 (15:49 -0600)] 
move match.c to match_p.h and remove match.h

6 years agoAdd scan-build cmake build to build matrix
Gabriel A. Devenyi [Tue, 11 Dec 2018 18:32:09 +0000 (13:32 -0500)] 
Add scan-build cmake build to build matrix

6 years agoUpdate travis to xenial
Gabriel A. Devenyi [Tue, 11 Dec 2018 18:26:50 +0000 (13:26 -0500)] 
Update travis to xenial

6 years agofix bug #263: hoist invariant loads
Sebastian Pop [Thu, 20 Dec 2018 20:08:16 +0000 (14:08 -0600)] 
fix bug #263: hoist invariant loads

This patch speeds up longest_match when compiled with clang for x86_64 to match
the performance of zlib.org. The performance of zlib-ng does not change much
when compiled with gcc for x86_64, or when zlib-ng is cross compiled by gcc or
llvm for aarch64.

Performance analysis for this patch:
https://gist.github.com/sebpop/7cdc5b2abd3297c5fcd2a8315bd9a209

6 years agoLimit hash table inserts after switch from stored deflate.
Mark Adler [Sat, 21 Jan 2017 09:50:26 +0000 (01:50 -0800)] 
Limit hash table inserts after switch from stored deflate.

This limits hash table inserts to the available data in the window
and to the sliding window size in deflate_stored(). The hash table
inserts are deferred until deflateParams() switches to a non-zero
compression level.

6 years agoSmall speedup to inflate [psumbera].
Mark Adler [Sun, 12 Feb 2017 07:21:41 +0000 (23:21 -0800)] 
Small speedup to inflate [psumbera].

Seeing a few percent speedup by using a pointer instead of an
assigned structure. This seems to help the compiler to optimize
better.

6 years agofix configure for arm cross compilation
Sebastian Pop [Fri, 21 Dec 2018 19:46:46 +0000 (13:46 -0600)] 
fix configure for arm cross compilation

configure used to end with ARCH=x86_64 even when using a cross compiler
targeting arm. When using a compiler targeting aarch64 there was no problem
detecting a correct ARCH.

6 years agoPermit a deflateParams() parameter change as soon as possible.
Mark Adler [Mon, 16 Jan 2017 17:49:35 +0000 (09:49 -0800)] 
Permit a deflateParams() parameter change as soon as possible.

This commit allows a parameter change even if the input data has
not all been compressed and copied to the application output
buffer, so long as all of the input data has been compressed to
the internal pending output buffer. This also allows an immediate
deflateParams change so long as there have been no deflate calls
since initialization or reset.

6 years agoFix bug when window full in deflate_stored().
Mark Adler [Sat, 21 Jan 2017 20:13:25 +0000 (12:13 -0800)] 
Fix bug when window full in deflate_stored().

6 years agoFix CLEAR_HASH macro to be usable as a single statement.
Mark Adler [Mon, 23 Jan 2017 07:38:52 +0000 (23:38 -0800)] 
Fix CLEAR_HASH macro to be usable as a single statement.

As it is used in deflateParams().

6 years agoAvoid a conversion error in gzseek when off_t type too small.
Mark Adler [Sun, 5 Feb 2017 07:58:37 +0000 (23:58 -0800)] 
Avoid a conversion error in gzseek when off_t type too small.

This is a problem in the odd case that the second argument of
LSEEK is a larger type than off_t. Apparently MinGW defines off_t
to be 32 bits, but _lseeki64 has a 64-bit second argument.

Also undo a previous commit to permit MinGW to use _lseeki64.

6 years agoHave Makefile return non-zero error code on test failure.
Mark Adler [Sun, 12 Feb 2017 06:38:09 +0000 (22:38 -0800)] 
Have Makefile return non-zero error code on test failure.

6 years agoAvoid some conversion warnings in gzread.c and gzwrite.c.
Mark Adler [Sun, 12 Feb 2017 06:45:27 +0000 (22:45 -0800)] 
Avoid some conversion warnings in gzread.c and gzwrite.c.

6 years agoReturn an error if the gzputs string length can't fit in an int.
Mark Adler [Sun, 12 Feb 2017 07:54:17 +0000 (23:54 -0800)] 
Return an error if the gzputs string length can't fit in an int.

6 years agoDon't compute check value for raw inflate if asked to validate.
Mark Adler [Thu, 30 Mar 2017 21:48:43 +0000 (14:48 -0700)] 
Don't compute check value for raw inflate if asked to validate.

6 years agofix several memory sanitizer errors on aarch64
Sebastian Pop [Mon, 17 Dec 2018 17:13:17 +0000 (11:13 -0600)] 
fix several memory sanitizer errors on aarch64

With this patch we have a clean run of make test on aarch64 when zlib-ng is
configured with --with-msan --with-fuzzers.

6 years agoremove unused configure check for HAVE_STDARG_H
Sebastian Pop [Thu, 13 Dec 2018 16:04:58 +0000 (10:04 -0600)] 
remove unused configure check for HAVE_STDARG_H

6 years agoFix compilation with --solo and --debug combined.
Mark Adler [Tue, 3 Jan 2017 02:18:31 +0000 (18:18 -0800)] 
Fix compilation with --solo and --debug combined.

However this ends up not really being solo, since it has to
include external libraries.

6 years agoavoid double definitions for LITTLE_ENDIAN and BYTE_ORDER
Sebastian Pop [Wed, 19 Dec 2018 17:30:25 +0000 (11:30 -0600)] 
avoid double definitions for LITTLE_ENDIAN and BYTE_ORDER

When compiling with `cmake; make` the compiler used to warn about double
definitions:

../gzendian.h:11:0: warning: "LITTLE_ENDIAN" redefined
 #  define LITTLE_ENDIAN __ORDER_LITTLE_ENDIAN__
 ^
In file included from /usr/include/x86_64-linux-gnu/bits/string2.h:51:0,
                 from /usr/include/string.h:630,
                 from ../zutil.h:24,
                 from ../deflate.h:15,
                 from ../functable.h:9,
                 from ../functable.c:7:
/usr/include/endian.h:45:0: note: this is the location of the previous definition
 # define LITTLE_ENDIAN __LITTLE_ENDIAN
 ^
In file included from ../deflate.h:16:0,
                 from ../functable.h:9,
                 from ../functable.c:7:
../gzendian.h:12:0: warning: "BYTE_ORDER" redefined
 #  define BYTE_ORDER LITTLE_ENDIAN
 ^
In file included from /usr/include/x86_64-linux-gnu/bits/string2.h:51:0,
                 from /usr/include/string.h:630,
                 from ../zutil.h:24,
                 from ../deflate.h:15,
                 from ../functable.h:9,
                 from ../functable.c:7:
/usr/include/endian.h:48:0: note: this is the location of the previous definition
 # define BYTE_ORDER __BYTE_ORDER
 ^

6 years agoCygwin does not have _wopen(), so do not create gzopen_w() there.
Mark Adler [Mon, 16 Jan 2017 17:38:36 +0000 (09:38 -0800)] 
Cygwin does not have _wopen(), so do not create gzopen_w() there.

6 years agoremove `unaligned store` UBsan warnings
Sebastian Pop [Fri, 7 Dec 2018 17:58:18 +0000 (11:58 -0600)] 
remove `unaligned store` UBsan warnings

This patch addresses several warnings from `make test` when
zlib-ng was configured -with-fuzzers -with-sanitizers:

zlib-ng/trees.c:798:5: runtime error: store to misaligned address 0x63100125c801 for type 'uint16_t' (aka 'unsigned short'), which requires 2 byte alignment
0x63100125c801: note: pointer points here
 00 80 76  01 8b 08 00 00 00 00 00  00 03 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00
              ^
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior zlib-ng/trees.c:798:5 in
zlib-ng/trees.c:799:5: runtime error: store to misaligned address 0x63100125c803 for type 'uint16_t' (aka 'unsigned short'), which requires 2 byte alignment
0x63100125c803: note: pointer points here
 76  01 f5 08 00 00 00 00 00  00 03 00 00 00 00 00 00  00 00 00 00 00 00 00 00  00 00 00 00 00 00 00
              ^

Instead of using `*(uint16_t*) foo = bar` to write a uint16_t, call
__builtin_memcpy which will be safe in case of memory page boundaries.

Without the patch:

 Performance counter stats for './minigzip -9 llvm.tar':

      13173.840115      task-clock (msec)         #    1.000 CPUs utilized
                27      context-switches          #    0.002 K/sec
                 0      cpu-migrations            #    0.000 K/sec
               129      page-faults               #    0.010 K/sec
    57,801,072,298      cycles                    #    4.388 GHz
   <not supported>      stalled-cycles-frontend
   <not supported>      stalled-cycles-backend
    75,270,723,557      instructions              #    1.30  insns per cycle
    17,797,368,302      branches                  # 1350.963 M/sec
       196,795,107      branch-misses             #    1.11% of all branches

      13.177897531 seconds time elapsed

45408 -rw-rw-r-- 1 spop spop 46493896 Dec 11 14:45 llvm.tar.gz

With remove-unaligned-stores patch:

      13184.736536      task-clock (msec)         #    1.000 CPUs utilized
                44      context-switches          #    0.003 K/sec
                 1      cpu-migrations            #    0.000 K/sec
               129      page-faults               #    0.010 K/sec
    57,882,724,316      cycles                    #    4.390 GHz
   <not supported>      stalled-cycles-frontend
   <not supported>      stalled-cycles-backend
    75,235,920,853      instructions              #    1.30  insns per cycle
    17,826,873,999      branches                  # 1352.084 M/sec
       196,050,096      branch-misses             #    1.10% of all branches

      13.185868238 seconds time elapsed

45408 -rw-rw-r-- 1 spop spop 46493896 Dec 11 14:46 llvm.tar.gz

6 years agobug #117: speed up inflate_fast
Sebastian Pop [Wed, 7 Nov 2018 21:11:27 +0000 (15:11 -0600)] 
bug #117: speed up inflate_fast

Based on a patch by Nigel Tao:
https://github.com/madler/zlib/pull/292/commits/e0ff1f330cc03ee04843f857869b4036593ab39d

This patch makes unzipping of files up to 1.2x faster on x86_64.  The other part
(1.3x speedup) of the patch by Nigel Tao is unsafe as discussed in the review of
that pull request.  zlib-ng already has a different way to optimize the memcpy
for that missing part.

The original patch was enabled only on little-endian machines. This patch adapts
the loading of 64 bits at a time to big endian machines.

Benchmarking notes from Hans Kristian Rosbach:
https://github.com/zlib-ng/zlib-ng/pull/224#issuecomment-444837182

Benchmark runs: 7, tested levels: 0-7, testfile 100M

develop at 796ad10 with -O3:
Level  Comp   Comptime min/avg/max  Decomptime min/avg/max
0     100.02%       0.01/0.01/0.02          0.08/0.09/0.11
1      47.08%       0.49/0.50/0.51          0.37/0.39/0.40
2      36.02%       1.10/1.12/1.13          0.39/0.39/0.40
3      34.77%       1.32/1.34/1.37          0.38/0.38/0.38
4      33.41%       1.50/1.53/1.56          0.37/0.37/0.38
5      33.07%       1.85/1.87/1.90          0.36/0.37/0.38
6      32.83%       2.54/2.57/2.61          0.36/0.37/0.38
avg    45.31%                 1.28                   0.34
tot                          62.60                  16.58

PR224 with -O3:
Level  Comp   Comptime min/avg/max  Decomptime min/avg/max
0     100.02%       0.01/0.01/0.02          0.09/0.09/0.10
1      47.08%       0.49/0.50/0.51          0.37/0.37/0.38
2      36.02%       1.09/1.11/1.13          0.38/0.38/0.39
3      34.77%       1.32/1.34/1.38          0.35/0.36/0.38
4      33.41%       1.49/1.52/1.54          0.36/0.36/0.37
5      33.07%       1.85/1.88/1.93          0.35/0.36/0.37
6      32.83%       2.55/2.58/2.65          0.35/0.35/0.36
avg    45.31%                 1.28                   0.33
tot                          62.48                  16.02

So I see about a 5.4% speedup on my x86_64 machine, not quite the 1.2x speedup
but a nice speedup nevertheless. This benchmark measures the total execution
time of minigzip, so that might have caused some inefficiencies.

At -O2, I only see a 2.7% speedup.

6 years agoEmphasize the need to continue decompressing gzip members.
Sebastian Pop [Thu, 13 Dec 2018 15:20:57 +0000 (09:20 -0600)] 
Emphasize the need to continue decompressing gzip members.

Also in zlib-ng.h.

6 years agoAdd Buildkite status badge
Hans Kristian Rosbach [Thu, 13 Dec 2018 14:00:56 +0000 (15:00 +0100)] 
Add Buildkite status badge

6 years agoFix Travis badge
Hans Kristian Rosbach [Thu, 13 Dec 2018 13:56:41 +0000 (14:56 +0100)] 
Fix Travis badge
Update readme file

6 years agoAvoid undefined behaviors of memcpy() in gz*printf().
Mark Adler [Fri, 13 Oct 2017 02:34:51 +0000 (19:34 -0700)] 
Avoid undefined behaviors of memcpy() in gz*printf().

6 years agoMake the names in functions declarations identical to definitions.
Mark Adler [Fri, 13 Oct 2017 03:03:51 +0000 (20:03 -0700)] 
Make the names in functions declarations identical to definitions.

6 years agoAssure that the number of bits for deflatePrime() is valid.
Mark Adler [Wed, 18 Apr 2018 05:44:41 +0000 (22:44 -0700)] 
Assure that the number of bits for deflatePrime() is valid.

6 years agoHandle case where inflateSync used when header never processed.
Mark Adler [Sun, 16 Apr 2017 15:35:33 +0000 (08:35 -0700)] 
Handle case where inflateSync used when header never processed.

If zlib and/or gzip header processing was requested, but a header
was never provided and inflateSync was used successfully, then the
inflate state would be inconsistent, trying to compute a check
value but with no flags set. This commit sets the inflate mode to
raw in this case, since there is no other assumption that can be
made if a header was requested but never seen.

6 years agoZlib patch: prevent uninitialized use of state->check
Sebastian Pop [Wed, 12 Dec 2018 15:35:43 +0000 (09:35 -0600)] 
Zlib patch: prevent uninitialized use of state->check

This CL fixes a security bug in zlib. It was reported upstream long ago
and the testcase was shared upstream but it's yet unsolved. As a fix,
state->check is set to the same value as the adler32 of an empty string.

Upstream bug: madler/zlib#245

Bug: chromium:697481 https://crbug.com/697481
Reviewed-on: https://chromium-review.googlesource.com/601193
Reviewed-by: Tom Sepez <tsepez@chromium.org>
Reviewed-by: Adam Langley <agl@chromium.org>
Commit-Queue: Nicolás Peña <npm@chromium.org>

6 years agoreturn an index for hash map collisions in insert_string
Sebastian Pop [Thu, 6 Dec 2018 19:23:17 +0000 (13:23 -0600)] 
return an index for hash map collisions in insert_string

The current version of insert_string_c and variations for sse2, arm, and aarch64
in zlib-ng has changed semantics from the original code of INSERT_STRING macro
in zlib:

 #define INSERT_STRING(s, str, match_head) \
   (UPDATE_HASH(s, s->ins_h, s->window[(str) + (MIN_MATCH-1)]), \
    match_head = s->prev[(str) & s->w_mask] = s->head[s->ins_h], \
    s->head[s->ins_h] = (Pos)(str))

The code of INSERT_STRING assigns match_head with the content of s->head[s->ins_h].

In zlib-ng, the assignment to match_head happens in the caller of insert_string().
zlib-ng's insert_string_*() functions return 0 instead of str+idx in case of
collision, i.e., when if (s->head[s->ins_h] == str+idx).

The effect of returning 0 instead of the content of s->head[s->ins_h] is that
the search for a longest_match through s->prev[] chains will be cut short when
arriving at 0. This leads to a shorter compression time at the expense of a
worse compression rate: returning 0 cuts out the search space.

With this patch:

 Performance counter stats for './minigzip -9 llvm.tar':

      13422.379017      task-clock (msec)         #    1.000 CPUs utilized
                20      context-switches          #    0.001 K/sec
                 0      cpu-migrations            #    0.000 K/sec
               130      page-faults               #    0.010 K/sec
    58,926,104,511      cycles                    #    4.390 GHz
   <not supported>      stalled-cycles-frontend
   <not supported>      stalled-cycles-backend
    77,543,740,646      instructions              #    1.32  insns per cycle
    17,158,892,214      branches                  # 1278.379 M/sec
       198,433,680      branch-misses             #    1.16% of all branches

      13.423365095 seconds time elapsed

45408 -rw-rw-r-- 1 spop spop 46493896 Dec 11 11:47 llvm.tar.gz

Without this patch the compressed file is larger:

 Performance counter stats for './minigzip -9 llvm.tar':

      13459.342312      task-clock (msec)         #    1.000 CPUs utilized
                25      context-switches          #    0.002 K/sec
                 0      cpu-migrations            #    0.000 K/sec
               129      page-faults               #    0.010 K/sec
    59,088,391,808      cycles                    #    4.390 GHz
   <not supported>      stalled-cycles-frontend
   <not supported>      stalled-cycles-backend
    77,600,766,958      instructions              #    1.31  insns per cycle
    17,486,130,785      branches                  # 1299.182 M/sec
       196,281,761      branch-misses             #    1.12% of all branches

      13.463512830 seconds time elapsed

45408 -rw-rw-r-- 1 spop spop 46493896 Dec 11 11:48 llvm.tar.gz

6 years agoCorrect the initialization requirements for deflateInit2().
Mark Adler [Wed, 31 Jan 2018 21:24:12 +0000 (13:24 -0800)] 
Correct the initialization requirements for deflateInit2().

6 years agofix bug #210: split statement in two to avoid substract overflow
Sebastian Pop [Wed, 26 Sep 2018 02:49:06 +0000 (21:49 -0500)] 
fix bug #210: split statement in two to avoid substract overflow

make check used to fail with:
trees.c:482:53: runtime error: unsigned integer overflow: 6 - 7 cannot be represented in type 'unsigned int'
SUMMARY: UndefinedBehaviorSanitizer: undefined-behavior trees.c:482:53 in

Patch from Mika Lindqvist.