]> git.ipfire.org Git - thirdparty/zstd.git/log
thirdparty/zstd.git
6 years agoFixed unsafe string copy and concat in `fileio.c`. 1290/head
Eden Zik [Tue, 21 Aug 2018 02:15:24 +0000 (22:15 -0400)] 
Fixed unsafe string copy and concat in `fileio.c`.

Per warnings from flawfinder: "Does not check for buffer overflows when
copying to destination [MS-banned] (CWE-120). Consider using snprintf,
strcpy_s, or strlcpy (warning: strncpy easily misused).".

Replaced called to strcpy and strcat in `fileio.c` to calls with a
specified size (`strncpy` and `strncat`).

Tested the changes on OSX, Linux, Windows.
On OSX + Linux, changes were tested with ASAN. The following flags were
used: 'check_initialization_order=1:strict_init_order=1:detect_odr_violation=1:detect_stack_use_after_return=1'

To reproduce warning:
./flawfinder.py ./programs/fileio.c

6 years agoMerge pull request #1236 from GeorgeLu97/paramgrillconstraints
Yann Collet [Mon, 13 Aug 2018 22:44:50 +0000 (15:44 -0700)] 
Merge pull request #1236 from GeorgeLu97/paramgrillconstraints

ParamgrillConstraints

6 years agoDefault lvl 1 1236/head
George Lu [Mon, 13 Aug 2018 20:56:18 +0000 (13:56 -0700)] 
Default lvl 1

MB to 2^20

7 years agoReorder declaration
George Lu [Thu, 9 Aug 2018 23:38:32 +0000 (16:38 -0700)] 
Reorder declaration

7 years agofixed assert() condition
Yann Collet [Thu, 9 Aug 2018 22:57:19 +0000 (15:57 -0700)] 
fixed assert() condition

7 years agominor code comments improvements
Yann Collet [Thu, 9 Aug 2018 22:16:31 +0000 (15:16 -0700)] 
minor code comments improvements

7 years agoMerge pull request #1273 from facebook/zstdgrep
Yann Collet [Thu, 9 Aug 2018 19:35:49 +0000 (12:35 -0700)] 
Merge pull request #1273 from facebook/zstdgrep

added zstdgrep documentation

7 years agoadded zstdgrep documentation 1273/head
Yann Collet [Thu, 9 Aug 2018 19:28:25 +0000 (12:28 -0700)] 
added zstdgrep documentation

presenting `zstdgrep` limit regarding dictionary compression
with workaround recommended by @tobwen (#1268)

7 years agoRemove ctx from benchMem
George Lu [Thu, 9 Aug 2018 19:07:57 +0000 (12:07 -0700)] 
Remove ctx from benchMem

7 years agoAdd consts
George Lu [Thu, 9 Aug 2018 18:38:09 +0000 (11:38 -0700)] 
Add consts

+ fix gcc-8 warnings

7 years agoargument parsing cleanup
George Lu [Tue, 7 Aug 2018 18:56:14 +0000 (11:56 -0700)] 
argument parsing cleanup

+ clarifying comment

7 years agoReadability improvements, renaming
George Lu [Tue, 7 Aug 2018 01:37:55 +0000 (18:37 -0700)] 
Readability improvements, renaming

7 years agoconst srcPtrs
George Lu [Tue, 7 Aug 2018 01:00:36 +0000 (18:00 -0700)] 
const srcPtrs

7 years agoChange speed representation from floating point to integral
George Lu [Tue, 7 Aug 2018 00:13:36 +0000 (17:13 -0700)] 
Change speed representation from floating point to integral

7 years agoUpdate Documentation
George Lu [Mon, 30 Jul 2018 18:30:38 +0000 (11:30 -0700)] 
Update Documentation

Change comment // to  /* */
Add more description of what functions do
Remove outdated comments

7 years agoRenaming / Style fixes
George Lu [Fri, 27 Jul 2018 15:49:25 +0000 (08:49 -0700)] 
Renaming / Style fixes

7 years agoSeparate capacity vs size
George Lu [Fri, 20 Jul 2018 21:35:09 +0000 (14:35 -0700)] 
Separate capacity vs size

Also:
Make suggested fixes
-varInds_t
-reorder some arguments
-remove code duplication
-update README / -h
-Fix memory leaks

7 years agoFix windows implicit casting bugs
George Lu [Tue, 17 Jul 2018 01:22:04 +0000 (18:22 -0700)] 
Fix windows implicit casting bugs

7 years agoAdd Levels
George Lu [Tue, 17 Jul 2018 01:04:57 +0000 (18:04 -0700)] 
Add Levels

7 years agoBenchMem with block compressed sizes passed back up
George Lu [Mon, 16 Jul 2018 23:16:31 +0000 (16:16 -0700)] 
BenchMem with block compressed sizes passed back up

7 years agoTotal Changes:
George Lu [Fri, 13 Jul 2018 00:30:39 +0000 (17:30 -0700)] 
Total Changes:

Add different constraint types (decompression speed, compression memory, parameter constraints)
Separate search space by strategy + strategy selection
Memoize results
Real random restarts
Support multiple files
Support Dictionary inputs
Debug Macro for extra printing

7 years agoDictionary + Multiple file Loading
George Lu [Tue, 10 Jul 2018 01:37:54 +0000 (18:37 -0700)] 
Dictionary + Multiple file Loading

7 years agoNot crashing
George Lu [Mon, 9 Jul 2018 20:44:01 +0000 (13:44 -0700)] 
Not crashing

7 years agoWorking V1
George Lu [Mon, 18 Jun 2018 18:59:45 +0000 (11:59 -0700)] 
Working V1

7 years agoParses additional parameters
George Lu [Wed, 6 Jun 2018 23:19:09 +0000 (16:19 -0700)] 
Parses additional parameters

Additional constraint checking

Minor fixes

more param parsing

Add Memory

Change paramVariation

work on feasibility

reformat bench

Changed Paramgrill to use bench.c benchmarking

customlevel macro

Printing Flag

Minor changes

Explicit casting

Makefile fix

casting, type fix

Printing Flag

Minor Changes

comments, helper fn's

7 years agoMerge pull request #1272 from facebook/travisTest
Yann Collet [Wed, 8 Aug 2018 21:56:00 +0000 (14:56 -0700)] 
Merge pull request #1272 from facebook/travisTest

gcc-8 travis test

7 years agoMerge pull request #1271 from felixhandte/circleci-2.0-config
Yann Collet [Wed, 8 Aug 2018 21:48:51 +0000 (14:48 -0700)] 
Merge pull request #1271 from felixhandte/circleci-2.0-config

Migrate Config to CircleCI 2.0

7 years agoPreserve Commented-Out Longer Tests 1271/head
W. Felix Handte [Wed, 8 Aug 2018 21:09:12 +0000 (14:09 -0700)] 
Preserve Commented-Out Longer Tests

7 years agosome errors may only happen when optimizations are enabled 1272/head
Yann Collet [Wed, 8 Aug 2018 19:37:08 +0000 (12:37 -0700)] 
some errors may only happen when optimizations are enabled

7 years agoRemove Old CircleCI Config
W. Felix Handte [Wed, 8 Aug 2018 19:31:37 +0000 (12:31 -0700)] 
Remove Old CircleCI Config

7 years agoFix Path
W. Felix Handte [Wed, 8 Aug 2018 19:32:55 +0000 (12:32 -0700)] 
Fix Path

7 years agoAdd an Auto-Gen'ed CircleCI 2.0 Config
W. Felix Handte [Wed, 8 Aug 2018 17:28:22 +0000 (10:28 -0700)] 
Add an Auto-Gen'ed CircleCI 2.0 Config

Built via the cci-config-generator.sh script.

7 years agotry adding gcc-8 compilation tests
Yann Collet [Wed, 8 Aug 2018 17:26:54 +0000 (10:26 -0700)] 
try adding gcc-8 compilation tests

7 years agotry to improve some travis test speed
Yann Collet [Wed, 8 Aug 2018 17:22:19 +0000 (10:22 -0700)] 
try to improve some travis test speed

7 years agoMerge pull request #1243 from facebook/fix1241
Yann Collet [Wed, 8 Aug 2018 17:18:02 +0000 (10:18 -0700)] 
Merge pull request #1243 from facebook/fix1241

fix #1241

7 years agoMerge pull request #1269 from felixhandte/fix-fix1241 1243/head
Yann Collet [Wed, 8 Aug 2018 16:25:40 +0000 (09:25 -0700)] 
Merge pull request #1269 from felixhandte/fix-fix1241

Fix fix1241: Fix CDict Attachment to Handle CDicts with Non-Zero Starts

7 years agoFix CDict Attachment to Handle CDicts with Non-Zero Starts 1269/head
W. Felix Handte [Wed, 8 Aug 2018 00:05:05 +0000 (17:05 -0700)] 
Fix CDict Attachment to Handle CDicts with Non-Zero Starts

CDicts were previously guaranteed to be generated with `lowLimit=dictLimit=0`.
This is no longer true, and so the old length and index calculations are no
longer valid. This diff fixes them to handle non-zero start indices in CDicts.

7 years agoMerge branch 'dev' into fix1241
Yann Collet [Fri, 3 Aug 2018 23:08:33 +0000 (16:08 -0700)] 
Merge branch 'dev' into fix1241

7 years agoMerge pull request #1263 from facebook/travisTest
Yann Collet [Fri, 3 Aug 2018 22:57:22 +0000 (15:57 -0700)] 
Merge pull request #1263 from facebook/travisTest

Fix Makefile on Linux and ZSTD_NOBENCH

7 years agofix unused variable warning 1263/head
Yann Collet [Fri, 3 Aug 2018 15:30:01 +0000 (08:30 -0700)] 
fix unused variable warning

7 years agofix .travis.yml
Yann Collet [Fri, 3 Aug 2018 14:59:33 +0000 (07:59 -0700)] 
fix .travis.yml

7 years agofix : zstd cli can be built with build macro ZSTD_NOBENCH
Yann Collet [Fri, 3 Aug 2018 14:54:29 +0000 (07:54 -0700)] 
fix : zstd cli can be built with build macro ZSTD_NOBENCH

which disables bench.c module

7 years agoMerge branch 'dev' into travisTest
Yann Collet [Thu, 2 Aug 2018 18:53:46 +0000 (11:53 -0700)] 
Merge branch 'dev' into travisTest

7 years agoMerge branch 'travisTest' of github.com:facebook/zstd into travisTest
Yann Collet [Thu, 2 Aug 2018 18:52:12 +0000 (11:52 -0700)] 
Merge branch 'travisTest' of github.com:facebook/zstd into travisTest

7 years agoMerge pull request #1255 from terrelln/norm-fix
Yann Collet [Thu, 2 Aug 2018 18:48:56 +0000 (11:48 -0700)] 
Merge pull request #1255 from terrelln/norm-fix

[FSE] Fix division by zero

7 years agoDisallow tableLog == srcLog 1255/head
Nick Terrell [Thu, 2 Aug 2018 18:12:17 +0000 (11:12 -0700)] 
Disallow tableLog == srcLog

7 years agoAdd non-optimize FASTCOVER (#1260)
Jennifer Liu [Wed, 1 Aug 2018 18:06:16 +0000 (11:06 -0700)] 
Add non-optimize FASTCOVER (#1260)

* Add non-optimize FASTCOVER

* Minor fix

* Pass param as value instead of pointer

7 years agoMerge pull request #1258 from jennifermliu/fastCover
Nick Terrell [Tue, 31 Jul 2018 21:31:05 +0000 (14:31 -0700)] 
Merge pull request #1258 from jennifermliu/fastCover

Use CDict instead of CCtx in analyzeEntropy

7 years agoRefactoring 1258/head
Jennifer Liu [Tue, 31 Jul 2018 20:58:54 +0000 (13:58 -0700)] 
Refactoring

7 years agoUse CDict instead of CCtx in analyzeEntropy
Jennifer Liu [Tue, 31 Jul 2018 17:36:45 +0000 (10:36 -0700)] 
Use CDict instead of CCtx in analyzeEntropy

7 years agoMerge pull request #1257 from jennifermliu/fastCover
Nick Terrell [Mon, 30 Jul 2018 23:03:18 +0000 (16:03 -0700)] 
Merge pull request #1257 from jennifermliu/fastCover

Increment frequency for every dmer occurence within same sample

7 years ago[FSE] Add division by zero test
Nick Terrell [Mon, 30 Jul 2018 19:57:11 +0000 (12:57 -0700)] 
[FSE] Add division by zero test

7 years agoIncrement frequency for every dmer occurence within same sample instead of at most... 1257/head
Jennifer Liu [Mon, 30 Jul 2018 19:54:22 +0000 (12:54 -0700)] 
Increment frequency for every dmer occurence within same sample instead of at most once per sample

7 years agospeed up appveyor tests
cyan4973 [Mon, 30 Jul 2018 15:08:40 +0000 (17:08 +0200)] 
speed up appveyor tests

7 years agofix %zu support under minGW
cyan4973 [Mon, 30 Jul 2018 14:56:18 +0000 (16:56 +0200)] 
fix %zu support under minGW

and relevant test on Appveyor

7 years agoensure appveyor test fails due to formatting error
cyan4973 [Mon, 30 Jul 2018 14:44:20 +0000 (16:44 +0200)] 
ensure appveyor test fails due to formatting error

to catch %zu incompatibility

7 years agoMerge branch 'dev' into fix1241
cyan4973 [Mon, 30 Jul 2018 14:30:35 +0000 (16:30 +0200)] 
Merge branch 'dev' into fix1241

7 years agosupport %zu under mingw
cyan4973 [Mon, 30 Jul 2018 14:29:20 +0000 (16:29 +0200)] 
support %zu under mingw

7 years agoMerge pull request #1254 from jennifermliu/fastCover
Nick Terrell [Sat, 28 Jul 2018 01:22:52 +0000 (18:22 -0700)] 
Merge pull request #1254 from jennifermliu/fastCover

 Save segmentFreqs in ctx instead of malloc and memset in SelectSegment

7 years agoDelete old benchmarking result 1254/head
Jennifer Liu [Sat, 28 Jul 2018 00:31:33 +0000 (17:31 -0700)] 
Delete old benchmarking result

7 years ago[FSE] Fix division by zero
Nick Terrell [Sat, 28 Jul 2018 00:30:03 +0000 (17:30 -0700)] 
[FSE] Fix division by zero

When the primary normalization method fails, and
`(1 << tableLog) == (maxSymbolValue + 1)`, and every symbol gets assigned
normalized weight 1 or -1 in the first loop, then the next division can
raise `SIGFPE`.

7 years agoUndo deleting clean in make
Jennifer Liu [Fri, 27 Jul 2018 23:56:50 +0000 (16:56 -0700)] 
Undo deleting clean in make

7 years agoRevert test.sh
Jennifer Liu [Fri, 27 Jul 2018 23:54:05 +0000 (16:54 -0700)] 
Revert test.sh

7 years agoSave segmentFreqs in ctx instead of malloc and memset in SelectSegment
Jennifer Liu [Fri, 27 Jul 2018 23:51:38 +0000 (16:51 -0700)] 
Save segmentFreqs in ctx instead of malloc and memset in SelectSegment

7 years agoMerge pull request #1250 from jennifermliu/fastCover
Nick Terrell [Fri, 27 Jul 2018 22:11:57 +0000 (15:11 -0700)] 
Merge pull request #1250 from jennifermliu/fastCover

Add Fast Cover Dictionary Builder

7 years agoUse same param after optimizing cover and fastCover and record k and d for benchmarking 1250/head
Jennifer Liu [Fri, 27 Jul 2018 20:39:19 +0000 (13:39 -0700)] 
Use same param after optimizing cover and fastCover and record k and d for benchmarking

7 years agoRerun cover and fastCover with optimized values
Jennifer Liu [Fri, 27 Jul 2018 02:03:01 +0000 (19:03 -0700)] 
Rerun cover and fastCover with optimized values

7 years agoBenchmark different f values
Jennifer Liu [Thu, 26 Jul 2018 23:24:13 +0000 (16:24 -0700)] 
Benchmark different f values

7 years agoMerge pull request #1252 from GeorgeLu97/bench-quiet-display-fix
Nick Terrell [Thu, 26 Jul 2018 23:12:15 +0000 (16:12 -0700)] 
Merge pull request #1252 from GeorgeLu97/bench-quiet-display-fix

Fixes bug

7 years agono zero 1252/head
George Lu [Thu, 26 Jul 2018 22:17:58 +0000 (15:17 -0700)] 
no zero

7 years agoAdd array to keep track of frequency within active segment, fix malloc bug, update...
Jennifer Liu [Thu, 26 Jul 2018 20:53:13 +0000 (13:53 -0700)] 
Add array to keep track of frequency within active segment, fix malloc bug, update benchmarking result

7 years agoAllow d=6
Jennifer Liu [Thu, 26 Jul 2018 01:10:09 +0000 (18:10 -0700)] 
Allow d=6

7 years agoBenchmark fast cover optimize vs k=200
Jennifer Liu [Thu, 26 Jul 2018 00:53:38 +0000 (17:53 -0700)] 
Benchmark fast cover optimize vs k=200

7 years agoInitial benchmarking result for fastCover
Jennifer Liu [Thu, 26 Jul 2018 00:05:54 +0000 (17:05 -0700)] 
Initial benchmarking result for fastCover

7 years agoMake hash value const
Jennifer Liu [Wed, 25 Jul 2018 23:54:08 +0000 (16:54 -0700)] 
Make hash value const

7 years agoAdd Fast Cover Dictionary Builder
Jennifer Liu [Wed, 25 Jul 2018 23:34:07 +0000 (16:34 -0700)] 
Add Fast Cover Dictionary Builder

7 years agoMerge pull request #1246 from jennifermliu/benchmark
Nick Terrell [Sat, 21 Jul 2018 01:09:31 +0000 (18:09 -0700)] 
Merge pull request #1246 from jennifermliu/benchmark

Benchmark dictionary builders

7 years agoMinor fix 1246/head
Jennifer Liu [Sat, 21 Jul 2018 00:41:22 +0000 (17:41 -0700)] 
Minor fix

7 years agoRefactoring and benchmark without dictionary
Jennifer Liu [Sat, 21 Jul 2018 00:03:47 +0000 (17:03 -0700)] 
Refactoring and benchmark without dictionary

7 years agoBenchmark dictionary builders
Jennifer Liu [Fri, 20 Jul 2018 18:32:39 +0000 (11:32 -0700)] 
Benchmark dictionary builders

7 years agoMerge pull request #1245 from terrelln/disable-arm
Nick Terrell [Thu, 19 Jul 2018 22:32:53 +0000 (15:32 -0700)] 
Merge pull request #1245 from terrelln/disable-arm

Revert "attempt to re-enable arm64 tests"

7 years agoRevert "attempt to re-enable arm64 tests" 1245/head
Nick Terrell [Thu, 19 Jul 2018 21:35:27 +0000 (14:35 -0700)] 
Revert "attempt to re-enable arm64 tests"

This reverts commit 9c277f137cbcaa385ff5b95ec4cbdce50675541d.

7 years agoMerge pull request #1238 from jennifermliu/random
Nick Terrell [Thu, 19 Jul 2018 20:52:15 +0000 (13:52 -0700)] 
Merge pull request #1238 from jennifermliu/random

Add random dictionary builder

7 years agoUpdate Makefile 1238/head
Jennifer Liu [Thu, 19 Jul 2018 20:44:27 +0000 (13:44 -0700)] 
Update Makefile

7 years agoRename cleanup
Jennifer Liu [Wed, 18 Jul 2018 19:15:49 +0000 (12:15 -0700)] 
Rename cleanup

7 years agoAdd cleanup to trainfromFiles and move RANDOM_segment_t declaration
Jennifer Liu [Wed, 18 Jul 2018 17:40:13 +0000 (10:40 -0700)] 
Add cleanup to trainfromFiles and move RANDOM_segment_t declaration

7 years agoMerge pull request #1230 from terrelln/train-out
Yann Collet [Wed, 18 Jul 2018 14:34:10 +0000 (16:34 +0200)] 
Merge pull request #1230 from terrelln/train-out

zstdcli: Allow -o before --train

7 years agoMerge pull request #1237 from terrelln/init-cstream-adv
Yann Collet [Wed, 18 Jul 2018 14:33:30 +0000 (16:33 +0200)] 
Merge pull request #1237 from terrelln/init-cstream-adv

Set requestedParams in ZSTD_initCStream*()

7 years agofix minor conversion warning
cyan4973 [Wed, 18 Jul 2018 14:10:23 +0000 (16:10 +0200)] 
fix minor conversion warning

7 years agoMerge pull request #1242 from facebook/mingw_traces
Yann Collet [Wed, 18 Jul 2018 10:23:07 +0000 (12:23 +0200)] 
Merge pull request #1242 from facebook/mingw_traces

zstdmt debug traces compatibles with mingw

7 years agoMerge pull request #1244 from terrelln/ferror
Yann Collet [Wed, 18 Jul 2018 10:22:41 +0000 (12:22 +0200)] 
Merge pull request #1244 from terrelln/ferror

fileio: Error in compression on read errors

7 years agoUpdate freeSampleInfo
Jennifer Liu [Tue, 17 Jul 2018 23:13:40 +0000 (16:13 -0700)] 
Update freeSampleInfo

7 years agoFix deallocation problem and add documentation
Jennifer Liu [Tue, 17 Jul 2018 23:01:44 +0000 (16:01 -0700)] 
Fix deallocation problem and add documentation

7 years agofileio: Error in compression on read errors 1244/head
Nick Terrell [Tue, 17 Jul 2018 21:57:27 +0000 (14:57 -0700)] 
fileio: Error in compression on read errors

We can write a corrupted file if the input file errors during a read.
We should return a non-zero error code in this case.

7 years agoMake test PHONY target
Jennifer Liu [Tue, 17 Jul 2018 19:42:53 +0000 (12:42 -0700)] 
Make test PHONY target

7 years agoMove file loading functions to new file for access by benchmarking tool
Jennifer Liu [Tue, 17 Jul 2018 19:35:09 +0000 (12:35 -0700)] 
Move file loading functions to new file for access by benchmarking tool

7 years agofix make test on Linux
cyan4973 [Tue, 17 Jul 2018 17:19:48 +0000 (19:19 +0200)] 
fix make test on Linux

MOREFLAGS+= doesn't work on Linux

7 years agoadded a test to be played on travis
cyan4973 [Tue, 17 Jul 2018 17:02:17 +0000 (19:02 +0200)] 
added a test to be played on travis

to check a make variable construction

7 years agofix #1241
cyan4973 [Tue, 17 Jul 2018 16:52:57 +0000 (18:52 +0200)] 
fix #1241

Ensure that first input position is valid for a match
even during first usage of context
by starting reference at 1
(avoiding the problematic 0).

7 years agozstdmt debug traces compatibles with mingw 1242/head
cyan4973 [Tue, 17 Jul 2018 12:39:44 +0000 (14:39 +0200)] 
zstdmt debug traces compatibles with mingw

since mingw does not have `sys/times.h`,
remove this path when detecting mingw compilation.