Nick Terrell [Fri, 30 Oct 2020 20:54:30 +0000 (13:54 -0700)]
[regression] Update results.csv
https://github.com/facebook/zstd/pull/2339 removes the single-pass zstdmt API.
This changes the compressed size, because we no longer take the # of threads into
account when deciding the job size.
Nick Terrell [Mon, 12 Oct 2020 21:47:55 +0000 (14:47 -0700)]
[lib] Avoid allocating the input buffer when ZSTD_c_stableInBuffer is set
We don't use it when we have a stable input buffer, so don't allocate
it. I had to slightly modify `ZSTD_copyCCtx()` by storing the
`ZSTD_buffered_policy_e` in the `ZSTD_CCtx`, since `inBuffSize > 0` is
no longer the correct signal for the buffered mode.
Nick Terrell [Mon, 12 Oct 2020 21:09:12 +0000 (14:09 -0700)]
[lib] Take the shortcut when ZSTD_c_stableOutBuffer is set
When we have a stable output buffer take the single-pass shortcut.
It is okay to return `dstSizeTooSmall` if the output buffer isn't
big enough, because we know it will never grow.
Nick Terrell [Mon, 12 Oct 2020 20:51:35 +0000 (13:51 -0700)]
[lib] Set ZSTD_c_stable{In,Out}Buffer in ZSTD_compress2()
Sets these parameters in ZSTD_compress2() then resets them to their
orignal values after the compression call.
An alternative design could be to add a flush mode `ZSTD_e_singlePass`
which implies `ZSTD_c_stable{In,Out}Buffer` but only for a single
compression call, by directly setting the applied parameters. I've opted
for the smaller change, but this is open for discussion.
Nick Terrell [Tue, 13 Oct 2020 02:30:23 +0000 (19:30 -0700)]
[lib] Set appliedParams in zstdmt mode
Previously only `nbWorkers` was set. Set all parameters, because that is
what is expected. This is needed for the `ZSTD_c_stable{In,Out}Buffer`
parameters.
Yann Collet [Thu, 22 Oct 2020 02:22:45 +0000 (19:22 -0700)]
automatically detect configuration changes
Makefile now automatically detects modifications of compilation flags,
and produce object files in directories dedicated to this compilation flags.
This makes it possible, for example, to compile libzstd with different DEBUGLEVEL.
Object files sharing the same configration will be generated into their dedicated directories.
Also : new compilation variables
- DEBUGLEVEL : select the debug level (assert & traces) inserted during compilation (default == 0 == release)
- HASH : select a hash function to differentiate configuration (default == md5sum)
- BUILD_DIR : skip the hash stage, store object files into manually specified directory
Yann Collet [Wed, 21 Oct 2020 01:16:11 +0000 (18:16 -0700)]
decouple zstd object files from lib/
That was a subtle one :
VPATH is affecting search for both %.c source and %.o object files.
This meant that, when an object file already exists in lib/,
it's used in programs/,
even though programs/ is supposed to generate its own %.o object files.
With the new vpath directive, this is no longer the case :
the search is only activated for %.c source files.
Now, local programs/%.o are always generated
even if equivalent ones are already created in lib/.
It more clearly guarantees that lib/ and programs/ can use different compilation directives
without mixing resulting %.o object files.
Yann Collet [Fri, 16 Oct 2020 07:01:41 +0000 (00:01 -0700)]
faster rebuild of zstd
Building the zstd CLI costs time.
Some part of it is incompressible, leading to substantial iteration delay when testing code modifications.
That's mainly because all source files from the library must be rebuilt from source every time.
The main reason we don't build the CLI from library object files
is that we can't just build the object directly in the lib/ directory
(which they would by default)
since they use different compilation flags.
Specifically, the CLI enables multithreading, while the library doesn't (by default).
This is solved in this commit, by generating the object files locally.
Now, the CLI and the library can employ different sets of flags, without tripping over each other.
All library object files are generated directly into programs/ dir.
This works because no 2 source files have the same name.
Now, modifying a file doesn't require to recompile the entire lib, just the modified files.
The recipe is also compatible with `-j` parallel build, leading to large build time reductions on multi-core systems.