Wayne Davison [Mon, 25 May 2020 21:01:52 +0000 (14:01 -0700)]
Two sparse fixes from Yuxuan Shui.
- Make "len" parameter of do_punch_hole an OFF_T.
- Clear sparse_past_write in sparse_end(), otherwise when write_sparse()
is called for the next file, do_punch_hole() will be called with a pos
that's not actually the current position in file, causing it to fail.
Jorrit Jongma [Sun, 24 May 2020 22:30:55 +0000 (00:30 +0200)]
Improve performance of file_checksum()
Previously files were hashed in blocks of CSUM_CHUNK (64) bytes. This
causes significant overhead. The CSUM_CHUNK define cannot be changed as
md5.c depends on it, but there is no obvious reason to use it in
file_checksum(). By using CHUNK_SIZE (32 kB) instead, in some test
cases throughput more than doubles.
Wayne Davison [Sun, 24 May 2020 22:45:59 +0000 (15:45 -0700)]
More compress changes
- Add the zlibx (external-code compatible) compression name.
- Re-enable zlib support with the external library so it can be
tried as a fallback if zlibx isn't available.
- Add --compress-choice=STR (aka -zz=STR) option.
- Make --cc=STR an alias for --checksum-choice=STR.
- Hook up the new compression negotiation logic.
Wayne Davison [Sun, 24 May 2020 05:05:16 +0000 (22:05 -0700)]
Some more checksum improvements
- Add/improve --debug=CSUM2 messages.
- Add an "xxh64" alias for "xxhash" name because we should be
getting a few more xxhash variations in the future.
- Tweak the matching code to handle entries that have multiple
names.
- Tweak some of the vars/defines.
Wayne Davison [Sun, 24 May 2020 04:46:15 +0000 (21:46 -0700)]
Improve some early debug-message newlines.
Avoid a newline issue during the output of --DEBUG=CSUM info from
both the server and the client -- we need to output the full message
with its newline as much as possible.
Jorrit Jongma [Fri, 22 May 2020 17:38:37 +0000 (19:38 +0200)]
Optimized assembler version of md5_process() for x86-64
Originally created by Marc Bevand and placed in the public domain.
Enable/disabled via the same --enable-simd configure switch as
the rolling checksum optimizations.
Jorrit Jongma [Fri, 22 May 2020 11:03:55 +0000 (13:03 +0200)]
AVX2 optimized version of get_checksum1() for x86-64
Additionally restructures build switches and defines from SSE2 to SIMD,
to allow potential reuse should patches become available with SIMD
instructions for other processor architectures.
(Some minor tweaks of Jorrit's patch to avoid requiring GNU make and to
avoid C++ comments in .c files.)
Wayne Davison [Fri, 22 May 2020 15:23:26 +0000 (08:23 -0700)]
Checksum negotiation & more bits for compat_flags
- Add checksum negotiation to the protocol so that we can easily add new
checksum algorithms and each will be used when both sides support it.
- Increase the size of the compat_flags value in the protocol from a
byte to an int.
Wayne Davison [Fri, 22 May 2020 15:10:41 +0000 (08:10 -0700)]
Fix a bug in the writing of the batch.sh file
Fix the code that writes the options and the default destination path
into the batch.sh file to be able to handle options being specified
after source/dest args.
Jorrit Jongma [Tue, 19 May 2020 12:52:40 +0000 (14:52 +0200)]
SSE2/SSSE3 optimized version of get_checksum1() for x86-64
Requires compilation using GCC C++ front end, build scripts have been
modified accordingly. C++ is only used when the optimization is enabled
(g++ as compiler, x86-64 build target, --enable-sse2 is passed to
configure).
(Wayne made a few tweaks, including making it disabled by default.)
Wayne Davison [Thu, 30 Apr 2020 00:55:41 +0000 (17:55 -0700)]
Use a varint to send the file-list flags
If both sides support the "V" compatibility flag, we send the file-list
flags as a varint instead of a 1-or-2 byte value. This upgrades the
number of reserved flag bits from 1 to 17 with very few extra bytes in
typical file-list data.
Wayne Davison [Wed, 22 Apr 2020 17:53:38 +0000 (10:53 -0700)]
Some more rsync-ssl improvements:
- Make the rsync-ssl default behavior more user friendly.
- Install rsync-ssl & rsync-ssl-rsh in the regular install rules.
- Add a manpage for rsync-ssl (which is also installed).
- Get rid of the rsync-ssl-client package in our spec file.
Wayne Davison [Sun, 19 Apr 2020 19:08:01 +0000 (12:08 -0700)]
Change the rsync-ssl helper script
The new rsh-ssl-rsync helper script (replacing stunnel-rsync) supports
openssl in addition to stunnel. The RSYNC_SSL_TYPE environment variable
can be set to specify which type of connection to use, and the first arg
to rsync-ssl can be --type=stunnel or --type=openssl to override the env
var or the default of "stunnel". The helper script now looks for
stunnel4 or stunnel on the PATH at runtime instead of having configure
look for it at compile time.
Wayne Davison [Thu, 9 Apr 2020 22:11:37 +0000 (15:11 -0700)]
Set Copyright years and make them easier to update
I replaced git-set-file-times with an improved version that I wrote
recently (in python3). A new script uses it to figure out the
last-modified year for each *.[ch] file and updates its copyright.
It also puts the latest year into the latest-year.h file for the
output of --version.
Wayne Davison [Wed, 8 Apr 2020 01:08:05 +0000 (18:08 -0700)]
Fix the default skip-compress list.
The default value of the skip-compress list actually comes from the
daemon's default lp_dont_compress() value, but a while back the vars
stopped getting default values in a non-daemon run. I added a call to
reset_daemon_vars() so that the "Vars" values get set from "Defaults".
Wayne Davison [Sun, 5 Apr 2020 23:41:15 +0000 (16:41 -0700)]
Don't throw an error if a potential fuzzy dir isn't a dir
Add a flag for calling get_dirlist() and for send_directory() that
indicates that the dirname is allowed to not be a directory. Based
on a patch by Ben Rubson. Fixes bug #13445.
Wayne Davison [Sun, 5 Apr 2020 21:52:32 +0000 (14:52 -0700)]
Add progress output via SIGINFO and SIGVTALRM
On BSD-ish systems you can type Ctrl+T to see the current file and
the progress output (in --info=progress2 format). On hosts w/o
SIGINFO, use something like "killall -VTALRM rsync" or a more
targetted "kill -VTALRM PID ..." call (as needed).
Wayne Davison [Sun, 5 Apr 2020 18:21:02 +0000 (11:21 -0700)]
Added the --write-devices option.
This is a fleshed out version of the old one in the patches repo with
documentation & proper handling of the implied --inplace option for a
daemon's option-rufusing considerations. I ommitted the -w short option
as I would hate for someone to turn this on accidentally.