Douglas Bagnall [Thu, 17 Nov 2022 03:09:26 +0000 (16:09 +1300)]
fuzz: add fuzz_lzxpress_huffman_compress
This differs from fuzz_lzxpress_huffman_round_trip (next commit) in
that the output buffer might be too small for the compressed data, in
which case we want to see an error and not a crash.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
With LZXHUFF_DEBUG_VERBOSE set, we measure the compression and
decompression rate relative to the decompressed size.
On reasonably long strings on my laptop, compiled with -O0, it turns
out to between 20 and 500 MB/s, both ways, depending on the complexity
of the string. Very short strings are of course dominated by overhead.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 10:14:58 +0000 (23:14 +1300)]
lib/compression: LZ77 + Huffman compression
This compresses files as described in MS-XCA 2.2, and as decompressed
by the decompressor in the previous commit.
As with the decompressor, there are two public functions -- one that
uses a talloc context, and one that uses pre-allocated memory. The
compressor requires a tightly bound amount of auxillary memory
(>220kB) in a few different buffers, which is all gathered together in
the public struct lzxhuff_compressor_mem. An instantiated but not
initialised copy of this struct is required by the non-talloc
function; it can be used over and over again.
Our compression speed is about the same as the decompression speed
(between 20 and 500 MB/s on this laptop, depending on the data), and
our compression ratio is very similar to that of Windows.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
In both cases the caller needs to know the *exact* decompressed size,
which is essential for decompression. The _talloc version allocates
the buffer for you, and uses the talloc context to allocate a 128k
working buffer. THe non-talloc function will allocate the working
buffer on the stack.
This compression format gives better compression for messages of
several kilobytes than the "plain" LXZPRESS compression, but is
probably a bit slower to decompress and is certainly worse for very
short messages, having a fixed 256 byte overhead for the first Huffman
table.
Experiments show decompression rates between 20 and 500 MB per second,
depending on the compression ratio and data size, on an i5-1135G7 with
no compiler optimisations.
This compression format is used in AD claims and in SMB, but that
doesn't happen with this commit.
I will not try to describe LZ77 or Huffman encoding here. Don't expect
an answer in MS-XCA either; instead read the code and/or Wikipedia.
but there's more earlier, particularly in June/July 2020, when
Aurélien Aptel was working on an implementation that ended up in
Wireshark.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Pair-programmed-with: Joseph Sutton <josephsutton@catalyst.net.nz> Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Thu, 17 Nov 2022 03:07:08 +0000 (16:07 +1300)]
testdata: add test vectors for LZ77+Huffman [de-]compression
Some of the decompressed files were found via fuzzing, some are public
domain texts, and some are designed to test one aspect or another of
the format. For example, some aspects of Huffman tree creation can
only be tested when there is an extreme imbalance in the frequency of
symbols.
See the README for what files are where.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Douglas Bagnall [Wed, 28 Sep 2022 01:40:10 +0000 (14:40 +1300)]
util: add stable sort functions
Sometimes (e.g. in lzxpress Huffman encoding, and in some of our
tests: c.f. https://lists.samba.org/archive/samba-technical/2018-March/126010.html)
we want a stable sort algorithm (meaning one that retains the previous
order of items that compare equal).
The GNU libc qsort() is *usually* stable, in that it first tries to
use a mergesort but reverts to quicksort if the necessary allocations
fail. That has led Samba developers to unthinkingly assume qsort() is
stable which is not the case on many platforms, and might not always
be on GNU/Linuxes either.
This adds four functions. stable_sort() sorts an array, and requires
an auxiliary working array of the same size. stable_sort_talloc()
takes a talloc context so it ca create a working array and call
stable_sort(). stable_sort_r() takes an opaque context blob that gets
passed to the compare function, like qsort_r() and ldb_qsort(). And
stable_sort_talloc_r() rounds out the quadrant.
These are LGPL so that the can be used in ldb, which has problems with
unstable sort.
The tests are borrowed and extended from test_ldb_qsort.c.
When sorting non-trivial structs this is roughly as fast as GNU qsort,
but GNU qsort has optimisations for small items, using direct
assignments of rather than memcpy where the size allows the item to be
cast as some kind of int.
Signed-off-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz> Reviewed-by: Joseph Sutton <josephsutton@catalyst.net.nz>
Jeremy Allison [Fri, 18 Nov 2022 18:50:35 +0000 (10:50 -0800)]
s3: smbd: Fix schedule_smb2_aio_read() to allow the last read in a compound to go async.
Remove knownfail.
Signed-off-by: Jeremy Allison <jra@samba.org> Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Thu Dec 1 16:04:07 UTC 2022 on sn-devel-184
Jeremy Allison [Fri, 18 Nov 2022 21:30:05 +0000 (13:30 -0800)]
s4: torture: Tweak the compound padding streamfile test to send 3 reads instead of 2, and check the middle read padding.
The protocol allows the last read in a related compound to be split
off and possibly go async (and smbd soon will do this). If the
last read is split off, then the padding is different. By sending
3 reads and checking the padding on the 2nd read, we cope with
the smbd change and are still correctly checking the padding
on a compound related read.
Do this for the stream filename compound padding test.
Signed-off-by: Jeremy Allison <jra@samba.org> Reviewed-by: Ralph Boehme <slow@samba.org>
Jeremy Allison [Fri, 18 Nov 2022 21:23:48 +0000 (13:23 -0800)]
s4: torture: Tweak the compound padding basefile test to send 3 reads instead of 2, and check the middle read padding.
The protocol allows the last read in a related compound to be split
off and possibly go async (and smbd soon will do this). If the
last read is split off, then the padding is different. By sending
3 reads and checking the padding on the 2nd read, we cope with
the smbd change and are still correctly checking the padding
on a compound related read.
Do this for the base filename compound padding test.
Signed-off-by: Jeremy Allison <jra@samba.org> Reviewed-by: Ralph Boehme <slow@samba.org>
Jeremy Allison [Fri, 18 Nov 2022 17:53:23 +0000 (09:53 -0800)]
s3: tests: Change smb2.compound_async to run against share aio_delay_inject instead of tmp.
It doesn't hurt the fsync compound async tests, and we need this for
the next commits to ensure smb2_read/smb2_write compound tests take
longer than 500ms so can be sure the last read/write in the compound
will go async.
Signed-off-by: Jeremy Allison <jra@samba.org> Reviewed-by: Ralph Boehme <slow@samba.org>
The timer for the timeout_cb() handler was created on a memory context
which doesn't get freed, so the timer was still valid when running
the next test and fired there. It was then writing into random memory
leading to segfaults.
Signed-off-by: Andreas Schneider <asn@samba.org> Reviewed-by: Stefan Metzmacher <metze@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Thu Dec 1 15:03:19 UTC 2022 on sn-devel-184
Signed-off-by: Ralph Boehme <slow@samba.org> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Mon Nov 28 10:14:12 UTC 2022 on sn-devel-184
vfs: fix the build of nfs4acl_xattr_ without rpc/xdr.h support
Signed-off-by: Stefan Metzmacher <metze@samba.org> Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Fri Nov 25 06:07:32 UTC 2022 on sn-devel-184
gitlab-ci: do some basic testing on ubuntu1804-32bit
For now we allow build warnings and only do some basic testing.
We also ignore timestamp related problems, as well as some charset
failures.
Over time we should try to address the situation by not allowing warnings
and verify if expected failures are harmless or not.
But it's already much better then having no 32bit testing at all!
Signed-off-by: Stefan Metzmacher <metze@samba.org> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Autobuild-User(master): Stefan Metzmacher <metze@samba.org>
Autobuild-Date(master): Thu Nov 24 12:05:26 UTC 2022 on sn-devel-184
selftest: samba-ktest-mit also needs $ENV{KRB5RCACHETYPE} = "none"
We need to pass --mitkrb5 to selftest.pl in all cases we use
system mit kerberos not only when we also test the kdc.
We can't use a replay cache in selftest verifies the stat.st_uid
against getuid().
BTW: while debugging this on ubuntu 22.04 I exported
KRB5_TRACE="/dev/stderr", which means we get tracing into
the servers log file and into selftest_prefix/subunit for the client...
Signed-off-by: Stefan Metzmacher <metze@samba.org> Reviewed-by: Andrew Bartlett <abartlet@samba.org>
s4:kdc: make sure reset_bad_password_netlogon() stops subreq before return
We pass the stack variable 'req' to dcerpc_winbind_SendToSam_r_send(),
so we need to make sure the runtime of the subreq in not longer
than the stack variable.
CVE-2021-20251: s4:auth: fix use after free in authsam_logon_success_accounting()
This fixes a use after free problem introduced by
commit 7b8e32efc336fb728e0c7e3dd6fbe2ed54122124,
which has msg = current; which means the lifetime
of the 'msg' memory is no longer in the scope of th
caller.
Volker Lendecke [Thu, 27 Oct 2022 10:59:53 +0000 (12:59 +0200)]
tests: Start testing smb2 symlink error returns
This still all fails, but if you run them against Windows they work.
Signed-off-by: Volker Lendecke <vl@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Nov 22 19:25:34 UTC 2022 on sn-devel-184
Volker Lendecke [Thu, 27 Oct 2022 10:48:59 +0000 (12:48 +0200)]
tests: Add nosymlinks_smb1allow share
The next commits will create symlinks via posix extensions to test the
smb2 symlink error return. Creating posix symlinks is not allowed with
follow symlinks = no, but it's currently our only way to create
symlinks over SMB. This could go away once we can create symlinks via
reparse points.
Signed-off-by: Volker Lendecke <vl@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 14 Oct 2022 15:12:26 +0000 (17:12 +0200)]
smbd: Pass unparsed_path_length to symlink_reparse_buffer_marshall()
[MS-FSCC] 2.1.2.4 Symbolic Link Reparse Data Buffer lists this field
as reserved, but [MS-SMB2] 2.2.2.2.1 Symbolic Link Error Response is
the exact same format with the reserved field as UnparsedPathLength.
Signed-off-by: Volker Lendecke <vl@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Fri, 14 Oct 2022 14:23:30 +0000 (16:23 +0200)]
smbd: Pass error_context_count through smbd_smb2_request_error_ex()
See [MS-SMB2] 2.2.2: This field MUST be set to 0 for SMB dialects
other than 3.1.1. For the SMB dialect 3.1.1, if this field is nonzero,
the ErrorData field MUST be formatted as a variable-length array of
SMB2 ERROR Context structures containing ErrorContextCount entries.
Not used right now yet, but once we start to return STOPPED_ON_SYMLINK properly
this is required.
Signed-off-by: Volker Lendecke <vl@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Wed, 9 Nov 2022 10:25:51 +0000 (11:25 +0100)]
libsmb: Add "DOMAIN" to authentication creds
If you want to create symlinks on Windows using reparse points, you
need to authenticate as local administrator, just "administrator" is
not enough. So this is required to run some tests against Windows.
Signed-off-by: Volker Lendecke <vl@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
Volker Lendecke [Thu, 10 Nov 2022 16:40:22 +0000 (17:40 +0100)]
libsmb: Fix removing a rogue reparse point
If you set a reparse point for which Windows server does not have a
handler, it returns NT_STATUS_IO_REPARSE_TAG_NOT_HANDLED when you
later open it without FILE_OPEN_REPARSE_POINT.
See the discussion thread starting with
https://lists.samba.org/archive/cifs-protocol/2022-November/003888.html
Signed-off-by: Volker Lendecke <vl@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
David Mulder [Fri, 18 Nov 2022 18:42:15 +0000 (11:42 -0700)]
gp: Test PAM Access with DENY_ALL
Signed-off-by: David Mulder <dmulder@samba.org> Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Mon Nov 21 22:05:01 UTC 2022 on sn-devel-184
Signed-off-by: Andreas Schneider <asn@samba.org> Reviewed-by: Volker Lendecke <vl@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Fri Nov 18 19:17:31 UTC 2022 on sn-devel-184
Signed-off-by: Jeremy Allison <jra@samba.org> Reviewed-by: Ralph Boehme <slow@samba.org>
Autobuild-User(master): Ralph Böhme <slow@samba.org>
Autobuild-Date(master): Thu Nov 17 05:55:42 UTC 2022 on sn-devel-184
Jeremy Allison [Tue, 18 Oct 2022 23:22:33 +0000 (16:22 -0700)]
s4: torture: Add an async SMB2_OP_FLUSH + SMB2_OP_CLOSE test to smb2.compound_async.
Shows we fail sending an SMB2_OP_FLUSH + SMB2_OP_CLOSE
compound. Internally the flush goes async and
we free the req, then we process the close.
When the flush completes it tries to access
already freed data.
Found using the Apple MacOSX client at SNIA SDC 2022.
Noel Power [Wed, 16 Nov 2022 15:37:52 +0000 (15:37 +0000)]
nsswitch: Fix uninitialized memory when allocating pwdlastset_prelim
BUG: https://bugzilla.samba.org/show_bug.cgi?id=15224 Signed-off-by: Noel Power <noel.power@suse.com> Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Wed Nov 16 19:29:21 UTC 2022 on sn-devel-184
Signed-off-by: Andreas Schneider <asn@samba.org> Reviewed-by: Alexander Bokovoy <ab@samba.org>
Autobuild-User(master): Andreas Schneider <asn@cryptomilk.org>
Autobuild-Date(master): Wed Nov 16 16:29:30 UTC 2022 on sn-devel-184
projects / thirdparty / samba.git / log
