Fix PEP8 warning W291 trailing whitespace

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Fix PEP8 warning E303 too many blank lines

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Fix PEP8 warning E302 expected 2 blank lines

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Fix PEP8 warning E231 missing whitespace after ','

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Fix PEP8 warning F401 'blah' imported but unused

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Fix PEP8 warning E225 missing whitespace around operator

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Fix PEP8 warning F841 local variable 'blah' is assigned to but never used

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

samba-tool domain passwordsettings: Avoid except Exception

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python/pso tests: use  string .format() style rather than C-style %s/%d.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Fix PEP8 warning E122/E126/E127 wrong indent for continuation lines

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Fix PEP8 warning E201/202/203 array/dict whitespace

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Fix PEP8 warning E711 comparison to None

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python3: reuse cmp_fn defined in compat.py

This will also fix PEP8 E306:

    expected 1 blank line before a nested definition, found 0

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

python/samba/tests: fix SamDB dummy replacement

In commit 6de9d878b, a dummy SamDB lambda was added:

    SamDB = lambda *x: None

The `*x` will only cover positional args. If we call it with kwargs:

    samdb = SamDB(url=url)

We will get TypeError:

    <lambda>() got an unexpected keyword argument 'url'

This commit fix this. It also fix PEP8 E731:

    do not assign a lambda expression, use a def

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13542

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

descriptor: add missing backslash for long sddl str

Find this bug while doing PEP8.
We are lucky this code was not used yet.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

WHATSNEW: Add information on new GPO features

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Fri Aug 17 02:44:41 CEST 2018 on sn-devel-144

gpo: Always use an SMB signed connection

This ensures data integrity in the backup.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

gpo: Maintain an XML DTD for reference of the backup

This may or may not actually parse, but is mostly for reference

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

tests/gpo: Tests using a static backup directory from gpo backup

In particular, we want to see that the binary matches, that the XML will
backup to the same values, that the fallback with copy-restore works,
and that the generalize will generalize over different restored
entities.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

tests/gpo: Add a backup for showing that GPO backup and restore works

This will be used to write a test in the coming patches.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Make restore with entities more robust

Sometimes the restore fails for unknown reasons, but rearranging the XML
such that the DTD is after the xml header appears to fix it. This might
be the case in certain files where no entities are used perhaps.

This could probably be made more tolerant using regex, but for the most
part we expect the fixed output from the minidom pretty-printed XML.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gitattributes: Ignore .SAMBABACKUP files

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Add a --generalize to the backup command

This normally prints out the entities in DTD form to be given to the restore
command with --entities. Specifying --entities during the backup conveniently
writes these entities to a file. Generalizing occurs after the standard backup
on the XML files, which will then re-write the XML file.

There are a number of files which can be further handled, including many of the
preferences XML files. This will require more annotation and parsing.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

fdeploy_ini: Generalize the share name SIDs

This overrides the custom entity handler defined in the top level parser.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_parse: Add a generalize XML function to the top level parser

In this function we take XML and using the required metadata, we rewrite
it into a generic form using entities. ElementTree unfortunately does
not allow us to store unescaped entities, and so we must do a textual
replace on the output XML.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_ini: Add a scripts ini parser for better generalization

We mark the command path argument as a network path.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_ini: Add a fdeploy1 parser for better generalization

We still fail to handle entities in fdeploy.ini (version 0) files. Here we
manage to factor out some of the SIDs, but not all of them. This will be
completed in a later patch. The overall idea is to split the SID values into
individual XML elements and annotate them. We also note down network paths for
the redirection folders.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_ini: Allow better overriding of behaviour in inherited classes

We will need this to parse the parameters or section names as SIDs for fdeploy1.ini

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_csv: Add CSV generalization metadata

There are user identifiers and ACLs which may be stored in the audit CSV.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Enable more specific parsers of GPO files

* .pol files
* .ini (and GPT.ini)
* audit.csv
* GptTmpl.inf

.aas is currently not handled.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_aas: Leave a placeholder for the .aas files for now

This is to be implemented, but the documentation is somewhat lacking for
the .aas files and we so we leave this for now. In particular, the
documentation doesn't seem to describe all the possible sections, nor do
we understand what happens if we replace certain aspects of the file --
and whether or not it will remain functional.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_inf: Parse the GptTmpl.inf file which stores security settings

This is NOT an ini file and CANNOT be parsed by Python ConfigParser
without losing information (it would likely eat meaningful whitespace
and so should not be done).

There are three main types of settings:

 * Name,Mode,ACL
 * key = value
 * registry key and value

   Note: This appears as key=value, but registry keys in the general
   case may have = in their names, so we record the entire string in
   order to be as safe as possible.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_csv: Parse the audit.csv file which records audit settings

Based on the setting, the csv will omit certain fields. Using this we
can later infer as to how to generalize the ACLs and SIDs.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_ini: Parse .ini files in SYSVOL

These are fdeploy, scripts + psscripts as well as the GPT.ini at the top
level. Note that GPT.ini has a different character encoding and we
specify it here.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_pol: Parse the .pol files (PReg) which stored winreg settings

Currently, we do not look inside the .pol files for any settings (and do
not generalize any so far).

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Add a restore command (for backups) from XML

Currently because no parsers have been written, this just copies the old
files and puts them in their places.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gpo: Add a backup command (similar to fetch)

The idea behind this command is that you will eventually backup a number
of XML files which can be user-editable and have generic entities to be
later restored in the same domain or a different domain.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

gp_parse: Introduce new module for parsing GPO files

This is the default parser which will cause the file to be restored
as-is -- leaving only an effectively blank XML file as a placeholder.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

preg: Use gensize to allow modification of winreg data to be repacked

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

preg: Using winreg_Data_GPO instead of DATA_BLOB

We need to make a duplicate in order to have reasonable python bindings.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

preg: Unpack winreg_Data for parsing

It seems that there might be pre-existing endianness issues which would be fixed by the ndr_push.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

preg: Build python preg bindings

These will be used in the GPO import/export.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

winreg: Add hyper REG_QWORD to parsing routines

This will be useful when exporting registry.pol files.

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

git: Treat .dump files as binary

This means that git grep will no longer show TDB dumps. This can be
changed at runtime using -a for all to include these files, while -I
will also omit any references to the files (no Binary file * matches).

Signed-off-by: Garming Sam <garming@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

buildtools: Split git ls-files output on newline, not any whitespace

This allows files to have a space in the filename within the Samba git tree.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

netcmd: Fix --kerberos=yes and --no-secrets domain backups

The --kerberos=yes and --no-secrets options didn't work in combination
for domain backups. The problem was creds.get_username() might not
necessarily match the kerberos user (such as in the selftest
environment). If this was the case, then trying to reset the admin
password failed (because the creds.get_username() didn't exist in
the DB).

Because the admin user always has a fixed RID, we can work out the
administrator based on its object SID, instead of relying on the
username in the creds.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13566

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Autobuild-User(master): Andrew Bartlett <abartlet@samba.org>
Autobuild-Date(master): Wed Aug 15 10:19:09 CEST 2018 on sn-devel-144

netcmd: Delete unnecessary function

Minor code cleanup. The last 2 patches gutted this function, to the
point where there's no longer any value in keeping it.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13566

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

netcmd: Fix kerberos option for domain backups

The previous fix still didn't work if you specified --kerberos=yes (in
which case the creds still doesn't have a password).

credopts.get_credentials(lp) should be enough to ensure a user/password
is set (it's all that the other commands seem to do).

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13566

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

netcmd: domain backup didn't support prompting for password

The online/rename backups only worked if you specified both the username
and password in the actual command itself. If you just entered the
username (expecting to be prompted for the password later), then the
command was rejected.

The problem was the order the code was doing things in. We were checking
credopts.creds.get_password() *before* we'd called
credopts.get_credentials(lp), whereas it should be the other way
around.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13566

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

third_party:build: Test for the flags, recognized by Clang.

Make amd64 SYSTEM_UNAME_MACHINE an alias for x86_64.

Signed-off-by: Timur I. Bakeyev <timur@iXsystems.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

emulate/traffic: add sAMAccountName in create_group

While using script/traffic_replay to generate users and groups, we get
autogenerated group name like:

    $2A6F42B2-39FAF4556E2BE379

This patch specify sAMAccountName to overwriten the name.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic-replay: add extra check

Make sure --average-groups-per-user is not more than --number-of-users

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

traffic: uniform stats output

The original code is trying to output different data format for tty or file.
This is unnecessary and cause confusion while writing script to parse result.

The human-readable one is also easy for code to parse.
Remove if check for isatty(), just make output the same.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

emulate/traffic: fix next usage

In commit b0c9de820c07d77c03b80505cb811ac1dac0808f, line 343:

    self.next_conversation_id = itertools.count().next

was changed to:

    self.next_conversation_id = next(itertools.count())

which is not correct, the first one is a function, the second one is a
int. This patch fixed it.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13573

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

samba-tool/drs: set dns_backend to SAMBA_INTERNAL in cmd_drs_clone_dc_database

The default value is "NONE", need to specify it to use SAMBA_INTERNAL so
that the DNS partitions are replicated.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dns_server: Avoid ldb_dn_add_child_fmt() on untrusted input

By using the new ldb_dn_add_child_val() we ensure that the user-controlled values are
not parsed as DN seperators.

Additionally, the casefold DN is obtained before the search to trigger
a full parse of the DN before being handled to the LDB search.

This is not normally required but is done here due to the nature
of the untrusted input.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13466

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

dns_server: Be strict when constructing a LDB DN from an untrusted DNS name

This changes our DNS server to be much more careful when constructing DNS names
into LDB DN values.

This avoids a segfault deep in the LDB code if the ldb_dn_get_casefold() fails there.

A seperate patch will address that part of the issue, and a later patch
will re-work this code to use single API: ldb_dn_add_child_val().  This
is not squahed with this work because this patch does not rely on a new
LDB release, and so may be helpful for a backport.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13466

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldb: Release LDB 1.5.1

* New API ldb_dn_add_child_val() avoids passing untrusted input to
  ldb_dn_add_child_fmt() (bug 13466)
* Free memory nearer to the allocation in calls made by ldbsearch
* Do not overwrite ldb_transaction_commit failure error messages
  with a pointless del_transaction()

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldb: extend API tests

These additional API tests just check that an invalid base DN
is never accepted.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldb: Add new function ldb_dn_add_child_val()

This is safer for untrusted input than ldb_dn_add_child_fmt()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13466

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldb_tdb: Remove pointless check of ldb_dn_is_valid()

If the DN is not valid the ltdb_search_dn1() will catch it with ldb_dn_validate() which
is the only safe way to check this.  ldb_dn_is_valid() does not actually check, but instead
returns only the result of the previous checks, if there was one.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

fix mem leak in ldbsearch

Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

fix mem leak in ltdb_index_dn_base_dn and ltdb_search_indexed

Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

ldb: no need to call del_transaction in ldb_transaction_commit

No matter commit succeeded or failed, transation will be delete afterwards.
So there is no need to delete it here.

Aganst Samba this causes an `LDAP error 51 LDAP_BUSY` error when the transaction
fails, say while we try to add users to groups in large amount and
the original error is lost.

In Samba, the rootdse module fails early in the del part of the
start/end/del pattern, and in ldb_tdb and ldb_mdb a failed commit
always ends the transaction, even on failure.

Signed-off-by: Joe Guo <joeg@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

s3:libads: Free addr before we free the context

Introduced by dbdbd4875ecac3e7334750f46f1f494b7afe6628

CID 1438395

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13567

Signed-off-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Tue Aug 14 22:02:06 CEST 2018 on sn-devel-144

ldb tests: fix assertion on wrong pointer

We are allocating msg02, but check in assertion msg01, which makes no
sense here.

Signed-off-by: Timur I. Bakeyev <timur@freebsd.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Jeremy Allison <jra@samba.org>

cracknames: Fix DoS (NULL pointer de-ref) when not servicePrincipalName is set on a user

This regression was introduced in Samba 4.7 by bug 12842 and in
master git commit eb2e77970e41c1cb62c041877565e939c78ff52d.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13552

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>
Autobuild-User(master): Karolin Seeger <kseeger@samba.org>
Autobuild-Date(master): Tue Aug 14 17:02:38 CEST 2018 on sn-devel-144

libsmb: Harden smbc_readdir_internal() against returns from malicious servers.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13453

CVE-2018-10858: Insufficient input validation on client directory
listing in libsmbclient.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

libsmb: Ensure smbc_urlencode() can't overwrite passed in buffer.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13453

CVE-2018-10858: Insufficient input validation on client directory
listing in libsmbclient.

Signed-off-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Ralph Boehme <slow@samba.org>

CVE-2018-10919 tests: Add extra test for dirsync deleted object corner-case

The acl_read.c code contains a special case to allow dirsync to
work-around having insufficient access rights. We had a concern that
the dirsync module could leak sensitive information for deleted objects.
This patch adds a test-case to prove whether or not this is happening.

The new test case is similar to the existing dirsync test except:
- We make the confidential attribute also preserve-on-delete, so it
  hangs around for deleted objcts. Because the attributes now persist
  across test case runs, I've used a different attribute to normal.
  (Technically, the dirsync search expressions are now specific enough
  that the regular attribute could be used, but it would make things
  quite fragile if someone tried to add a new test case).
- To handle searching for deleted objects, the search expressions are
  now more complicated. Currently dirsync adds an extra-filter to the
  '!' searches to exclude deleted objects, i.e. samaccountname matches
  the test-objects AND the object is not deleted. We now extend this to
  include deleted objects with lastKnownParent equal to the test OU.
  The search expression matches either case so that we can use the same
  expression throughout the test (regardless of whether the object is
  deleted yet or not).

This test proves that the dirsync corner-case does not actually leak
sensitive information on Samba. This is due to a bug in the dirsync
code - when the buggy line is removed, this new test promptly fails.
Test also passes against Windows.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13434

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-10919 acl_read: Fix unauthorized attribute access via searches

A user that doesn't have access to view an attribute can still guess the
attribute's value via repeated LDAP searches. This affects confidential
attributes, as well as ACLs applied to an object/attribute to deny
access.

Currently the code will hide objects if the attribute filter contains an
attribute they are not authorized to see. However, the code still
returns objects as results if confidential attribute is in the search
expression itself, but not in the attribute filter.

To fix this problem we have to check the access rights on the attributes
in the search-tree, as well as the attributes returned in the message.

Points of note:
- I've preserved the existing dirsync logic (the dirsync module code
  suppresses the result as long as the replPropertyMetaData attribute is
  removed). However, there doesn't appear to be any test that highlights
  that this functionality is required for dirsync.
- To avoid this fix breaking the acl.py tests, we need to still permit
  searches like 'objectClass=*', even though we don't have Read Property
  access rights for the objectClass attribute. The logic that Windows
  uses does not appear to be clearly documented, so I've made a best
  guess that seems to mirror Windows behaviour.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13434

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-10919 acl_read: Flip the logic in the dirsync check

This better reflects the special case we're making for dirsync, and gets
rid of a 'if-else' clause.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13434

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-10919 acl_read: Small refactor to aclread_callback()

Flip the dirsync check (to avoid a double negative), and use a helper
boolean variable.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13434

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-10919 acl_read: Split access_mask logic out into helper function

So we can re-use the same logic laster for checking the search-ops.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13434

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-10919 security: Fix checking of object-specific CONTROL_ACCESS rights

An 'Object Access Allowed' ACE that assigned 'Control Access' (CR)
rights to a specific attribute would not actually grant access.

What was happening was the remaining_access mask for the object_tree
nodes would be Read Property (RP) + Control Access (CR). The ACE mapped
to the schemaIDGUID for a given attribute, which would end up being a
child node in the tree. So the CR bit was cleared for a child node, but
not the rest of the tree. We would then check the user had the RP access
right, which it did. However, the RP right was cleared for another node
in the tree, which still had the CR bit set in its remaining_access
bitmap, so Samba would not grant access.

Generally, the remaining_access only ever has one bit set, which means
this isn't a problem normally. However, in the Control Access case there
are 2 separate bits being checked, i.e. RP + CR.

One option to fix this problem would be to clear the remaining_access
for the tree instead of just the node. However, the Windows spec is
actually pretty clear on this: if the ACE has a CR right present, then
you can stop any further access checks.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13434

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-10919 tests: test ldap searches for non-existent attributes.

It is perfectly legal to search LDAP for an attribute that is not part
of the schema.  That part of the query should simply not match.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13434

Signed-off-by: Gary Lockyer <gary@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>

CVE-2018-10919 tests: Add test case for object visibility with limited rights

Currently Samba is a bit disclosive with LDB_OP_PRESENT (i.e.
attribute=*) searches compared to Windows.

All the acl.py tests are based on objectClass=* searches, where Windows
will happily tell a user about objects they have List Contents rights,
but not Read Property rights for. However, if you change the attribute
being searched for, suddenly the objects are no longer visible on
Windows (whereas they are on Samba).

This is a problem, because Samba can tell you about which objects have
confidential attributes, which in itself could be disclosive.

This patch adds a acl.py test-case that highlights this behaviour. The
test passes against Windows but fails against Samba.

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-10919 tests: Add tests for guessing confidential attributes

Adds tests that assert that a confidential attribute cannot be guessed
by an unprivileged user through wildcard DB searches.

The tests basically consist of a set of DB searches/assertions that
get run for:
- basic searches against a confidential attribute
- confidential attributes that get overridden by giving access to the
  user via an ACE (run against a variety of ACEs)
- protecting a non-confidential attribute via an ACL that denies read-
  access (run against a variety of ACEs)
- querying confidential attributes via the dirsync controls

These tests all pass when run against a Windows Dc and all fail against
a Samba DC.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13434

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-10919 security: Add more comments to the object-specific access checks

Reading the spec and then reading the code makes sense, but we could
comment the code more so it makes sense on its own.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13434

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-10919 security: Move object-specific access checks into separate function

Object-specific access checks refer to a specific section of the
MS-ADTS, and the code closely matches the spec. We need to extend this
logic to properly handle the Control-Access Right (CR), so it makes
sense to split the logic out into its own function.

This patch just moves the code, and should not alter the logic (apart
from ading in the boolean grant_access return variable.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13434

Signed-off-by: Tim Beale <timbeale@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-1140 dns: Add a test to trigger the LDB casefolding issue on invalid chars

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13466

Signed-off-by: Kai Blin <kai@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

Release LDB 1.5.0 for CVE-2018-1140

* Security fix for CVE-2018-1140 (NULL pointer de-reference, bug 13374)
* Fix memory leaks and missing error checks (bug 13459, 13471, 13475)

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>

CVE-2018-1140 ldb: Add tests for search add and rename with a bad dn= DN

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374

CVE-2018-1140 ldb_tdb: Check for DN validity in add, rename and search

This ensures we fail with a good error code before an eventual ldb_dn_get_casefold() which
would otherwise fail.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374

CVE-2018-1140 ldb_tdb: Ensure the dn in distinguishedName= is valid before use

ldb_dn_from_ldb_val() does not validate this untrusted input, so a later
call to ldb_dn_get_casefold() can fail if the input is not valid.

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374

CVE-2018-1140 ldb: Check for ldb_dn_get_casefold() failure in ldb_sqlite

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374

CVE-2018-1140 Add NULL check for ldb_dn_get_casefold() in ltdb_index_dn_attr()

Signed-off-by: Andrej Gessel <Andrej.Gessel@janztec.com>
Reviewed-by: Douglas Bagnall <douglas.bagnall@catalyst.net.nz>
Reviewed-by: Andrew Bartlett <abartlet@samba.org>
BUG: https://bugzilla.samba.org/show_bug.cgi?id=13374

CVE-2018-1139 libcli/auth: Do not allow ntlmv1 over SMB1 when it is disabled via "ntlm auth".

This fixes a regression that came in via 00db3aba6cf9ebaafdf39ee2f9c7ba5ec2281ea0.

Found by Vivek Das <vdas@redhat.com> (Red Hat QE).

In order to demonstrate simply run:

smbclient //server/share -U user%password -mNT1 -c quit \
--option="client ntlmv2 auth"=no \
--option="client use spnego"=no

against a server that uses "ntlm auth = ntlmv2-only" (our default
setting).

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13360

CVE-2018-1139: Weak authentication protocol allowed.

Guenther

Pair-Programmed-With: Stefan Metzmacher <metze@samba.org>
Signed-off-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-1139 selftest: verify whether ntlmv1 can be used via SMB1 when it is disabled.

Right now, this test will succeed.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13360

CVE-2018-1139: Weak authentication protocol allowed.

Guenther

Signed-off-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-1139 s3-utils: use enum ntlm_auth_level in ntlm_password_check().

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13360

CVE-2018-1139: Weak authentication protocol allowed.

Guenther

Signed-off-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-1139 libcli/auth: fix debug messages in hash_password_check()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13360

CVE-2018-1139: Weak authentication protocol allowed.

Guenther

Signed-off-by: Guenther Deschner <gd@samba.org>
Reviewed-by: Andreas Schneider <asn@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

CVE-2018-1139 libcli/auth: Add initial tests for ntlm_password_check()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13360

Signed-off-by: Andrew Bartlett <abartlet@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Reviewed-by: Gary Lockyer <gary@catalyst.net.nz>

g_lock: Simplify g_lock_trylock

While chasing a bug in g_lock (not in master) I saw some opportunity to
simplify g_lock_trylock a bit. This is array handling, and array
handling is just extremely error-prone. This *might* be a little less
efficient or large numbers of READ locks, but this remains to be
seen. For now, simplify the code.

First, we make two passes now: One to remove ourselves, and the other
one to search for conflicts. Mixing up both made it pretty hard for me
to follow the code.

Second, I've removed the _mylock and mylock pointer/struct logic and
replaced it with the "mylock.pid.pid != 0 ? &mylock : NULL" when calling
g_lock_store. To me, this focuses the logic whether to add ourselves in
one place instead of spreading it around in the whole routine.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Volker Lendecke <vl@samba.org>
Autobuild-Date(master): Tue Aug 14 11:42:10 CEST 2018 on sn-devel-144

g_lock: Avoid a double call to serverid_exist

If we try to G_LOCK_READ while a G_LOCK_WRITE is active, we do the
serverid_exists call twice. Avoid that.

Signed-off-by: Volker Lendecke <vl@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

selftest: Load time_audit and full_audit modules for all tests

Previously the only test was to load these modules to trigger the
smb_vfs_assert_all_fns check. As these modules just pass through the
calls, they can be loaded for all tests to ensure that the codepaths are
exercised. This would have found the problem in
smb_time_audit_offload_read_recv.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13568

Signed-off-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Mon Aug 13 22:35:20 CEST 2018 on sn-devel-144

s3: vfs: time_audit: fix handling of token_blob in smb_time_audit_offload_read_recv()

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13568

Signed-off-by: Ralph Wuerthner <ralph.wuerthner@de.ibm.com>
Reviewed-by: Christof Schmitt <cs@samba.org>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3:winbind: Fix memory leak in nss_init()

Found by covscan.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13567

Pair-Programmed-With: Justin Stephenson <jstephen@redhat.com>
Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Justin Stephenson <jstephen@redhat.com>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3:registry: Fix possible memory leak in _reg_perfcount_multi_sz_from_tdb()

Found by covscan.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13567

Pair-Programmed-With: Justin Stephenson <jstephen@redhat.com>
Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Justin Stephenson <jstephen@redhat.com>
Reviewed-by: Jeremy Allison <jra@samba.org>
Autobuild-User(master): Jeremy Allison <jra@samba.org>
Autobuild-Date(master): Sat Aug 11 04:43:15 CEST 2018 on sn-devel-144

s3:libads: Fix memory leaks in ads_krb5_chg_password()

Found by covscan.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13567

Pair-Programmed-With: Justin Stephenson <jstephen@redhat.com>
Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Justin Stephenson <jstephen@redhat.com>
Reviewed-by: Jeremy Allison <jra@samba.org>

s3:client: Avoid a possible fd leak in do_get()

Found by covscan.

BUG: https://bugzilla.samba.org/show_bug.cgi?id=13567

Pair-Programmed-With: Justin Stephenson <jstephen@redhat.com>
Signed-off-by: Andreas Schneider <asn@samba.org>
Signed-off-by: Justin Stephenson <jstephen@redhat.com>
Reviewed-by: Jeremy Allison <jra@samba.org>