drh [Thu, 2 Jan 2020 14:42:42 +0000 (14:42 +0000)]
Enhancements to aid testing and debugging:
In PRAGMA vdbe_trace=on output, show pScopyFrom dependencies on register
values. Add the sqlite3VdbeRegisterDump() procedure, callable from a
debugger, that shows the values of all registers. Pass the VDBE pointer
into test_trace_breakpoint() so that sqlite3VdbeRegisterDump() is callable
from the breakpoint.
drh [Thu, 2 Jan 2020 13:26:49 +0000 (13:26 +0000)]
Add the test_trace_breakpoint() subroutine that is invoked after each
instruction is printed while running PRAGMA vdbe_trace=on. Only works for
SQLITE_DEBUG builds. Also add parameters "pc" and "pOp" to
test_addop_breakpoint() to make it easier to set conditionals.
drh [Wed, 1 Jan 2020 23:02:35 +0000 (23:02 +0000)]
Provide the -DSQLITE_ENABLE_INTERNAL_FUNCTIONS=1 compile-time option. Fix
the ".testctrl internal_function" command in the CLI so that it does not
signal an error on a valid input.
drh [Wed, 1 Jan 2020 21:14:30 +0000 (21:14 +0000)]
When generating code for a subquery, make a copy of the Select object and
generate the code out of the copy, in case the code generator makes
modifications to expression and the Select object needs to be reused.
dan [Wed, 1 Jan 2020 20:17:15 +0000 (20:17 +0000)]
Ensure that when code for a scalar SELECT featuring window functions is generated more than once by the planner, separate ephemeral tables are opened for each instance.
drh [Wed, 1 Jan 2020 15:43:30 +0000 (15:43 +0000)]
New test-only SQL functions: implies_nonnull_row(), expr_compare(), and
expr_implies_expr(). The SQLITE_TESTCTRL_INTERNAL_FUNCTIONS test-control
is modified to toggle internal function access on and off for a single
database connection.
drh [Tue, 31 Dec 2019 18:12:36 +0000 (18:12 +0000)]
Set the SQLITE_DIRECTONLY flag on the fts3_tokenizer() function of FTS3,
thus preventing that function from being called from within a trigger or
view.
dan [Mon, 30 Dec 2019 06:55:31 +0000 (06:55 +0000)]
In ALTER TABLE, rename columns and tables in expressions that are optimized out by the "AND 0" optimization. Doing this also fixes an otherwise harmless assert() failure.
drh [Sun, 29 Dec 2019 22:08:20 +0000 (22:08 +0000)]
Do not allow triggers that run as part of REPLACE conflict resolution
during an UPDATE to modify the the table being updated. Otherwise, those
triggers might delete content out from under the update operation, leading
to all kinds of problems. Ticket [314cc133e5ada126]
drh [Sun, 29 Dec 2019 00:52:41 +0000 (00:52 +0000)]
Add the OP_FinishSeek opcode which completes an OP_DeferredSeek if the seek
has not already completed. Also add the sqlite3WhereUsesDeferredSeek()
interface to the query planner. The UPDATE implementation adds an
OP_FinishSeek before running the final OP_Insert if one is needed.
Ticket [ec8abb025e78f40c] and also an assertion fault reported by Yongheng.
dan [Sat, 28 Dec 2019 18:08:39 +0000 (18:08 +0000)]
Change an assert() in where.c to a testcase() macro, since the condition may be false. This was a problem with the assert() only, there is no bug in release builds that omit assert().
drh [Sat, 28 Dec 2019 16:20:23 +0000 (16:20 +0000)]
Disable early coding of transitive constraints at the end of each loop in
the WHERE clause processing if the loop being coded is for a LEFT JOIN,
even if the loop is part of an OR-clause optimization for virtual tables.
Test cases in TH3.
dan [Sat, 28 Dec 2019 15:24:02 +0000 (15:24 +0000)]
Fix an instance where the planner might choose to use the OR-optimization when it adds no benefit. The same quirk causes an assert() to fail. This is not a bug in released versions - without the assert() the library still gets the right answer, it just does so less efficiently than it should.
drh [Sat, 28 Dec 2019 14:07:22 +0000 (14:07 +0000)]
New enhancements to .wheretrace. The 0x20000 flag shows the WHERE clause
before and after coding each loop. The 0x800 flag shows status at the start
and at the end of each loop. An extra "C" tag is shown on coded terms.
drh [Sat, 28 Dec 2019 13:17:11 +0000 (13:17 +0000)]
Add the --enable-all option to the main configure script as a short-hand
to enable FTS4, FTS5, Geopoly/Rtree, JSON, and Sessions. In the
amalgamation-autoconf, the --enable-rtree option (which is enabled by
default) also now activates Geopoly.
drh [Sat, 28 Dec 2019 13:01:52 +0000 (13:01 +0000)]
Simplifications to the initialization of the sqlite3_index_info structure
that is used to communicate with virtual table modules. Avoid adding
unused constraints to the sqlite3_index_info structure. Extra constraints
are harmless, but might be confusing to people trying to understand the code.
drh [Sat, 28 Dec 2019 02:40:49 +0000 (02:40 +0000)]
Convert an assert() back into a conditional. The conditional was converted
into an assert() by check-in [6ae4ad6ebee4db88] (2009-05-28) because we were
unable to find a test case for it. Yongheng's fuzzer just now
found that test case.
drh [Sat, 28 Dec 2019 01:52:46 +0000 (01:52 +0000)]
When an INSERT is receiving content from a SELECT, run an OP_ReleaseReg opcode
at the top of each iteration of the loop in order to prevent spurious
OP_SCopy misuse complaints. Ticket [de4b04149b9fdeae]
drh [Fri, 27 Dec 2019 13:30:46 +0000 (13:30 +0000)]
Follow-up to check-in [c8c6dd0e6582ec91] - change the xAccess() method to
return true if the file exists and it is anything other than a regular file,
or if it is a regular file with a non-zero file size.
dan [Fri, 27 Dec 2019 08:57:08 +0000 (08:57 +0000)]
When determining if an aggregate within a sub-query should be processed as part of the sub-query or an outer query, consider any FILTER clause in the same way as the arguments to the aggregate.
drh [Fri, 27 Dec 2019 01:50:46 +0000 (01:50 +0000)]
Ensure that the Pager.changeCountDone flag is cleared whenever dropping
the write lock, even when transitioning from EXCLUSIVE locking mode into
NORMAL locking mode while in WAL mode. Ticket [fb3b3024ea238d5c].
drh [Fri, 27 Dec 2019 00:19:53 +0000 (00:19 +0000)]
Fix buffer underflows in the zipfile extension associated with zero-length
or NULL filename in the ZIP archive. But report on the mailing list by
Yongheng and Rui.
drh [Thu, 26 Dec 2019 23:16:18 +0000 (23:16 +0000)]
An UPDATE of a table that is indexed by a constant virtual column that uses
the one-pass optimization might cause the table seek to be omitted before
reaching row DELETE/INSERT. Fix this by coding an extra OP_Column in that
circumstance. Ticket [ec8abb025e78f40c]
drh [Thu, 26 Dec 2019 00:56:50 +0000 (00:56 +0000)]
In the xAccess() method of the unix VFS, return true if the named object
is a directory, regardless of what stat() reports as the st_size for the
object. Different filesystems report st_size differently for directories.
Problem reported on the mailing list by Stefan Brüns.
drh [Wed, 25 Dec 2019 23:54:21 +0000 (23:54 +0000)]
When the sqlite3WindowRewrite() routine detects and error, have it convert
the SELECT statement into just "SELECT null" so that it does not leave the
parse tree in a goofy state that can cause problems with subsequent code
before the stack has a chance to unwind and report the error.
Ticket [d87336c81c7d0873]
drh [Tue, 24 Dec 2019 21:01:37 +0000 (21:01 +0000)]
Backout the early VTable cursor close change from yesterday, as Yongheng and
Rui have found a test case for which it fails. The new test case is added
to fuzzcheck.
dan [Tue, 24 Dec 2019 14:27:03 +0000 (14:27 +0000)]
Fix a spurious report of corruption that could be made by the fts5 integrity-check in SQLITE_DEBUG builds if the fts5 index contains malformed utf text.
drh [Mon, 23 Dec 2019 20:41:39 +0000 (20:41 +0000)]
Thoroughly reset the rtree cursor at the start of each VFilter operation,
including clearing its cache. This prevents left over pages in the cache
which can cause problems on shutdown after a LEFT JOIN.
Ticket [5eadca17c4dde90c]
drh [Mon, 23 Dec 2019 19:28:34 +0000 (19:28 +0000)]
Do an early close of virtual table cursors to avoid unnecessary cursor
contention in UPDATE for some virtual table implementations.
Ticket [56a74875be799b85]
dan [Mon, 23 Dec 2019 14:20:46 +0000 (14:20 +0000)]
For expressions like (x, y) IN (SELECT ...) where the SELECT uses window-functions, require that all columns on the LHS be indexed before an index can be used. Fix for [d9ed4ebe].
drh [Mon, 23 Dec 2019 03:37:46 +0000 (03:37 +0000)]
Fix the OP_Cast operator so that when casting to TEXT, it always leaves
the result in the encoding of the database. Ticket [0911b5d161b039c6].
Test cases in TH3.
drh [Mon, 23 Dec 2019 02:18:49 +0000 (02:18 +0000)]
Enhance the sqlite3VdbeMemAboutToChange() shallow-copy validation mechanism
by adding the new OP_ReleaseReg opcode to tell MemAboutToChange() that a
range of registers is no longer needed so that the source register can be
freely changed. This is a change to debugging and test builds only and
does not impact release builds. Fix for ticket
[c62c5e58524b204d] and [5ad2aa6921faa1ee]. The previous fix to ticket
[5ad2aa6921faa1ee] is backed out by this change since this change is a better
fix.
drh [Sun, 22 Dec 2019 23:48:36 +0000 (23:48 +0000)]
Change the code generator for the IN operator so that it avoids creating
OP_Eq and OP_Ne opcode with the same P1 and P3 arguments. This enables us
to back out check-in [ddb17d92df194337] and also fix ticket [188f912b51cd802].
drh [Sun, 22 Dec 2019 20:29:25 +0000 (20:29 +0000)]
Make a hard copy of strings in constraint checks prior to applying
OP_RealAffinity, to avoid problems with a pointer accounting assert.
This change is not strictly necessary - the correct answer is obtained
without it and no UB occurs - however the pointer accounting asserts are
useful to prevent other problems so it is a simple matter to bring this
piece into compliance. Ticket [5ad2aa6921faa1ee]
drh [Sun, 22 Dec 2019 20:03:29 +0000 (20:03 +0000)]
When constructing the virtual MATCH term of the WHERE clause for a virtual
table that is in a LEFT JOIN, be sure to set the correct Expr.iRightJoinTable
value. This value does not appear to ever be used, except inside of a single
assert(). But it is good to set it correctly, nevertheless. This fixes
ticket [7929c1efb2d67e98], which as far as I can tell is completely harmless.
drh [Sun, 22 Dec 2019 18:55:04 +0000 (18:55 +0000)]
Fix to the optimization of check-in [a47efb7c8520a011] that reads the values
of expressions used in an index-on-expression directly from the index rather
than recomputing the value. If the expression has a top-level COLLATE or
unlikely() operator, be sure to clear the corresponding flags prior to
converting it into a TK_COLUMN expression. Failure to do this is most likely
harmless in production, but might cause an assertion fault in debugging builds.
Ticket [b0cb8aff561a6dcd]. Test cases in TH3.
drh [Sun, 22 Dec 2019 18:06:49 +0000 (18:06 +0000)]
When parsing a CREATE TABLE from the sqlite_master table, delete the CHECK
constraints if there are any errors, since there might otherwise be attempts
to use those CHECK constraints if PRAGMA writable_schema=ON is set.
This undoes the fix in check-in [ea721b34477ab8b4] for a more general
solution.
dan [Sun, 22 Dec 2019 17:32:25 +0000 (17:32 +0000)]
Ensure sqlite3WindowRewrite() is called on a SELECT statement before any terms aremoved from it as part of IN() clause processing. Fix for [f00d096ca].
drh [Sat, 21 Dec 2019 19:37:09 +0000 (19:37 +0000)]
When creating a new virtual table, ensure that the OP_ParseSchema opcode
processes the correct entry in the sqlite_master table even if there is
a second entry with the same name and table values due to database corruption
and the use of writable_schema=ON. Dbsqlfuzz find.
drh [Sat, 21 Dec 2019 14:09:30 +0000 (14:09 +0000)]
When a corrupt schema is loaded using writable_schema=ON, the CHECK constraints
(or other expressions in the table definition) might not be fully resolved.
Ensure that the code generator can deal with this if the table is subsequently
used in a DML statement. dbsqlfuzz find.
drh [Fri, 20 Dec 2019 22:46:41 +0000 (22:46 +0000)]
Do not try to access a generated column through an index if the collating
sequence for the generated column is non-standard.
Part 2 of ticket [e0a8120553f4b082]