drh [Sat, 28 Dec 2019 02:40:49 +0000 (02:40 +0000)]
Convert an assert() back into a conditional. The conditional was converted
into an assert() by check-in [6ae4ad6ebee4db88] (2009-05-28) because we were
unable to find a test case for it. Yongheng's fuzzer just now
found that test case.
drh [Sat, 28 Dec 2019 01:52:46 +0000 (01:52 +0000)]
When an INSERT is receiving content from a SELECT, run an OP_ReleaseReg opcode
at the top of each iteration of the loop in order to prevent spurious
OP_SCopy misuse complaints. Ticket [de4b04149b9fdeae]
drh [Fri, 27 Dec 2019 13:30:46 +0000 (13:30 +0000)]
Follow-up to check-in [c8c6dd0e6582ec91] - change the xAccess() method to
return true if the file exists and it is anything other than a regular file,
or if it is a regular file with a non-zero file size.
dan [Fri, 27 Dec 2019 08:57:08 +0000 (08:57 +0000)]
When determining if an aggregate within a sub-query should be processed as part of the sub-query or an outer query, consider any FILTER clause in the same way as the arguments to the aggregate.
drh [Fri, 27 Dec 2019 01:50:46 +0000 (01:50 +0000)]
Ensure that the Pager.changeCountDone flag is cleared whenever dropping
the write lock, even when transitioning from EXCLUSIVE locking mode into
NORMAL locking mode while in WAL mode. Ticket [fb3b3024ea238d5c].
drh [Fri, 27 Dec 2019 00:19:53 +0000 (00:19 +0000)]
Fix buffer underflows in the zipfile extension associated with zero-length
or NULL filename in the ZIP archive. But report on the mailing list by
Yongheng and Rui.
drh [Thu, 26 Dec 2019 23:16:18 +0000 (23:16 +0000)]
An UPDATE of a table that is indexed by a constant virtual column that uses
the one-pass optimization might cause the table seek to be omitted before
reaching row DELETE/INSERT. Fix this by coding an extra OP_Column in that
circumstance. Ticket [ec8abb025e78f40c]
drh [Thu, 26 Dec 2019 00:56:50 +0000 (00:56 +0000)]
In the xAccess() method of the unix VFS, return true if the named object
is a directory, regardless of what stat() reports as the st_size for the
object. Different filesystems report st_size differently for directories.
Problem reported on the mailing list by Stefan BrĂ¼ns.
drh [Wed, 25 Dec 2019 23:54:21 +0000 (23:54 +0000)]
When the sqlite3WindowRewrite() routine detects and error, have it convert
the SELECT statement into just "SELECT null" so that it does not leave the
parse tree in a goofy state that can cause problems with subsequent code
before the stack has a chance to unwind and report the error.
Ticket [d87336c81c7d0873]
drh [Tue, 24 Dec 2019 21:01:37 +0000 (21:01 +0000)]
Backout the early VTable cursor close change from yesterday, as Yongheng and
Rui have found a test case for which it fails. The new test case is added
to fuzzcheck.
dan [Tue, 24 Dec 2019 14:27:03 +0000 (14:27 +0000)]
Fix a spurious report of corruption that could be made by the fts5 integrity-check in SQLITE_DEBUG builds if the fts5 index contains malformed utf text.
drh [Mon, 23 Dec 2019 20:41:39 +0000 (20:41 +0000)]
Thoroughly reset the rtree cursor at the start of each VFilter operation,
including clearing its cache. This prevents left over pages in the cache
which can cause problems on shutdown after a LEFT JOIN.
Ticket [5eadca17c4dde90c]
drh [Mon, 23 Dec 2019 19:28:34 +0000 (19:28 +0000)]
Do an early close of virtual table cursors to avoid unnecessary cursor
contention in UPDATE for some virtual table implementations.
Ticket [56a74875be799b85]
dan [Mon, 23 Dec 2019 14:20:46 +0000 (14:20 +0000)]
For expressions like (x, y) IN (SELECT ...) where the SELECT uses window-functions, require that all columns on the LHS be indexed before an index can be used. Fix for [d9ed4ebe].
drh [Mon, 23 Dec 2019 03:37:46 +0000 (03:37 +0000)]
Fix the OP_Cast operator so that when casting to TEXT, it always leaves
the result in the encoding of the database. Ticket [0911b5d161b039c6].
Test cases in TH3.
drh [Mon, 23 Dec 2019 02:18:49 +0000 (02:18 +0000)]
Enhance the sqlite3VdbeMemAboutToChange() shallow-copy validation mechanism
by adding the new OP_ReleaseReg opcode to tell MemAboutToChange() that a
range of registers is no longer needed so that the source register can be
freely changed. This is a change to debugging and test builds only and
does not impact release builds. Fix for ticket
[c62c5e58524b204d] and [5ad2aa6921faa1ee]. The previous fix to ticket
[5ad2aa6921faa1ee] is backed out by this change since this change is a better
fix.
drh [Sun, 22 Dec 2019 23:48:36 +0000 (23:48 +0000)]
Change the code generator for the IN operator so that it avoids creating
OP_Eq and OP_Ne opcode with the same P1 and P3 arguments. This enables us
to back out check-in [ddb17d92df194337] and also fix ticket [188f912b51cd802].
drh [Sun, 22 Dec 2019 20:29:25 +0000 (20:29 +0000)]
Make a hard copy of strings in constraint checks prior to applying
OP_RealAffinity, to avoid problems with a pointer accounting assert.
This change is not strictly necessary - the correct answer is obtained
without it and no UB occurs - however the pointer accounting asserts are
useful to prevent other problems so it is a simple matter to bring this
piece into compliance. Ticket [5ad2aa6921faa1ee]
drh [Sun, 22 Dec 2019 20:03:29 +0000 (20:03 +0000)]
When constructing the virtual MATCH term of the WHERE clause for a virtual
table that is in a LEFT JOIN, be sure to set the correct Expr.iRightJoinTable
value. This value does not appear to ever be used, except inside of a single
assert(). But it is good to set it correctly, nevertheless. This fixes
ticket [7929c1efb2d67e98], which as far as I can tell is completely harmless.
drh [Sun, 22 Dec 2019 18:55:04 +0000 (18:55 +0000)]
Fix to the optimization of check-in [a47efb7c8520a011] that reads the values
of expressions used in an index-on-expression directly from the index rather
than recomputing the value. If the expression has a top-level COLLATE or
unlikely() operator, be sure to clear the corresponding flags prior to
converting it into a TK_COLUMN expression. Failure to do this is most likely
harmless in production, but might cause an assertion fault in debugging builds.
Ticket [b0cb8aff561a6dcd]. Test cases in TH3.
drh [Sun, 22 Dec 2019 18:06:49 +0000 (18:06 +0000)]
When parsing a CREATE TABLE from the sqlite_master table, delete the CHECK
constraints if there are any errors, since there might otherwise be attempts
to use those CHECK constraints if PRAGMA writable_schema=ON is set.
This undoes the fix in check-in [ea721b34477ab8b4] for a more general
solution.
dan [Sun, 22 Dec 2019 17:32:25 +0000 (17:32 +0000)]
Ensure sqlite3WindowRewrite() is called on a SELECT statement before any terms aremoved from it as part of IN() clause processing. Fix for [f00d096ca].
drh [Sat, 21 Dec 2019 19:37:09 +0000 (19:37 +0000)]
When creating a new virtual table, ensure that the OP_ParseSchema opcode
processes the correct entry in the sqlite_master table even if there is
a second entry with the same name and table values due to database corruption
and the use of writable_schema=ON. Dbsqlfuzz find.
drh [Sat, 21 Dec 2019 14:09:30 +0000 (14:09 +0000)]
When a corrupt schema is loaded using writable_schema=ON, the CHECK constraints
(or other expressions in the table definition) might not be fully resolved.
Ensure that the code generator can deal with this if the table is subsequently
used in a DML statement. dbsqlfuzz find.
drh [Fri, 20 Dec 2019 22:46:41 +0000 (22:46 +0000)]
Do not try to access a generated column through an index if the collating
sequence for the generated column is non-standard.
Part 2 of ticket [e0a8120553f4b082]
drh [Fri, 20 Dec 2019 15:35:56 +0000 (15:35 +0000)]
When computing dependencies on WHERE-clause terms, be sure to take into
account the FILTER clause of aggregate functions. Problem reported by
Manuel Rigger.
dan [Fri, 20 Dec 2019 14:18:29 +0000 (14:18 +0000)]
Fix a potential problem with ALTER TABLE commands on schemas that contains a triggers or view featuring a sub-query that uses an ON clause with a join that is not a LEFT JOIN.
dan [Fri, 20 Dec 2019 13:24:11 +0000 (13:24 +0000)]
Ensure fts5 consistently calls sqlite3_value_text() before sqlite3_value_bytes(). This matters when storing blobs in fts5 tables within utf-16 databases. Fix for [cf36cb4e02].
drh [Thu, 19 Dec 2019 20:37:32 +0000 (20:37 +0000)]
When an error occurs while rewriting the parser tree for window functions
in the sqlite3WindowRewrite() routine, make sure that pParse->nErr is set,
and make sure that this shuts down any subsequent code generation that might
depend on the transformations that were implemented. This fixes a problem
discovered by the Yongheng and Rui fuzzer.
drh [Thu, 19 Dec 2019 15:15:40 +0000 (15:15 +0000)]
Fix the zipfile extension so that INSERT works even if the pathname of
the file being inserted is a NULL. Bug discovered by the
Yongheng and Rui fuzzer.
drh [Wed, 18 Dec 2019 20:51:58 +0000 (20:51 +0000)]
Continue to back away from the LEFT JOIN optimization of check-in [41c27bc0ff1d3135]
by disallowing query flattening if the outer query is DISTINCT. Without this fix,
if an index scan is run on the table within the view on the right-hand side of the
LEFT JOIN, stale result registers might be accessed yielding incorrect results,
and/or an OP_IfNullRow opcode might be invoked on the un-opened table, resulting
in a NULL-pointer dereference. This problem was found by the Yongheng and Rui fuzzer.
drh [Wed, 18 Dec 2019 01:31:21 +0000 (01:31 +0000)]
Remove an assert() in the code generator that can fail due to an invalid
INSERT statement. The same assert() appears again in the VDBE and will fire
there if the problem persists in an SQL statement that does not contain
errors. Gramfuzz find.
drh [Wed, 18 Dec 2019 00:05:50 +0000 (00:05 +0000)]
When processing constant integer values in ORDER BY clauses of window
definitions (see check-in [7e4809eadfe99ebf]) be sure to fully disable
the constant value to avoid an invalid pointer dereference if the expression
is ever duplicated. This fixes a crash report from Yongheng and Rui.
drh [Mon, 16 Dec 2019 16:52:22 +0000 (16:52 +0000)]
When a table is part of a LEFT JOIN and should be a completely NULL row due to
the semantics of a LEFT JOIN, make sure any generated columns on that row
evaluate to NULL. Ticket [3b84b42943644d6f]
drh [Sun, 15 Dec 2019 02:49:32 +0000 (02:49 +0000)]
Ensure that all ON CONFLICT REPLACE indexes are sorted to the end of the list
of indexes for a table, even for weird cases where the same UNIQUE constraint
occurs twice with the ON CONFLICT REPLACE clause only on the second one. This
avoids an out-of-order contraint processing problem that can arise due
to the optimization of check-in [469a62ca33081854].
drh [Sat, 14 Dec 2019 19:55:31 +0000 (19:55 +0000)]
When a generated column expression is initially parsed, if it contains errors
convert the expression to a simple NULL. This prevents additional pieces of
the expression tree that are allocated from lookaside from leaking into the
schema, where lookaside allocations are not allowed.
drh [Sat, 14 Dec 2019 18:08:22 +0000 (18:08 +0000)]
Due to the previous change, the p5 parameter to OP_VColumn no longer ever
contains extraneous bits, so change a testcase() into an assert() to show
as much.
drh [Sat, 14 Dec 2019 17:43:37 +0000 (17:43 +0000)]
Do not set OP_Column flags on the instructions generated by
sqlite3ExprCodeGetColumn() if the opcode generated is not really an
OP_Column, which might happen if the column is virtual.
Fix for ticket [b439bfcfb7deedc6]