dan [Tue, 6 Dec 2022 18:48:06 +0000 (18:48 +0000)]
Have sqlite3_stmt_scanstatus_v2() return an NCYCLE value for all loops, not just virtual tables ones. The value returned is the sum of the NCYCLE counts for the various opcodes that move or read data from the table or index cursor associated with the loop.
drh [Tue, 6 Dec 2022 13:12:33 +0000 (13:12 +0000)]
In the unix backend, when implementing the defenses against small file
descriptors, delete a file just created if it was opened with
O_EXCL|O_CREAT so that it can be created again the next time through the
loop. Fix for the problem described by
[forum:/forumpost/699af709ab3a8ccf|forum post 699af709ab3a8ccf].
drh [Mon, 5 Dec 2022 02:42:30 +0000 (02:42 +0000)]
Use the smaller estimated row size for searching IPK tables, but use the
original larger row size estimate for scanning, since the leaves can have large
rows.
stephan [Sat, 3 Dec 2022 13:10:58 +0000 (13:10 +0000)]
Rename wasm.xWrap.resultAdapter() X:free entries to X:dealloc for consistency with wasm.dealloc(). Add an undocumented feature to replace wasm.alloc/dealloc/realloc() with the C-standard allocators (after an allocator misuse led down a several-hour rabbit hole trying to discover a mis-free() violation). Related test updates.
stephan [Fri, 2 Dec 2022 18:06:26 +0000 (18:06 +0000)]
OPFS VFS: translate createSyncAccessHandle() exceptions which appear to be locking violations to SQLITE_BUSY. This seems to improve concurrency considerably even with a reduced retry count of 5 (was 6).
drh [Fri, 2 Dec 2022 17:52:52 +0000 (17:52 +0000)]
For the sqlite3_bind and sqlite3_result interfaces for UTF16 strings, round
the number of bytes down to the next even number, to avoid creating a UTF16
string that is an odd number of bytes.
[forum:/forumpost/411199488d065f83|Forum post 411199488d065f83].
drh [Fri, 2 Dec 2022 15:31:47 +0000 (15:31 +0000)]
Use sqlite3_result_int64() instead of sqlite3_result_int() when returning
potentially large values from the DBSTAT virtual table, to avoid integer
overflows in the result.
[forum:/forumpost/ada2ab044f|Forum post ada2ab044f].
stephan [Fri, 2 Dec 2022 10:43:14 +0000 (10:43 +0000)]
wasm builds: explicitly set a default stack size because emsdk 3.1.27 reduced it from 4MB to only 64kb, leading to memory corruption when kvvfs is used (it requires at least twice that for I/O).
stephan [Fri, 2 Dec 2022 08:29:03 +0000 (08:29 +0000)]
sqlite3-wasm.c: code legibility and coding style tweaks. Increase SQLITE_DEFAULT_PAGE_SIZE from 4k to 8k, as that improves OPFS speedtest1 performance by roughly 12%.
stephan [Fri, 2 Dec 2022 07:14:56 +0000 (07:14 +0000)]
Expand JS tests for db export/import and document reason it cannot currently work with kvvfs. Fix a minor JS build dependencies bug. Update page title with PASS/FAIL prefix for tester1.js to improve overview when launching multiple test tabs. Add ability of tester1 should-run-test predicates to report why a given test is disabled.
stephan [Thu, 1 Dec 2022 15:22:03 +0000 (15:22 +0000)]
sqlite3_js_create_file() now accepts an ArrayBuffer data source. Add test for OPFS-based export/re-import. The (sqlite3*) argument converter now optionally accepts sqlite3.oo1.DB instances.
stephan [Thu, 1 Dec 2022 03:55:28 +0000 (03:55 +0000)]
Expand "sqlite3_vfs*" JS-to-WASM function argument conversions to accept VFS names (JS strings) and capi.sqlite3_vfs instances. Implement sqlite3_js_vfs_create_file() to facilitate creation of file-upload features which store the file in VFS-specific storage (where possible, e.g. "unix" and "opfs" VFSes). Correct an argument type check in the SQLite3Error and WasmAllocError constructors.
stephan [Wed, 30 Nov 2022 18:21:01 +0000 (18:21 +0000)]
Rename some JS files from X.js to X.c-pp.js to keep the maintainer, and downstream build customizers, aware that those files contain constructs specific to the c-pp preprocessor and will not run as-is in JS.
stephan [Wed, 30 Nov 2022 11:50:16 +0000 (11:50 +0000)]
Install sqlite3_malloc/sqlite3_free() as the JS-side WASM allocator (as opposed to replacing C-level's malloc()/free() with them). All tests work and this eliminates the potential for allocator discrepancies when using the (de)serialize APIs.
stephan [Wed, 30 Nov 2022 08:37:17 +0000 (08:37 +0000)]
Simplify how the OPFS VFS async proxy copies initial state sent to it from the synchronous side of the connection. Make the lock-wait time a multiple of the wait-loop timeout interval.
stephan [Wed, 30 Nov 2022 07:17:29 +0000 (07:17 +0000)]
Rename some OPFS JS files. Prevent JS bindings of sqlite3_uri_...() from performing JS-to-C-string argument conversion on their first argument, as doing so is specifically illegal.
stephan [Wed, 30 Nov 2022 05:27:36 +0000 (05:27 +0000)]
Refactor a significant chunk of the OPFS sqlite3_vfs init code into sqlite3.VfsHelper, and internal-use-only API encapsulating code relevant to creating new VFSes in JS. Intended to assist in pending experimentation with an alternative OPFS VFS.
stephan [Tue, 29 Nov 2022 18:28:40 +0000 (18:28 +0000)]
sqlite3.oo1.OpfsDb: default to journal_mode=persist, as current tests show it to be marginally faster than truncate/delete in Chrome v109. Also increase default busy_timeout from 2 seconds to 3, admittedly on a whim.
drh [Tue, 29 Nov 2022 17:52:04 +0000 (17:52 +0000)]
Change the handling of hwtime.h to make it easier to compile performance
measurement builds that make use of hwtime.h. This should not affect
productions builds.
stephan [Tue, 29 Nov 2022 05:25:08 +0000 (05:25 +0000)]
Internal restructuring of the OPFS sqlite3_vfs in order to facilitate certain experimentation and improve error reporting/hints if it cannot be activated. Deprecate the name sqlite3.opfs.OpfsDb, preferring sqlite3.oo1.OpfsDb for consistency with JsStorageDb and any future DB subclasses.
stephan [Tue, 29 Nov 2022 02:23:12 +0000 (02:23 +0000)]
Add an explicit warning about the current API-instability of the sqlite3.opfs namespace, which may need to be eliminated based on re-thinking of how the OPFS sqlite3_vfs is registered. Comment changes only - no code.
drh [Mon, 28 Nov 2022 19:42:48 +0000 (19:42 +0000)]
Implementation of enhancement request [695a1a53dea6b240]: The query planner
has improved awareness of when indexes on expressions are covering and adjusts
their costs accordingly.
drh [Mon, 28 Nov 2022 15:23:53 +0000 (15:23 +0000)]
Rework the covering index checking routine, whereIsCoveringIndex(), so that
it can return a "maybe" result for aggregate queries where we are not exactly
sure. The index is scored as if it is covering, but the main table is
still opened.
stephan [Sun, 27 Nov 2022 00:57:56 +0000 (00:57 +0000)]
Document an OPFS API change in Chrome v108 which does not break our code but does change several formerly async methods to synchronous. No code changes.
stephan [Sat, 26 Nov 2022 15:24:58 +0000 (15:24 +0000)]
OPFS VFS: add the opfs-unlock-asap=1 URI flag which tells the VFS to release implicit locks ASAP instead of during VFS idle time. This improves concurrency notably in the test app but brings a significant performance penalty in speedtest1 (roughly 4x slowdown). This is not the final word in OPFS concurrency, but gets us a step further.
drh [Fri, 25 Nov 2022 17:05:55 +0000 (17:05 +0000)]
Relax restriction (8) on the push-down optimization so that it only applies
if one or more columns uses a collating sequence other than BINARY.
See [forum:/forumpost/3824ced748baa808|forum post 3824ced748baa808] and
check-in [346a3b12b861ce7b].
drh [Fri, 25 Nov 2022 16:32:59 +0000 (16:32 +0000)]
Fix #ifdefs so that restrictions (8) and (9) of the push-down optimization
are still enforced even if compiled with SQLITE_OMIT_WINDOWFUNC. This
fixes a bug introduced by check-in [346a3b12b861ce7b].
drh [Fri, 25 Nov 2022 16:10:48 +0000 (16:10 +0000)]
Immprove the query planner such that it is able to make use of indexed
expressions within an aggregate query with GROUP BY. This implements
enhancement request [99378177930f87bd].
drh [Fri, 25 Nov 2022 15:52:00 +0000 (15:52 +0000)]
Add restriction (9) to the push-down optimization: If the subquery is
a compound then all arms of the compound must have the same affinity.
dbsqlfuzz 3a548de406a50e896c1bf7142692d35d339d697f.