drh [Mon, 5 Dec 2022 02:42:30 +0000 (02:42 +0000)]
Use the smaller estimated row size for searching IPK tables, but use the
original larger row size estimate for scanning, since the leaves can have large
rows.
stephan [Sat, 3 Dec 2022 13:10:58 +0000 (13:10 +0000)]
Rename wasm.xWrap.resultAdapter() X:free entries to X:dealloc for consistency with wasm.dealloc(). Add an undocumented feature to replace wasm.alloc/dealloc/realloc() with the C-standard allocators (after an allocator misuse led down a several-hour rabbit hole trying to discover a mis-free() violation). Related test updates.
stephan [Fri, 2 Dec 2022 18:06:26 +0000 (18:06 +0000)]
OPFS VFS: translate createSyncAccessHandle() exceptions which appear to be locking violations to SQLITE_BUSY. This seems to improve concurrency considerably even with a reduced retry count of 5 (was 6).
drh [Fri, 2 Dec 2022 17:52:52 +0000 (17:52 +0000)]
For the sqlite3_bind and sqlite3_result interfaces for UTF16 strings, round
the number of bytes down to the next even number, to avoid creating a UTF16
string that is an odd number of bytes.
[forum:/forumpost/411199488d065f83|Forum post 411199488d065f83].
drh [Fri, 2 Dec 2022 15:31:47 +0000 (15:31 +0000)]
Use sqlite3_result_int64() instead of sqlite3_result_int() when returning
potentially large values from the DBSTAT virtual table, to avoid integer
overflows in the result.
[forum:/forumpost/ada2ab044f|Forum post ada2ab044f].
stephan [Fri, 2 Dec 2022 10:43:14 +0000 (10:43 +0000)]
wasm builds: explicitly set a default stack size because emsdk 3.1.27 reduced it from 4MB to only 64kb, leading to memory corruption when kvvfs is used (it requires at least twice that for I/O).
stephan [Fri, 2 Dec 2022 08:29:03 +0000 (08:29 +0000)]
sqlite3-wasm.c: code legibility and coding style tweaks. Increase SQLITE_DEFAULT_PAGE_SIZE from 4k to 8k, as that improves OPFS speedtest1 performance by roughly 12%.
stephan [Fri, 2 Dec 2022 07:14:56 +0000 (07:14 +0000)]
Expand JS tests for db export/import and document reason it cannot currently work with kvvfs. Fix a minor JS build dependencies bug. Update page title with PASS/FAIL prefix for tester1.js to improve overview when launching multiple test tabs. Add ability of tester1 should-run-test predicates to report why a given test is disabled.
stephan [Thu, 1 Dec 2022 15:22:03 +0000 (15:22 +0000)]
sqlite3_js_create_file() now accepts an ArrayBuffer data source. Add test for OPFS-based export/re-import. The (sqlite3*) argument converter now optionally accepts sqlite3.oo1.DB instances.
stephan [Thu, 1 Dec 2022 03:55:28 +0000 (03:55 +0000)]
Expand "sqlite3_vfs*" JS-to-WASM function argument conversions to accept VFS names (JS strings) and capi.sqlite3_vfs instances. Implement sqlite3_js_vfs_create_file() to facilitate creation of file-upload features which store the file in VFS-specific storage (where possible, e.g. "unix" and "opfs" VFSes). Correct an argument type check in the SQLite3Error and WasmAllocError constructors.
stephan [Wed, 30 Nov 2022 18:21:01 +0000 (18:21 +0000)]
Rename some JS files from X.js to X.c-pp.js to keep the maintainer, and downstream build customizers, aware that those files contain constructs specific to the c-pp preprocessor and will not run as-is in JS.
stephan [Wed, 30 Nov 2022 11:50:16 +0000 (11:50 +0000)]
Install sqlite3_malloc/sqlite3_free() as the JS-side WASM allocator (as opposed to replacing C-level's malloc()/free() with them). All tests work and this eliminates the potential for allocator discrepancies when using the (de)serialize APIs.
stephan [Wed, 30 Nov 2022 08:37:17 +0000 (08:37 +0000)]
Simplify how the OPFS VFS async proxy copies initial state sent to it from the synchronous side of the connection. Make the lock-wait time a multiple of the wait-loop timeout interval.
stephan [Wed, 30 Nov 2022 07:17:29 +0000 (07:17 +0000)]
Rename some OPFS JS files. Prevent JS bindings of sqlite3_uri_...() from performing JS-to-C-string argument conversion on their first argument, as doing so is specifically illegal.
stephan [Wed, 30 Nov 2022 05:27:36 +0000 (05:27 +0000)]
Refactor a significant chunk of the OPFS sqlite3_vfs init code into sqlite3.VfsHelper, and internal-use-only API encapsulating code relevant to creating new VFSes in JS. Intended to assist in pending experimentation with an alternative OPFS VFS.
stephan [Tue, 29 Nov 2022 18:28:40 +0000 (18:28 +0000)]
sqlite3.oo1.OpfsDb: default to journal_mode=persist, as current tests show it to be marginally faster than truncate/delete in Chrome v109. Also increase default busy_timeout from 2 seconds to 3, admittedly on a whim.
drh [Tue, 29 Nov 2022 17:52:04 +0000 (17:52 +0000)]
Change the handling of hwtime.h to make it easier to compile performance
measurement builds that make use of hwtime.h. This should not affect
productions builds.
stephan [Tue, 29 Nov 2022 05:25:08 +0000 (05:25 +0000)]
Internal restructuring of the OPFS sqlite3_vfs in order to facilitate certain experimentation and improve error reporting/hints if it cannot be activated. Deprecate the name sqlite3.opfs.OpfsDb, preferring sqlite3.oo1.OpfsDb for consistency with JsStorageDb and any future DB subclasses.
stephan [Tue, 29 Nov 2022 02:23:12 +0000 (02:23 +0000)]
Add an explicit warning about the current API-instability of the sqlite3.opfs namespace, which may need to be eliminated based on re-thinking of how the OPFS sqlite3_vfs is registered. Comment changes only - no code.
drh [Mon, 28 Nov 2022 19:42:48 +0000 (19:42 +0000)]
Implementation of enhancement request [695a1a53dea6b240]: The query planner
has improved awareness of when indexes on expressions are covering and adjusts
their costs accordingly.
drh [Mon, 28 Nov 2022 15:23:53 +0000 (15:23 +0000)]
Rework the covering index checking routine, whereIsCoveringIndex(), so that
it can return a "maybe" result for aggregate queries where we are not exactly
sure. The index is scored as if it is covering, but the main table is
still opened.
stephan [Sun, 27 Nov 2022 00:57:56 +0000 (00:57 +0000)]
Document an OPFS API change in Chrome v108 which does not break our code but does change several formerly async methods to synchronous. No code changes.
stephan [Sat, 26 Nov 2022 15:24:58 +0000 (15:24 +0000)]
OPFS VFS: add the opfs-unlock-asap=1 URI flag which tells the VFS to release implicit locks ASAP instead of during VFS idle time. This improves concurrency notably in the test app but brings a significant performance penalty in speedtest1 (roughly 4x slowdown). This is not the final word in OPFS concurrency, but gets us a step further.
drh [Fri, 25 Nov 2022 17:05:55 +0000 (17:05 +0000)]
Relax restriction (8) on the push-down optimization so that it only applies
if one or more columns uses a collating sequence other than BINARY.
See [forum:/forumpost/3824ced748baa808|forum post 3824ced748baa808] and
check-in [346a3b12b861ce7b].
drh [Fri, 25 Nov 2022 16:32:59 +0000 (16:32 +0000)]
Fix #ifdefs so that restrictions (8) and (9) of the push-down optimization
are still enforced even if compiled with SQLITE_OMIT_WINDOWFUNC. This
fixes a bug introduced by check-in [346a3b12b861ce7b].
drh [Fri, 25 Nov 2022 16:10:48 +0000 (16:10 +0000)]
Immprove the query planner such that it is able to make use of indexed
expressions within an aggregate query with GROUP BY. This implements
enhancement request [99378177930f87bd].
drh [Fri, 25 Nov 2022 15:52:00 +0000 (15:52 +0000)]
Add restriction (9) to the push-down optimization: If the subquery is
a compound then all arms of the compound must have the same affinity.
dbsqlfuzz 3a548de406a50e896c1bf7142692d35d339d697f.
drh [Thu, 24 Nov 2022 19:02:40 +0000 (19:02 +0000)]
Take care not to try to add comments to a TK_AGG_COLUMN opcode that does
not have an associated Table object because it is a reference to an indexed
expression.
dan [Thu, 24 Nov 2022 15:32:00 +0000 (15:32 +0000)]
Update multiplex3.test to account for the fact that the multiplexor xDelete method may return an error even if it manages to delete the first chunk of a file.