]> git.ipfire.org Git - thirdparty/sqlite.git/log
thirdparty/sqlite.git
2 years agoUse the smaller estimated row size for searching IPK tables, but use the qp-tuning
drh [Mon, 5 Dec 2022 02:42:30 +0000 (02:42 +0000)] 
Use the smaller estimated row size for searching IPK tables, but use the
original larger row size estimate for scanning, since the leaves can have large
rows.

FossilOrigin-Name: df3818997b822743ac407dde45c5fd75845ca40f461e31350d86963dffec6cd6

2 years agoIncrease the nominal row size for IPK index lookups slightly, for better
drh [Sat, 3 Dec 2022 19:04:09 +0000 (19:04 +0000)] 
Increase the nominal row size for IPK index lookups slightly, for better
balance.

FossilOrigin-Name: 1a61c500add4a2bfe80c0c691d559cfca166dc5f8262651a58da7ec16a51d430

2 years agoAdd TOTAL rows to the output generated by tool/vdbe_profile.tcl.
drh [Sat, 3 Dec 2022 18:21:30 +0000 (18:21 +0000)] 
Add TOTAL rows to the output generated by tool/vdbe_profile.tcl.

FossilOrigin-Name: 9800586393c9d3b82459ef657620d245a7985ef5fa389b8a9ea633d6a29c7299

2 years agoAdd a test case to show that ticket [e8b674241947eb3b] has been fixed.
drh [Sat, 3 Dec 2022 17:23:29 +0000 (17:23 +0000)] 
Add a test case to show that ticket [e8b674241947eb3b] has been fixed.

FossilOrigin-Name: e20de6d450c2b4d5bde737f625de16ff53262c22ce7aa6917b64f1665170d33f

2 years agoFurther improvements to the estimated cost of sorting. Take into account
drh [Sat, 3 Dec 2022 17:09:15 +0000 (17:09 +0000)] 
Further improvements to the estimated cost of sorting.  Take into account
the number of columns to be sorted.

FossilOrigin-Name: f3290cf83b7c02d17d85d8942954f052b486c370cd5ec732969da9061dc1d19a

2 years agoTuning the query planner by adjusting the weights that predict the relative
drh [Sat, 3 Dec 2022 00:52:21 +0000 (00:52 +0000)] 
Tuning the query planner by adjusting the weights that predict the relative
performance of sorting and index lookup.

FossilOrigin-Name: 9f2806da4d88beceac2e81e05421f00481dd3dd100b096cd2ae6c828adb42ca7

2 years agoMinor internal tweaks to the OPFS VFS. Resolve a missing result code which lead to...
stephan [Fri, 2 Dec 2022 18:56:37 +0000 (18:56 +0000)] 
Minor internal tweaks to the OPFS VFS. Resolve a missing result code which lead to a null deref in xFileSize().

FossilOrigin-Name: 57dd593ef0efa17dfb3a9f4eac36d5b8b879e271de817d8cd94a8c8b56d31870

2 years agoOPFS VFS: translate createSyncAccessHandle() exceptions which appear to be locking...
stephan [Fri, 2 Dec 2022 18:06:26 +0000 (18:06 +0000)] 
OPFS VFS: translate createSyncAccessHandle() exceptions which appear to be locking violations to SQLITE_BUSY. This seems to improve concurrency considerably even with a reduced retry count of 5 (was 6).

FossilOrigin-Name: 0d36021d107d3afca190ad61c3380536ad0cc2d493d345d48f9f9c1191741128

2 years agoFor the sqlite3_bind and sqlite3_result interfaces for UTF16 strings, round
drh [Fri, 2 Dec 2022 17:52:52 +0000 (17:52 +0000)] 
For the sqlite3_bind and sqlite3_result interfaces for UTF16 strings, round
the number of bytes down to the next even number, to avoid creating a UTF16
string that is an odd number of bytes.
[forum:/forumpost/411199488d065f83|Forum post 411199488d065f83].

FossilOrigin-Name: b57e3c3db00a6bc6db20c82530479f9eba7e37b731f0da6fe81682e84c7ac916

2 years agoUse sqlite3_result_int64() instead of sqlite3_result_int() when returning
drh [Fri, 2 Dec 2022 15:31:47 +0000 (15:31 +0000)] 
Use sqlite3_result_int64() instead of sqlite3_result_int() when returning
potentially large values from the DBSTAT virtual table, to avoid integer
overflows in the result.
[forum:/forumpost/ada2ab044f|Forum post ada2ab044f].

FossilOrigin-Name: 5652154a8c93cf3b1ff6c2e55e94abbe995b0bb625f733461df20e006c2f13f8

2 years agoMinor touchups to the JS test index page and test server push rules.
stephan [Fri, 2 Dec 2022 11:35:21 +0000 (11:35 +0000)] 
Minor touchups to the JS test index page and test server push rules.

FossilOrigin-Name: 0881f3e92364b3dc81443220451e853c0763efaf97653a6348bc39bd9bdb23ad

2 years agowasm builds: explicitly set a default stack size because emsdk 3.1.27 reduced it...
stephan [Fri, 2 Dec 2022 10:43:14 +0000 (10:43 +0000)] 
wasm builds: explicitly set a default stack size because emsdk 3.1.27 reduced it from 4MB to only 64kb, leading to memory corruption when kvvfs is used (it requires at least twice that for I/O).

FossilOrigin-Name: 758112460fb624198a144ed7d8d2324497fa72a2eee80010bd22b621e064b389

2 years agowasm: after building snapshot zip file, emit instructions for pushing it to the test...
stephan [Fri, 2 Dec 2022 09:23:38 +0000 (09:23 +0000)] 
wasm: after building snapshot zip file, emit instructions for pushing it to the test server.

FossilOrigin-Name: 9615c77919fee60d708d72729c741b373a89900aabc318e0ceb4154638cdf339

2 years agoCorrect the problem which triggered the rollback in [7eec635562f6]: an incorrect...
stephan [Fri, 2 Dec 2022 08:51:22 +0000 (08:51 +0000)] 
Correct the problem which triggered the rollback in [7eec635562f6]: an incorrect default db page size (not a multiple of 512 bytes).

FossilOrigin-Name: e06e490c240aa56b616e6f0380b5d08abf06f35f9e683e5150c5ac464eae9e55

2 years agoRoll back the SQLITE_DEFAULT_PAGE_SIZE part of [c260895faacb34] because kvvfs does...
stephan [Fri, 2 Dec 2022 08:38:04 +0000 (08:38 +0000)] 
Roll back the SQLITE_DEFAULT_PAGE_SIZE part of [c260895faacb34] because kvvfs does not work at all with a page size of 8kb.

FossilOrigin-Name: 7eec635562f65592121d7ff0a31a7b4ff6cdfe38e657f326302ef851b3b37c3f

2 years agosqlite3-wasm.c: code legibility and coding style tweaks. Increase SQLITE_DEFAULT_PAGE...
stephan [Fri, 2 Dec 2022 08:29:03 +0000 (08:29 +0000)] 
sqlite3-wasm.c: code legibility and coding style tweaks. Increase SQLITE_DEFAULT_PAGE_SIZE from 4k to 8k, as that improves OPFS speedtest1 performance by roughly 12%.

FossilOrigin-Name: c260895faacb3458c557778630756d02a8520c0f1864bddcf86cdd27ef4a42bd

2 years agoExpand JS tests for db export/import and document reason it cannot currently work...
stephan [Fri, 2 Dec 2022 07:14:56 +0000 (07:14 +0000)] 
Expand JS tests for db export/import and document reason it cannot currently work with kvvfs. Fix a minor JS build dependencies bug. Update page title with PASS/FAIL prefix for tester1.js to improve overview when launching multiple test tabs. Add ability of tester1 should-run-test predicates to report why a given test is disabled.

FossilOrigin-Name: 75f610d3a4cf3d972220f9abc27cdf5990451e3835ceb9cf66973934004dfc5c

2 years agoRemove extraneous/unused sqlite3.oo1.version object. Add httpd makefile target.
stephan [Fri, 2 Dec 2022 03:37:49 +0000 (03:37 +0000)] 
Remove extraneous/unused sqlite3.oo1.version object. Add httpd makefile target.

FossilOrigin-Name: 8e4d30ac033a6d9019a7eeedfe788dc0120f565cef2ae8f09d2bf32eb94d8a33

2 years agosqlite3_js_create_file() now accepts an ArrayBuffer data source. Add test for OPFS...
stephan [Thu, 1 Dec 2022 15:22:03 +0000 (15:22 +0000)] 
sqlite3_js_create_file() now accepts an ArrayBuffer data source. Add test for OPFS-based export/re-import. The (sqlite3*) argument converter now optionally accepts sqlite3.oo1.DB instances.

FossilOrigin-Name: 14a84b67fb17e16a5691ea4bf7f374123ac73a361a5d3d0efca53788d2001e3a

2 years agoReformulate and simplify some JS tests related to the previous checkin.
stephan [Thu, 1 Dec 2022 04:45:51 +0000 (04:45 +0000)] 
Reformulate and simplify some JS tests related to the previous checkin.

FossilOrigin-Name: 9ea2d3dcf798393a7fd231e199c0e2c6302949fe2a7f2573178fb0e50c78a2f4

2 years agoExpand "sqlite3_vfs*" JS-to-WASM function argument conversions to accept VFS names...
stephan [Thu, 1 Dec 2022 03:55:28 +0000 (03:55 +0000)] 
Expand "sqlite3_vfs*" JS-to-WASM function argument conversions to accept VFS names (JS strings) and capi.sqlite3_vfs instances. Implement sqlite3_js_vfs_create_file() to facilitate creation of file-upload features which store the file in VFS-specific storage (where possible, e.g. "unix" and "opfs" VFSes). Correct an argument type check in the SQLite3Error and WasmAllocError constructors.

FossilOrigin-Name: e1009b16d351b23676ad7bffab0c91b373a92132eb855c9af61991b50cd237ed

2 years agoAdd a testcase() macro to verify that the case of a NOT NULL error message
drh [Wed, 30 Nov 2022 21:18:23 +0000 (21:18 +0000)] 
Add a testcase() macro to verify that the case of a NOT NULL error message
hitting the string length limit.

FossilOrigin-Name: 91f50964c10fb12d889bda7d597d8edf475d97d2d8b534b4400e0fed1d753c6a

2 years agoFix an over-zealous assert() reported by Yong Heng.
drh [Wed, 30 Nov 2022 20:22:31 +0000 (20:22 +0000)] 
Fix an over-zealous assert() reported by Yong Heng.

FossilOrigin-Name: 6ee61f8cede4998f0c838d6506b058c6b09f34b3d7f30ed296100785c93f8d00

2 years agoAttempt to rationalize the bits associated with ".wheretrace". Provide
drh [Wed, 30 Nov 2022 19:11:31 +0000 (19:11 +0000)] 
Attempt to rationalize the bits associated with ".wheretrace".  Provide
a decoder key in sqliteInt.h for what each bit is intended to do.

FossilOrigin-Name: 8ec361695a107a94f2cf6a7fe509656a99d85d49bd7c74133c69903f059a2675

2 years agoRename some JS files from X.js to X.c-pp.js to keep the maintainer, and downstream...
stephan [Wed, 30 Nov 2022 18:21:01 +0000 (18:21 +0000)] 
Rename some JS files from X.js to X.c-pp.js to keep the maintainer, and downstream build customizers, aware that those files contain constructs specific to the c-pp preprocessor and will not run as-is in JS.

FossilOrigin-Name: 2eade7c7b17a186735c72974c11a34798a08364861d0f307e897ba765c0a93c7

2 years agoAlways use nanosleep() (instead of usleep() or sleep) if the
drh [Wed, 30 Nov 2022 13:44:31 +0000 (13:44 +0000)] 
Always use nanosleep() (instead of usleep() or sleep) if the
_POSIX_C_SOURCE macro says it should be available.

FossilOrigin-Name: 6620c57b9d3eae7226a412318b43393196df069b5b90aae0cf1743fdd2d102dd

2 years agoInstall sqlite3_malloc/sqlite3_free() as the JS-side WASM allocator (as opposed to...
stephan [Wed, 30 Nov 2022 11:50:16 +0000 (11:50 +0000)] 
Install sqlite3_malloc/sqlite3_free() as the JS-side WASM allocator (as opposed to replacing C-level's malloc()/free() with them). All tests work and this eliminates the potential for allocator discrepancies when using the (de)serialize APIs.

FossilOrigin-Name: 95c78f6b46e0d8efa4313061f47677479f48610b7a7261dc8d0fb1859aca2ad9

2 years agoSimplify how the OPFS VFS async proxy copies initial state sent to it from the synchr...
stephan [Wed, 30 Nov 2022 08:37:17 +0000 (08:37 +0000)] 
Simplify how the OPFS VFS async proxy copies initial state sent to it from the synchronous side of the connection. Make the lock-wait time a multiple of the wait-loop timeout interval.

FossilOrigin-Name: eddafafffa634a42ceeed70aa3fc58be130527612157a4bf4ff9e65c7f6dc26c

2 years agoOPFS VFS: remove an invalid TODO and fix a property name typo which caused xCheckRese...
stephan [Wed, 30 Nov 2022 07:48:35 +0000 (07:48 +0000)] 
OPFS VFS: remove an invalid TODO and fix a property name typo which caused xCheckReservedLock() to always report false.

FossilOrigin-Name: 3b037caa2fa07b6c44c485574e9e5dc71f4a8e82bc902c1321bb0b918b139c74

2 years agoRename some OPFS JS files. Prevent JS bindings of sqlite3_uri_...() from performing...
stephan [Wed, 30 Nov 2022 07:17:29 +0000 (07:17 +0000)] 
Rename some OPFS JS files. Prevent JS bindings of sqlite3_uri_...() from performing JS-to-C-string argument conversion on their first argument, as doing so is specifically illegal.

FossilOrigin-Name: 79832808de2cbdba140ed9e0558f1502b51d131ab4315265315922cda7b748cb

2 years agoRefactor a significant chunk of the OPFS sqlite3_vfs init code into sqlite3.VfsHelper...
stephan [Wed, 30 Nov 2022 05:27:36 +0000 (05:27 +0000)] 
Refactor a significant chunk of the OPFS sqlite3_vfs init code into sqlite3.VfsHelper, and internal-use-only API encapsulating code relevant to creating new VFSes in JS. Intended to assist in pending experimentation with an alternative OPFS VFS.

FossilOrigin-Name: e25d7b080a807e35b32cb885ea75b384130e5c6e936dfef783c5b45d9bfe77d8

2 years agoDoc and logging text tweaks in the OPFS async proxy and test app.
stephan [Wed, 30 Nov 2022 03:08:50 +0000 (03:08 +0000)] 
Doc and logging text tweaks in the OPFS async proxy and test app.

FossilOrigin-Name: 7ce8608e221924d2c7067687eb6eef0f3cab181d5b4132e55a67d8514b6ce94b

2 years agosqlite3.oo1.OpfsDb: default to journal_mode=persist, as current tests show it to...
stephan [Tue, 29 Nov 2022 18:28:40 +0000 (18:28 +0000)] 
sqlite3.oo1.OpfsDb: default to journal_mode=persist, as current tests show it to be marginally faster than truncate/delete in Chrome v109. Also increase default busy_timeout from 2 seconds to 3, admittedly on a whim.

FossilOrigin-Name: d0c8fa30a31c691bc1be5e98d806eeb1e23a8fc6cd54d87e5c1b720aa936e707

2 years agoOPFS speedtest1: hide a currently-broken/to-fix WASMFS-build link.
stephan [Tue, 29 Nov 2022 18:00:01 +0000 (18:00 +0000)] 
OPFS speedtest1: hide a currently-broken/to-fix WASMFS-build link.

FossilOrigin-Name: cde95d382f8debcbca27c4aaf08470ffda35ab10d723a887786669367590ad3c

2 years agoChange the handling of hwtime.h to make it easier to compile performance
drh [Tue, 29 Nov 2022 17:52:04 +0000 (17:52 +0000)] 
Change the handling of hwtime.h to make it easier to compile performance
measurement builds that make use of hwtime.h.  This should not affect
productions builds.

FossilOrigin-Name: f64a224244743ab121371abd516fccbfc93c110e0952211764bd1b217e792c1b

2 years agoFix legacy build system main.mk to include basexx.c in testfixture builds.
dan [Tue, 29 Nov 2022 11:28:45 +0000 (11:28 +0000)] 
Fix legacy build system main.mk to include basexx.c in testfixture builds.

FossilOrigin-Name: 8c9200b7e156206c4270ff60631c57d5898e2769f7f68294ce652bfec4cfa7c2

2 years agoRename one instance of opfs.OpfsDb to oo1.OpfsDb, as per [0c5c51f4fb04].
stephan [Tue, 29 Nov 2022 06:56:09 +0000 (06:56 +0000)] 
Rename one instance of opfs.OpfsDb to oo1.OpfsDb, as per [0c5c51f4fb04].

FossilOrigin-Name: 75b04c9b302ec66749d8f072a49e61de9640b868bec99eadeb5b4bdef354c336

2 years agoMinor internal cleanups and docs in the OPFS sqlite3_vfs.
stephan [Tue, 29 Nov 2022 06:09:32 +0000 (06:09 +0000)] 
Minor internal cleanups and docs in the OPFS sqlite3_vfs.

FossilOrigin-Name: 61799b05ff232c2ac349169c27bfe7f8d9277366093b0c9dd2739828993b3066

2 years agoInternal restructuring of the OPFS sqlite3_vfs in order to facilitate certain experim...
stephan [Tue, 29 Nov 2022 05:25:08 +0000 (05:25 +0000)] 
Internal restructuring of the OPFS sqlite3_vfs in order to facilitate certain experimentation and improve error reporting/hints if it cannot be activated. Deprecate the name sqlite3.opfs.OpfsDb, preferring sqlite3.oo1.OpfsDb for consistency with JsStorageDb and any future DB subclasses.

FossilOrigin-Name: 0c5c51f4fb04a4b90c50ec9704cfea9a3fb7d7d0ee55c1b0d4476129188217a6

2 years agoAdd an explicit warning about the current API-instability of the sqlite3.opfs namespa...
stephan [Tue, 29 Nov 2022 02:23:12 +0000 (02:23 +0000)] 
Add an explicit warning about the current API-instability of the sqlite3.opfs namespace, which may need to be eliminated based on re-thinking of how the OPFS sqlite3_vfs is registered. Comment changes only - no code.

FossilOrigin-Name: 0cb2fd14179397051a25d066256a553fc198656d5668c7010c016f2b8f495bf4

2 years agoCheck-in [8d5b76593d82b3a5] contained an error that was causing some obscure
drh [Mon, 28 Nov 2022 21:17:30 +0000 (21:17 +0000)] 
Check-in [8d5b76593d82b3a5] contained an error that was causing some obscure
error codes to be lost.  Fixed here.

FossilOrigin-Name: 46cdd3637d6a206ad2bcf8653cc6f2c7a886a16cc7685c45967938609941a755

2 years agoGet ext/misc/basexx.c into the testfixture.exe build for MSC.
larrybr [Mon, 28 Nov 2022 21:09:49 +0000 (21:09 +0000)] 
Get ext/misc/basexx.c into the testfixture.exe build for MSC.

FossilOrigin-Name: bb2c5d088e4784f6763eb0ea2a8542ad2129529f181fa92963c7231d7a68f25c

2 years agoFix harmless compiler warnings.
drh [Mon, 28 Nov 2022 20:08:15 +0000 (20:08 +0000)] 
Fix harmless compiler warnings.

FossilOrigin-Name: 8d5b76593d82b3a57bc904ced33c24cd49369868b5dd2fe7f2c7b114f5aee2f6

2 years agoImplementation of enhancement request [695a1a53dea6b240]: The query planner
drh [Mon, 28 Nov 2022 19:42:48 +0000 (19:42 +0000)] 
Implementation of enhancement request [695a1a53dea6b240]: The query planner
has improved awareness of when indexes on expressions are covering and adjusts
their costs accordingly.

FossilOrigin-Name: e3474d79b27298e96e7686e5a6f9a8e736b8a6568a0a08fb1abe5bcca038ca63

2 years agoAdd the SQLITE_FCNTL_RESET_CACHE verb. Use it to ensure that the page cache is purged...
dan [Mon, 28 Nov 2022 18:41:41 +0000 (18:41 +0000)] 
Add the SQLITE_FCNTL_RESET_CACHE verb. Use it to ensure that the page cache is purged before and after a the recovery extension is run.

FossilOrigin-Name: 6db0bc4bc0d272b610bef2aeeae43f539ed6e7cc0a9cc767d5af85ecb0019d5f

2 years agoTest cases added derived from the enhancment ticket. covering-indexed-expr
drh [Mon, 28 Nov 2022 18:17:42 +0000 (18:17 +0000)] 
Test cases added derived from the enhancment ticket.

FossilOrigin-Name: c022c0152ad61a4f56e36f4062609073e2273fbf6f826c20652159be229c2d46

2 years agoMerge trunk changes into covering-index-expr branch.
drh [Mon, 28 Nov 2022 16:11:10 +0000 (16:11 +0000)] 
Merge trunk changes into covering-index-expr branch.

FossilOrigin-Name: 89d775ada39aed4dc532374ace02156d07dc06e8ae54a194608af0c5a582d20f

2 years agoRework the covering index checking routine, whereIsCoveringIndex(), so that
drh [Mon, 28 Nov 2022 15:23:53 +0000 (15:23 +0000)] 
Rework the covering index checking routine, whereIsCoveringIndex(), so that
it can return a "maybe" result for aggregate queries where we are not exactly
sure.  The index is scored as if it is covering, but the main table is
still opened.

FossilOrigin-Name: b8eec4214363192e6f3e12b3faa5810d8269a5fdaecab3ec09b02e5002cf798a

2 years agoAdd (optional) base64 and base85 UDF extensions.
larrybr [Mon, 28 Nov 2022 14:51:50 +0000 (14:51 +0000)] 
Add (optional) base64 and base85 UDF extensions.

FossilOrigin-Name: b44ab10c49bc2895483a9d40813be3798710ee713cc4bf04e449dce55a68452a

2 years agoSync w/trunk, zap surplus space. base_convert
larrybr [Mon, 28 Nov 2022 14:11:48 +0000 (14:11 +0000)] 
Sync w/trunk, zap surplus space.

FossilOrigin-Name: b8345630a2a322234bda49ee4b996f6ba20e2b080621e229a2ec5e820892a663

2 years agoEnsure the RETURNING clause is honoured when a row of a temp table is updated by...
dan [Mon, 28 Nov 2022 13:47:27 +0000 (13:47 +0000)] 
Ensure the RETURNING clause is honoured when a row of a temp table is updated by an ON CONFLICT clause.

FossilOrigin-Name: a2449bcc2c71d0f4c3289621fbf1cb97f0f407c9f7b5bf18245b7854a07c6cfa

2 years agoConform CLI .trace arg handling to its help.
larrybr [Mon, 28 Nov 2022 02:28:44 +0000 (02:28 +0000)] 
Conform CLI .trace arg handling to its help.

FossilOrigin-Name: 31546ea320e8daa020f8a9f491718ffc3dde0e32954018afed88e8494a6aff6c

2 years agoDocument an OPFS API change in Chrome v108 which does not break our code but does...
stephan [Sun, 27 Nov 2022 00:57:56 +0000 (00:57 +0000)] 
Document an OPFS API change in Chrome v108 which does not break our code but does change several formerly async methods to synchronous. No code changes.

FossilOrigin-Name: e9dd87e28a96eb6560943b9a2cb3cb0ff14698c96fe63865944dcfed73b67bd3

2 years agoBetter reporting of when the WHERE clause analysis thinks that an index
drh [Sat, 26 Nov 2022 20:52:38 +0000 (20:52 +0000)] 
Better reporting of when the WHERE clause analysis thinks that an index
is covering.

FossilOrigin-Name: 17ebcf316b91042c823eea7bb6f1309325023cb5c70538cdb2ce932caee2ef05

2 years agoShow which AggInfo.aCol an Expr node represents in the tree trace debugging
drh [Sat, 26 Nov 2022 20:48:56 +0000 (20:48 +0000)] 
Show which AggInfo.aCol an Expr node represents in the tree trace debugging
logic.  Does not affect production builds.

FossilOrigin-Name: b8076f4dc55810855f668dfd4221bdb555cc78c61df200cf0cb87eb137e22850

2 years agoMerge trunk fixes into the covering-indexed-expr branch.
drh [Sat, 26 Nov 2022 20:13:54 +0000 (20:13 +0000)] 
Merge trunk fixes into the covering-indexed-expr branch.

FossilOrigin-Name: 9ac67ff968e874cf955e46e3993e3215c766feec3d5bdd38d77884eedd86b59e

2 years agoFix an incorrect assertion-fault in the "TREETRACE" debugging logic on
drh [Sat, 26 Nov 2022 20:12:02 +0000 (20:12 +0000)] 
Fix an incorrect assertion-fault in the "TREETRACE" debugging logic on
select.c.  The problem does not affect production builds.

FossilOrigin-Name: f11bb42292d94e7b7687bd2134f72afe1353182238cb4837fc0a6d78a021dd7e

2 years agoOPFS VFS: add the opfs-unlock-asap=1 URI flag which tells the VFS to release implicit...
stephan [Sat, 26 Nov 2022 15:24:58 +0000 (15:24 +0000)] 
OPFS VFS: add the opfs-unlock-asap=1 URI flag which tells the VFS to release implicit locks ASAP instead of during VFS idle time. This improves concurrency notably in the test app but brings a significant performance penalty in speedtest1 (roughly 4x slowdown). This is not the final word in OPFS concurrency, but gets us a step further.

FossilOrigin-Name: 9542f9ce9e023b489e2d93661f719fb0751c1e28f72fded9d3c2156d5777e7b1

2 years agoExperimental changes to help the query planner detect when an expression
drh [Sat, 26 Nov 2022 14:19:47 +0000 (14:19 +0000)] 
Experimental changes to help the query planner detect when an expression
index is coverting.  Works somewhat, but there are tests that fail.

FossilOrigin-Name: 968c189bcf29a9b25305251a58d09b7d52ab9dd08f5057dc3ab1f7ad1a5316a0

2 years agoRelax restriction (8) on the push-down optimization so that it only applies
drh [Fri, 25 Nov 2022 17:05:55 +0000 (17:05 +0000)] 
Relax restriction (8) on the push-down optimization so that it only applies
if one or more columns uses a collating sequence other than BINARY.
See [forum:/forumpost/3824ced748baa808|forum post 3824ced748baa808] and
check-in [346a3b12b861ce7b].

FossilOrigin-Name: adbca3448e2099f0d6149a073978f230ed9a92a2f384779879ef89e672231bcf

2 years agoFix #ifdefs so that restrictions (8) and (9) of the push-down optimization
drh [Fri, 25 Nov 2022 16:32:59 +0000 (16:32 +0000)] 
Fix #ifdefs so that restrictions (8) and (9) of the push-down optimization
are still enforced even if compiled with SQLITE_OMIT_WINDOWFUNC.  This
fixes a bug introduced by check-in [346a3b12b861ce7b].

FossilOrigin-Name: 09e1e42e0ff26f9a71cbd128169f060a66425828d637bf8f781490ca38d99103

2 years agoImmprove the query planner such that it is able to make use of indexed
drh [Fri, 25 Nov 2022 16:10:48 +0000 (16:10 +0000)] 
Immprove the query planner such that it is able to make use of indexed
expressions within an aggregate query with GROUP BY.  This implements
enhancement request [99378177930f87bd].

FossilOrigin-Name: b9190d3da70c41717eb188474fd225ee43d0b46646e1b03de5967bd332553870

2 years agoAdd restriction (9) to the push-down optimization: If the subquery is
drh [Fri, 25 Nov 2022 15:52:00 +0000 (15:52 +0000)] 
Add restriction (9) to the push-down optimization:  If the subquery is
a compound then all arms of the compound must have the same affinity.
dbsqlfuzz 3a548de406a50e896c1bf7142692d35d339d697f.

FossilOrigin-Name: 1ad41840c5e0fa702ba2c0df77a3ea126bd695b910b5d1271fa3129c38c58f5f

2 years agoRemove a redundant assert() statement. agg-with-indexed-expr
drh [Fri, 25 Nov 2022 13:26:51 +0000 (13:26 +0000)] 
Remove a redundant assert() statement.

FossilOrigin-Name: e3e1b453dc47884ddc9e51c4302fa2f4d40bca3d69ac7c13d8e2ae1adc81ac56

2 years agoImprove the robustness of the updateAccumulator() routine against OOM.
drh [Fri, 25 Nov 2022 13:15:48 +0000 (13:15 +0000)] 
Improve the robustness of the updateAccumulator() routine against OOM.

FossilOrigin-Name: d2844f577b69fdc16a0a2568c0958fc3d8aff33e9a0ef80e0f58e92f01097432

2 years agoImproved comments. Add assert()s to verify that the AggInfo structure
drh [Fri, 25 Nov 2022 13:08:20 +0000 (13:08 +0000)] 
Improved comments.  Add assert()s to verify that the AggInfo structure
is unchanged after registers have been assigned.

FossilOrigin-Name: 5200b84195ee1ccaa387f7032eae3d463724c48cb53ba0251bbc79e927dd9752

2 years agoCorrectly deal with IF-NULL-ROW operators when dealing with
drh [Thu, 24 Nov 2022 23:35:27 +0000 (23:35 +0000)] 
Correctly deal with IF-NULL-ROW operators when dealing with
indexed expressions in aggregates.

FossilOrigin-Name: 939cb47025354e2df047de7654c0b06f791957cfe4e904abe8892207cea90215

2 years agoSpeed up base85() conversions and sync w/trunk.
larrybr [Thu, 24 Nov 2022 20:11:34 +0000 (20:11 +0000)] 
Speed up base85() conversions and sync w/trunk.

FossilOrigin-Name: 17b823500a2ed135c1f40aa7f4d87ba5b2eab35c0abd9e0856041cf0f510cbee

2 years agoTake care not to try to add comments to a TK_AGG_COLUMN opcode that does
drh [Thu, 24 Nov 2022 19:02:40 +0000 (19:02 +0000)] 
Take care not to try to add comments to a TK_AGG_COLUMN opcode that does
not have an associated Table object because it is a reference to an indexed
expression.

FossilOrigin-Name: 5fc23863e4a6388884ce061cd69546757b4c39d731bbb58a8121b92fd5144f7b

2 years agoChange a NEVER() into an assert().
drh [Thu, 24 Nov 2022 18:45:56 +0000 (18:45 +0000)] 
Change a NEVER() into an assert().

FossilOrigin-Name: a0fd44f4e5a0ec83465203a0009f307ca50223833575895a3ebf8a289515714f

2 years agoFix a test case in fts3expr4.test to account for different locales.
dan [Thu, 24 Nov 2022 17:58:55 +0000 (17:58 +0000)] 
Fix a test case in fts3expr4.test to account for different locales.

FossilOrigin-Name: a2b6883ac2ef878f525ee847b170beb9f9ab9d1fa67557101be2cdae1e7f7a57

2 years agoMore work on the OPFS concurrency testing app. opfs-unlock-asap
stephan [Thu, 24 Nov 2022 17:53:09 +0000 (17:53 +0000)] 
More work on the OPFS concurrency testing app.

FossilOrigin-Name: c0458caca3508d5d252f9b5198bda4f51a5c1874540f014b17e409f2daab1706

2 years agoJS documentation cleanups. No code changes.
stephan [Thu, 24 Nov 2022 16:03:49 +0000 (16:03 +0000)] 
JS documentation cleanups. No code changes.

FossilOrigin-Name: 18e89a436daa18a8c972caf06b298da43c97a6ea3e2e5229dccb6920c27bfdb9

2 years agoUpdate multiplex3.test to account for the fact that the multiplexor xDelete method...
dan [Thu, 24 Nov 2022 15:32:00 +0000 (15:32 +0000)] 
Update multiplex3.test to account for the fact that the multiplexor xDelete method may return an error even if it manages to delete the first chunk of a file.

FossilOrigin-Name: 1a7f3254735054ed8ca32d5ec7c8cde9195a64702638bdc50392007e396fead2

2 years agoAdd NEVER() and ALWAYS() macros on branches that are believed to be
drh [Thu, 24 Nov 2022 15:04:23 +0000 (15:04 +0000)] 
Add NEVER() and ALWAYS() macros on branches that are believed to be
unreachable.

FossilOrigin-Name: 3a901e88c87fc76c7fe42e47a976a5706830f0dbd6027605663e4d55f4f33590

2 years agoNew test cases. Fix the logic so that it works for GROUP BY aggregates
drh [Thu, 24 Nov 2022 13:19:25 +0000 (13:19 +0000)] 
New test cases.  Fix the logic so that it works for GROUP BY aggregates
that do not require sorting.

FossilOrigin-Name: ef6ebe7922f56c1584a005deedc85ca1070b4fe5082ada8bbf8d06df54f1c9ef

2 years agoSpeed up base64 conversions, and add test with more data for the baseNN conversion...
larrybr [Thu, 24 Nov 2022 02:59:33 +0000 (02:59 +0000)] 
Speed up base64 conversions, and add test with more data for the baseNN conversion to grind.

FossilOrigin-Name: 6c84ae4ba83713c751fddff8be41686bbcb525ac8135e1520436c62d0bc23d2c

2 years agoAdd sqlite3.oo1.DB.prototype.checkRc() and tests for both that method and its class...
stephan [Thu, 24 Nov 2022 02:35:03 +0000 (02:35 +0000)] 
Add sqlite3.oo1.DB.prototype.checkRc() and tests for both that method and its class-level counterpart.

FossilOrigin-Name: f7eaa6ba2147bfd6dbdc2444d0f919d846aa7f9b68cccab17ef585ffdacf3d60

2 years agoMerge the latest trunk changes into the agg-with-indexed-expr branch to
drh [Thu, 24 Nov 2022 01:41:44 +0000 (01:41 +0000)] 
Merge the latest trunk changes into the agg-with-indexed-expr branch to
simplify diffs.

FossilOrigin-Name: 38c3d3f1ed0fd2bb62aa8a7e5a27f2b247123e094e2fdb0a2475d788c3dfbc04

2 years agoAdd explanatory comment to the new optimization. And add a test case.
drh [Thu, 24 Nov 2022 01:40:20 +0000 (01:40 +0000)] 
Add explanatory comment to the new optimization.  And add a test case.

FossilOrigin-Name: e6c20f61de7d048eee65c8e74a3eb36760ab9747ebd1ab50e49642b777c10306

2 years agoMinor JS doc updates.
stephan [Wed, 23 Nov 2022 21:09:51 +0000 (21:09 +0000)] 
Minor JS doc updates.

FossilOrigin-Name: 27efd63ad7fb3bf8d0d07f2c9f48652c8cacc4e697c229c8085120a8e6b50a39

2 years agoAdd optional zSchema argument to sqlite3_js_db_export().
stephan [Wed, 23 Nov 2022 21:03:22 +0000 (21:03 +0000)] 
Add optional zSchema argument to sqlite3_js_db_export().

FossilOrigin-Name: 9c23644b1e5bf44bfb431a35fd1674c11ccb99e9eb0989f10175b0cb2a858eaa

2 years agoOPFS concurrency test: add a URL flag to enable/disable unlock-asap mode.
stephan [Wed, 23 Nov 2022 20:49:08 +0000 (20:49 +0000)] 
OPFS concurrency test: add a URL flag to enable/disable unlock-asap mode.

FossilOrigin-Name: 1c1bf22eadae2a5a7d4358e7cdd22641c2efb9296f42e7376749293b3a58b114

2 years agoAdd an experimental OPFS VFS-specific URI flag, opfs-unlock-asap, which tells the...
stephan [Wed, 23 Nov 2022 19:03:22 +0000 (19:03 +0000)] 
Add an experimental OPFS VFS-specific URI flag, opfs-unlock-asap, which tells the VFS to release implicit locks ASAP. This permits higher concurrency but hurts performance considerably. This may or may not be obsoleted by other concurrency-related experimentation.

FossilOrigin-Name: d23c917013485ec2793125221f3936b05c39d6eca941629fb819b6b4aa714520

2 years agoAggregates with GROUP BY now make use of expressions on indexes. This code
drh [Wed, 23 Nov 2022 18:51:04 +0000 (18:51 +0000)] 
Aggregates with GROUP BY now make use of expressions on indexes.  This code
works and gets the correct answer for the test case in the ticket.  Lots more
testing and documentation is needed, however.

FossilOrigin-Name: 8dcf9f2031c16f296d187fe876d4204c71fc96fec120984ff11b6d8b03d58a5f

2 years agoThis attempt at modifying AggInfo to make use of indexed expressions does not
drh [Wed, 23 Nov 2022 17:56:00 +0000 (17:56 +0000)] 
This attempt at modifying AggInfo to make use of indexed expressions does not
work.  It gets an incorrect answer for the test case shown in the ticket.

FossilOrigin-Name: 84c06023f4a1606664fdb9811312603b31f7c94a43d0e443ba7dde7fdba029e3

2 years agoInitial infrastructure for adding a mode to the OPFS VFS which causes implicit locks...
stephan [Wed, 23 Nov 2022 16:39:07 +0000 (16:39 +0000)] 
Initial infrastructure for adding a mode to the OPFS VFS which causes implicit locks to be released ASAP, which increases concurrency at the cost of performance.

FossilOrigin-Name: c5b7a9715a13b696ab3ee965aa0a310f59b65f07cecd72faa2e3504bfd8eb632

2 years agoUpdate Makefile.in to include new target "sqlite3r.c". For generating "sqlite3r.c...
dan [Wed, 23 Nov 2022 16:08:49 +0000 (16:08 +0000)] 
Update Makefile.in to include new target "sqlite3r.c". For generating "sqlite3r.c" and "sqlite3r.h", versions of the amalgamation that include the recover extension. To build the shell tool against these files, add -DSQLITE_HAVE_SQLITE3R.

FossilOrigin-Name: 5f135575b923cb59947667071c6af9ff14063c933cedf37d6c2a0a1b86c85032

2 years agoRemove a bit of over-cleverness which breaks loading of sqlite3.js in some main-threa...
stephan [Wed, 23 Nov 2022 15:52:52 +0000 (15:52 +0000)] 
Remove a bit of over-cleverness which breaks loading of sqlite3.js in some main-thread cases.

FossilOrigin-Name: 220cc4c6399b772b4ece03305a41b437ef0654d586a8a1c3dc5e7606fd36d655

2 years agoOPFS concurrency tester: ensure that the work interval timer does not overlap with...
stephan [Wed, 23 Nov 2022 15:33:49 +0000 (15:33 +0000)] 
OPFS concurrency tester: ensure that the work interval timer does not overlap with the work time.

FossilOrigin-Name: 8f2076553bc486ea6a17934695ecc02217461af2082d891697b62aab89bd1b43

2 years agoFurther foundation prep work prior to starting to flesh-out the
drh [Wed, 23 Nov 2022 14:13:39 +0000 (14:13 +0000)] 
Further foundation prep work prior to starting to flesh-out the
optimizeAggregateUseOfIndexedExpr() routine.

FossilOrigin-Name: 23145fe999ff74d787a3999baedd4ffe755c5f1f1275082ed0338ba637ecc56e

2 years agoAdd more baseNN tests, get oversize error trapping working, and sync w/trunk
larrybr [Tue, 22 Nov 2022 22:46:41 +0000 (22:46 +0000)] 
Add more baseNN tests, get oversize error trapping working, and sync w/trunk

FossilOrigin-Name: 03819e9368fd9f78f351147a1dc865743f9634893e43a9d1e3d7cbaf4c966069

2 years agoMerge enhancements and fixes from trunk.
drh [Tue, 22 Nov 2022 20:58:18 +0000 (20:58 +0000)] 
Merge enhancements and fixes from trunk.

FossilOrigin-Name: f8932e04d4d18eb9d71edda15aa08af2eb139ff14d77ca147ea6e9b173e0f5e0

2 years agoAdd the stub function: optimizeAggregateUsingIndexedExpr(). The hope is that
drh [Tue, 22 Nov 2022 20:37:41 +0000 (20:37 +0000)] 
Add the stub function: optimizeAggregateUsingIndexedExpr().  The hope is that
we can fill this in with a routine that does useful optimizations.

FossilOrigin-Name: d85bb724fdd6fbad2b88ed7f60e4174e3f65182356f404d04620c5cf6b17f77e

2 years agoAvoid naming collision between the sha1 and shathree extensions.
mistachkin [Tue, 22 Nov 2022 20:04:00 +0000 (20:04 +0000)] 
Avoid naming collision between the sha1 and shathree extensions.

FossilOrigin-Name: 9ec923b5dc24d6082da8d42bc0ee8ab1c418912625c0c56de9627be2c818ef98

2 years agoInclude the treetrace bitmap comment accidentally omitted from the previous
drh [Tue, 22 Nov 2022 19:56:54 +0000 (19:56 +0000)] 
Include the treetrace bitmap comment accidentally omitted from the previous
check-in.

FossilOrigin-Name: db07471c531766a8eec1d5b41c9fd0283b5e64ee13166dc3391f70a1e1946121

2 years agoMerge trunk fixes and enhancements into the agg-with-indexed-expr branch.
drh [Tue, 22 Nov 2022 19:51:16 +0000 (19:51 +0000)] 
Merge trunk fixes and enhancements into the agg-with-indexed-expr branch.

FossilOrigin-Name: 070634781a5eb41f96b001e48b064e3cd8c82314f576335eb1fcd43792179291

2 years agoRename the SELECTTRACE macro to TREETRACE, so that is corresponds to the new
drh [Tue, 22 Nov 2022 19:49:16 +0000 (19:49 +0000)] 
Rename the SELECTTRACE macro to TREETRACE, so that is corresponds to the new
CLI command.  Renumber all of the bits in the bitmask used to enable
various kinds of tracing, and add a trace bitmap decoder in sqliteInt.h.
Changes to debugging logic only.  No (intentional) changes to production code.

FossilOrigin-Name: 8036445a36d9d982c1305935e7e37367bdf9e466b923eb6286b52524802e3ccd

2 years agoSync w/trunk. Ready for merge to trunk.
larrybr [Tue, 22 Nov 2022 19:12:24 +0000 (19:12 +0000)] 
Sync w/trunk. Ready for merge to trunk.

FossilOrigin-Name: ff67460e1a3d21c9ca7cbd171fbc5e6cbdb3797de359887f851776b73b732fdf

2 years agoAdd test/basexx1.test to test is_base85(t), base85(x) and base64(x). Sync w/trunk.
larrybr [Tue, 22 Nov 2022 19:04:09 +0000 (19:04 +0000)] 
Add test/basexx1.test to test is_base85(t), base85(x) and base64(x). Sync w/trunk.

FossilOrigin-Name: 4e4334547795f659b4a5dc5fdaf842535643750a5f1ce1af799c526931a473e4