From a8a17ccd4bc21b0138f8df30f8b8d3e9f82a2c7b Mon Sep 17 00:00:00 2001 From: larrybr Date: Sat, 4 Dec 2021 18:40:51 +0000 Subject: [PATCH] Add design doc for shell extensibility FossilOrigin-Name: 96b8ffb05497b6c44f491fbb56d5ff6580b4fea112274f9f19faf24f79727460 --- doc/shell_extend.html | 534 ++++++++++++++++++++++++++++++++++++++++++ manifest | 11 +- manifest.uuid | 2 +- 3 files changed, 541 insertions(+), 6 deletions(-) create mode 100644 doc/shell_extend.html diff --git a/doc/shell_extend.html b/doc/shell_extend.html new file mode 100644 index 0000000000..3f0eee16a1 --- /dev/null +++ b/doc/shell_extend.html @@ -0,0 +1,534 @@ + + + + + + CLI Shell Extensibility + + + + +

CLI Shell Extensibility

Introduction

The command line shell for SQLite can be customized + to modify or add certain kinds of features, without altering its source. + This document details this extensibility + and how such extension may be accomplished by shell users. +

Shell extensibility serves to reduce the tension between + keeping the shell simple with broadly useful features + and allowing the shell to become an ever-growing tool + meeting diverse needs for those not ready to satisfy them + with a custom program using the SQLite libary. + Extensions contributed by the SQLite developers or others + will become (or are) available for use in situations + that may not justify permanently adding the same features + to the core shell published by the SQLite project in binary form. +

Extensible Features

Only certain categories of features may be + added or modified via extension, namely: + +

New or Revised Meta-Commands

The "dot" commands implemented by the shell, + along with help text for them, may be augmented or overridden. + A meta-command effected with an extension may be used + in the same ways as one available in the core, non-extended shell. + +

New or Revised Import Modes

The ways in which data may be imported to a DB table + may be augmented or overriden. Once that is done, the .import + meta-command will either have a new option to specify a new + import handler or an existing .import option can be used for this. + +

New or Revised Query Result Handling

The display formatting or other disposition of query results + may be augmented or overriden. Once that is done, the .mode + meta-command will either have a new option to specify a new + result handler or an existing .mode option can be used for this. + +

Generalized Import Modes and Query Result Handling

New handlers added via extension are not restricted to + importing data from a file or result display formatting. + They may be considered more generally to be either + a data source or a data sink, producing or accepting + data row sets. The origin or destination of + the data is up to the handler, as may be affected + by arguments to the .import or most recent .mode command. + +

Extension Methods, Dynamic or Static

Shell extension may be effected at different times + in different ways according to convenience and need. + +

Runtime Extension

Extension at runtime (when the shell is running) + is effected via the .load command with a --shell flag. + In this way, a dynamically loaded library (DLL) is loaded with + provision made for its sqlite3_X_init() function to obtain + the shell extension API entry points and thereby + register with the shell core any + new meta-commands, or import or query result handlers + that it implements. + +

Build-Time Extension

Extension when the shell is built is effected by specifying + certain option values to either the "make" invocation or + to a utility, (tool/mkshellc.tcl), which assembles and transforms + sources to produce the shell's unitary source file (shell.c). + +

Extension Source Code

With certain coding conventions and methods followed, + the same source code can be used either + to produce most of a runtime shell extension DLL or to + be incorporated into shell.c as a built-in shell extension. + (See the extension code samples for details.) + +

Build-Time Diminuation

Just as new meta-commands can be readily incorporated into + the shell when it is built, many of the core meta-commands + can be readily omitted from the shell build. This is done + with a variation of the option values that may be given + to "make" or to tool/mkshellc.tcl as the shell is built. + (See tool/mkshellc.tcl --help output for details.) + Such omission of meta-commands might be done when building + a customized shell which need not have the various meta-commands + which exist for the purpose of testing the SQLite library. + +

Interface

The following details relate to src/shext_linkage.h, a header + in which declarations appear for objects and functions + that facilitate runtime interaction between the shell core + and shell extensions written in C/C++. + Comments in that header tersely summarize these explanations: + +

struct ShellStateX

This struct consists of a public portion, which is available and + stable for use in shell/extension interactions, and a private + portion which may not be stable. Shell extension code used only for + build-time extension might use the private part, (to which it has + access because such code is compiled within the same translation unit + as the core shell code), but such usage generally precludes (or makes + hazardous) use of a runtime loadable extension built from the same code. + +

ExtensionId typedef and eid Member

An object of this type serves to uniquely identify an extension + loaded at runtime so that it may be unloaded later. It must be + passed back to the shell (in the ShellExtensionLink eid member) + by the sqlite3_X_init() function if the extension DLL is ever + to be unloaded during that shell session. + +

extensionDtor member

The function addressed by this member will be called prior to + the extension being unloaded (if the pointer is non-zero.) + This is an out parameter from the sqlite3_X_init() function. + It may perform any cleanup or deallocations necessitated by + successful initialization (and will never be called after + failed initialization.) + +

Notes Regarding Object Interfaces for C vs C++ Writers

The objects registered with the shell core to provided extension + functionality may be implemented in C or C++ (or anything else + producing the same ABI.) In the below descriptions of their + interfaces, it should be understood that: C++ implementations + will not need to explicitly deal with anything like a WhatsitVtable + struct and will refer to the object pointer, passed implicitly, + as "this"; and C implementations will need to populate a static + WhatsitVtable and refer to the initial object pointer as "pThis". + +

MetaCommand typedef

These objects represent an extension meta-command, including a + dispatch table for the public interface and any accompanying + data (which is opaque to the core shell.) Such objects are created + by extensions and passed to the core shell only by reference. + They are made known to the shell core via registerMetaCommand() calls. + +

MetaCommandVtable typedef

These objects represent the dispatch table of a MetaCommand object. +

All methods are given the same leading (or lone) argument:
+ (1) the address of the registered MetaCommand object. + +

destruct_free method

This method is called prior to unloading a runtime extension + for any registered MetaCommand object, provided its dispatch + table entry is non-zero. + It should free resources allocated during the sqlite3_X_init() call. + +

name method

This method returns the name of the meta-command (sans leading '.'.) + The returned pointer must remain valid throughout the lifetime of + the registered MetaCommand object. + +

help method

This method returns help text for the meta-command. This text + should be formatted and aligned with the built-in meta-command + help text so that it can be displayed seamlessly. +

The return is either a C string or null pointer. The C string + will not be freed by the core shell, and must remain valid + during the lifetime of the MetaCommand object. + +

argsRange method

This method returns a pair of unsigned integers indicating the range + of valid argument counts for the meta-command. +

The returned .minArgs value indicates the minimum number of + arguments required for a successful execute call. + The returned .maxArgs value indicates the maximum number of + arguments allowed for a successful execute call, with ~0 + indicating no (practical) upper limit. The execute method will + never be called with an argument count not within this range. + (This significantly simplifies argument checking.) + +

execute method

This method performs whatever work the meta-command is supposed + to do when invoked. It has 4 additional arguments:
+ (2) present ShellStateX, passed by reference (an in/out parameter);
+ (3) an error message pointer, passed by reference, set upon error but + otherwise not modified, to be freed by the shell core;
+ (4) the number of invocation arguments;
+ (5) an array of C strings constituting the invocation arguments;
+

The return is 0 for success, or anything else to indicate error. + The special value SHELL_INVALID_ARGS (aka SQLITE_MISUSE) should be + returned when argument checking shows an invalid call. This will + cause the command dispatcher to issue usage help (in interactive + shell sessions) without further fuss by the execute() method itself. + (And if an extension causes this error to be returned from the SQLite + library, which is a serious error by itself, it must either translate + it for return or trouble its users with a misleading error message + from the dispatcher.) If the output error message is set, it will + be issued in lieu of the standard error message for invalid calls. + +

FormatXfrInfo typedef

An object of this type is passed between the shell core and its + import or query result handlers to convey or keep parameters and data + related to formatting or parsing data rows in an external form, + or to keep state associated with the progression of an import or result + handling operation from initiation to completion. +

The shell core .mode and .import implementations also use + an instance of this type to affect result output and import operations. + That instance resides in the ShellStateX so that extension meta-commands + can access it, possibly to change it. Meta-commands which alter this + instance for their own purposes (rather than for intended effect) + should take care to restore its value as the meta-command completes. + +

OutModeHandler typedef

These objects represent an extension query result handler, including + a dispatch table for the public interface and any accompanying + data which is opaque to the core shell. Such objects are created + by extensions and passed to the core shell only by reference. + They are made known to the shell core via registerOutMode() calls. + +

OutModeHandlerVtable typedef

These objects represent the dispatch table of an OutModeHandler object. + All methods in the dispatch table are given at + least this leading argument:
+ (1) The OutModeHandler address registered via registerOutMode();
+ +

destruct_free method

This method is called prior to unloading a runtime extension + for any registered OutModeHandler object, provided its dispatch + table entry is non-zero. + It should free resources allocated during the sqlite3_X_init() call. + +

name method

This method returns the name of the OutModeHandler, which users + specify to the .mode command (as the mode) to designate use of the + registered OutModeHandler for subsequent query results. + The returned pointer must remain valid throughout the lifetime of + the registered MetaCommand object. + +

help method

This method returns help text for the OutModeHandler. +

There is one additional argument:
+ (2) an integer directing what help text to return, with + 0 indicating primary, single-line help, or + 1 indicating any more detailed help beyond the primary level. + The primary help is included in the .mode command's own + detailed help text, so it should be aligned accordingly. + The detailed help is shown by the .mode command's --help option. +

The return is either a C string or null pointer. The C string + will not be freed by the core shell, and must remain valid + during the lifetime of the MetaCommand object. + +

Common arguments

The following methods are given these 2 additional arguments:
+ (2) A FormatXfrInfo object passed by reference; and
+ (3) An error message pointer, passed by reference, to receive errors.
+ +

openResultsOutStream method

This method is called when a query output is setup, + (via the .mode command with the handler's name given as a --flag.) +

It is given 3 additional arguments:
+ (4) the number of arguments in the said .mode command;
+ (5) an array of C strings with the said argument values; and
+ (6) the name given as a --flag which caused the handler to be used. +

When the extension handler is activated via a .mode command, + parsing that command and behaving accordingly is the responsibility + of this method alone. (The command is parsed and acted upon by the + default .mode implementation only when no OutModeHandler is used.) +

Once this method is called and succeeds, it is guaranteed that the + closeResultsOutStream method will be called. +

This method should return SQLITE_OK only upon success. + Any other return will abort remaining calls in the handling sequence. + +

prependResultsOut method

This method is called when a query succeeds for which this handler + will be given the results. This is purely preparatory; zero or + more result rows may follow. It is up to this method to determine + if any results can even be had (by considering the return from + sqlite3_column_count()) and acting accordingly. +

It is given 1 additional argument:
+ (4) the query prepared statement (pointer), not yet stepped. +

This method should return SQLITE_OK only upon success. + Any other return will abort the next 2 calls in the handling sequence. + (TBD: Such aborts without error should be supported for DDL and DML.) + +

rowResultsOut method

This method is called when a query's prepared statement is stepped, + for each result row available. +

It is given 1 additional argument:
+ (4) the query prepared statement (pointer), stepped with + a result row available. It is permitted for this method to + perform the remaining stepping, as indicated by the return. +

This method should return SQLITE_OK only upon success + without having completed stepping. If this method completes stepping, + (by making its own calls to sqlite3_step() until it returns SQLITE_DONE), + it must return SQLITE_DONE. Any other return will abort the next call + in the handling sequence. + +

appendResultsOut method

This method is called when result set stepping has been completed. +

It is given additional argument:
+ (4) the query prepared statement (pointer), completely stepped. +

This method should return SQLITE_OK upon success, or may return + something else to indicate error with no effect upon succeeding calls. + +

closeResultsOutStream method

This method is called when a new .mode command is invoked specifying + some output mode not using this OutModeHandler or when the extension + is about to be unloaded with this OutModeHandler selected for output. + It should free resources allocated or held as a result of the + previous openResultsOutStream call. + +

ImportHandler typedef

These objects represent an extension data import handler, including + a dispatch table for the public interface and any accompanying + data which is opaque to the core shell. Such objects are created + by extensions and passed to the core shell only by reference. + They are made known to the shell core via registerImporter() calls. + +

ImportHandlerVtable typedef

These objects represent the dispatch table of an ImportHandler object. + All methods in the dispatch table are given this 1 leading argument:
+ (1) The ImportHandler address registered via registerImporter(). + +

destruct_free method

name method

This method returns the name of the importer, which is to + be passed with leading '--' (or '-') to the .import command + to specify use of the registered importer for that invocation. + The returned pointer must remain valid throughout the lifetime of + the registered MetaCommand object. + +

help method

This method returns help text for the importer. +

There is one additional argument:
+ (2) an integer directing what help text to return, with + 0 indicating primary, single-line help, or + 1 indicating any more detailed help beyond the primary level. + The primary help is included in the .import command's own + detailed help text, so it should be aligned accordingly. + The detailed help is shown by the .import command's --help option. +

The return is either a C string or null pointer. The C string + will not be freed by the core shell, and must remain valid + during the lifetime of the MetaCommand object. + +

Common arguments

The following methods are given these 2 additional arguments:
+ (2) A FormatXfrInfo object passed by reference; and
+ (3) An error message pointer, passed by reference, to receive errors. + +

openDataInStream method

This method is called when a .import command is invoked with + the '--' (or '-') prefixed name of the ImportHandler as an argument. +

These 3 additional arguments are passed:
+ (4) the number of arguments to said .import command;
+ (5) an array of C strings with the said argument values; and
+ (6) the name of the ImportHandler (as its name method would return.) +

When the extension handler is activated via a .import command, + responsibility for parsing that command and behaving accordingly is + shared between the shell core .import implementation and this method. + The shell core normally takes the last argument as the name of a table + (to be created if necessary) which will receive the imported data. + (But see return values for exceptions to this treatment.) + The shell core also detects any --flag argument selecting a registered + ImportHandler (giving effect to the last one) and calls this and + following methods to perform the input part of the import operation. + This method may interpret any or all of the arguments as needed + and (supposedly) documented by the associated help(...) method returns. +

Once this method is called and succeeds, it is guaranteed that the + closeDataInStream method will be called. +

This method should return SQLITE_OK upon success where disposition + of the imported data (into a table named by the last argument) is + to be handled normally, by the shell core. + An alternative success return is SQLITE_DONE, indicating that + disposition of the imported data has been done by the ImportHandler. + In that case, the next 3 methods (*DataInput(...)) will not be called + and this method should have completed the whole import operation. + After a success return, closeDataInStream is guaranteed to be called. + Other returns will abort all remaining calls in the handling sequence. + +

prepareDataInput method

This method prepares for the particular import operation commenced + with a .import invocation. It may take into account the shape of + the input data (and possibly its type) as discovered during the call. +

This 1 additional argument is passed:
+ (4) a to-be-prepared statement (pointer), passed by reference. + This is an out parameter conveying a prepared statement created + by this method specifically for the single import operation. +

The prepared statement should, in some manner wholly determined + by the extension handler, incorporate compiled SQL (possibly with + as-yet unbound parameters) which will produce a result set. + (This may be no more than a query such as "SELECT @1 as one, ...", + or could be a SELECT from a temporary table used for buffering.) +

The return should be SQLITE_OK upon success, in which case the + following 2 methods will also be called. Any other return will + abort the {prepare,row,finish}DataInput() call sequence. In that + case, no prepared statement should be returned either. + +

rowDataInput method

This method is called to collect imported data, making it available + through the prepared statement passed as the last argument with as + many steps as are needed to get SQLITE_DONE from sqlite3_step(). + It will be called repeatedly until its return indicates no more data. +

It is given 1 additional argument:
+ (4) the prepared statement (pointer) returned by prepareDataInput().
+ This prepared statement can have values bound to it or be reset as + necessary to return some or more data. However, it must remain the + same sqlite3_statement instance returned by prepareDataInput(). +

It is given 1 additional argument:
+ (4) the prepared statement (pointer) returned by prepareDataInput(). +

The return should be SQLITE_DONE when no more data is available, + or SQLITE_ROW to indicate that more data is or might be available. + Any other return (including SQLITE_OK) indicates a condition which + will abort the data collection phase of the import operation. Any + of those 3 returns is treated as success by the shell core. + +

finishDataInput method

This method is called to complete the data input phase of the + import operation commenced by prepareDataInput(). +

It is given 1 additional argument:
+ (4) the prepared statement (pointer) returned by prepareDataInput().
+ This prepared statement should be finalized by this method. Other + cleanup or import closing related to the transfer may also be performed. + +

closeDataInStream method

This method is called as a .import command which specified this + ImportHandler completes. + It should free resources allocated or held as a result of the + previous openDataInStream call. + +

ShellExtensionLink typedef

An object of this type is passed (somewhat indirectly) to the + sqlite3_X_init(...) function which is called when a runtime + extension is loaded via a .load command with a --shell flag. + It is used to establish linkage between the loaded extension + and a shell core API exposed specifically for extensibility. +

At present, the extension API is limited to registration of + meta-commands, query result handlers, and import handlers + implemented by extensions. This API may be extended in future + versions of the core shell, in a backwards-compatible manner. + (See the pExtra sentinel, whose offset may increase.) + +

Establishing Shell/Extension Linkage

The 1st or 2nd parameter passed to the sqlite3_X_init() function + as an extension is dynamically loaded is used to obtain a pointer + to the ShellExtensionLink object (provided that .load was + invoked with the --shell flag.) To verify that the correct + object type was passed and reference it if so, the extension's + sqlite3_X_init() function can use one of these two means: +

The least code-intensive means is implemented by a macro, + EXTENSION_LINKAGE_PTR(pzem), parameter of which is the char** + passed as the 2nd sqlite3_X_init() argument. This is reliable + except in the case of a maliciously written shell core (which + portends worse problems than the undefined behavior which + could arise from invoking .load from such a shell.) +

A more code-intensive means, (which is no safer in the + case of a maliciously written shell core), is to use + a C macro, DEFINE_SHDB_TO_SHEXT_API(function_name), + in the extension source to define a function named per + the macro argument. Then that function may be called with its + lone argument being the sqlite3 * (db) pointer passed as the + 1st argument to sqlite3_X_init(). +

Either means will return (or yield) either a null pointer + or a verified ShellExtensionLink pointer. In the former + case, sqlite3_X_init() may return SHELL_INVALID_ARGS to induce + the emission of help on using the .load command. (Said help + will mention the need to use the --shell flag for extensions.) + In the latter case, the obtained pointer may be called to + register the extension's feature implementor(s) as described. + +

Development Aspects of Extensibility and Its Evolution

+ +

There is critical tension between competing uses of the ShellState + object which is kept and used by the core shell for state which + must persist between meta-command invocations. For unhindered + implementation and feature expansion flexibility, the structure + of the ShellState data should be unconstrained across versions + of the shell. However, unless extension meta-commands and data + transferers are to act entirely independently of built-in meta-commands, + (excepting interaction through the above-described extension methods), + some portion of the ShellState data needs to be shared in a stable + manner between extensions and the core shell code. +

With respect to interface stability concerns, it is nearly immaterial + whether the sharing occurs through exposed data structures + or an extension API devised to convey similar data. (Only data layout + and possible change notification are at stake in that choice.) +

For simplicity, and because it can work in a way consistent with how + built-in shell features are implemented now, the chosen sharing method + is to simply directly expose a subset of the ShellState data. That + subset will be extremely limited to minimize hinderance of what has + previously been unfettered change to that data structure/meaning. + It will initially be: the present output stream, as affected by + .output and .once commands; the currently open user DB (if any); + the dedicated shell DB; and the data related to formatting + and transfer of data in external forms. This data resides in the + ShellState object as a FormatXfrInfo struct. +

As shell extensibility evolves, additional data items may need + to be moved into the publicly exposed portion of ShellState, either + directly (via exposed data members) or by means of some additional + extension APIs defined in the ShellExtensionLink object (which has + been defined to accommodate growth of its function pointer list in + a backwards-compatible manner.) + + + + diff --git a/manifest b/manifest index 31155c25d0..4771a2bff7 100644 --- a/manifest +++ b/manifest @@ -1,5 +1,5 @@ -C Extension\sinterface\stweaks\sin\ssupport\sof\splanned\susage -D 2021-12-03T19:27:06.906 +C Add\sdesign\sdoc\sfor\sshell\sextensibility +D 2021-12-04T18:40:51.887 F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1 F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea F LICENSE.md df5091916dbb40e6e9686186587125e1b2ff51f022cc334e886c19a0e9982724 @@ -40,6 +40,7 @@ F contrib/sqlitecon.tcl 210a913ad63f9f991070821e599d600bd913e0ad F doc/F2FS.txt c1d4a0ae9711cfe0e1d8b019d154f1c29e0d3abfe820787ba1e9ed7691160fcd F doc/lemon.html efc0cd2345d66905505d98f862e1c571512def0ceb5b016cb658fd4918eb76a3 F doc/pager-invariants.txt 27fed9a70ddad2088750c4a2b493b63853da2710 +F doc/shell_extend.html b369e9dc53b7d07e2fc8e4acdf3183784826c522a0111960026f6058081c1a98 F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a F doc/vdbesort-memory.md 4da2639c14cd24a31e0af694b1a8dd37eaf277aff3867e9a8cc14046bc49df56 F doc/vfs-shm.txt e101f27ea02a8387ce46a05be2b1a902a021d37a @@ -1934,7 +1935,7 @@ F vsixtest/vsixtest.tcl 6a9a6ab600c25a91a7acc6293828957a386a8a93 F vsixtest/vsixtest.vcxproj.data 2ed517e100c66dc455b492e1a33350c1b20fbcdc F vsixtest/vsixtest.vcxproj.filters 37e51ffedcdb064aad6ff33b6148725226cd608e F vsixtest/vsixtest_TemporaryKey.pfx e5b1b036facdb453873e7084e1cae9102ccc67a0 -P eab1e1af5b51eb267e9bceaf6412d2be12b8f3deb435631e55291211099720ff -R d47b4cfbdf90e9b5dfc09603b7ee76a8 +P ce2a91438a3403f55cddc6c5e26db292bf6dd10e805f55416063a63986d28740 +R e58755c0a075efc772ebe9b493832561 U larrybr -Z 1371e2d598bc40c6fc66804e5df68b0c +Z 7adbcc5b389a42b1ea01dd812dbf6613 diff --git a/manifest.uuid b/manifest.uuid index 668dba7b61..1bb5212862 100644 --- a/manifest.uuid +++ b/manifest.uuid @@ -1 +1 @@ -ce2a91438a3403f55cddc6c5e26db292bf6dd10e805f55416063a63986d28740 \ No newline at end of file +96b8ffb05497b6c44f491fbb56d5ff6580b4fea112274f9f19faf24f79727460 \ No newline at end of file -- 2.47.3