From dbafc78aa849584a2aa9433d23afdb6bee4099f6 Mon Sep 17 00:00:00 2001 From: larrybr Date: Mon, 11 Apr 2022 08:07:16 +0000 Subject: [PATCH] Add embedded doc, Intro to Tcl Extension FossilOrigin-Name: 33591833326e98882e0fa6225a7a55a7e915063a36bf24bb3c17c35e693d972e --- doc/tcl_extension_intro.md | 403 +++++++++++++++++++++++++++++++++++++ manifest | 11 +- manifest.uuid | 2 +- 3 files changed, 410 insertions(+), 6 deletions(-) create mode 100644 doc/tcl_extension_intro.md diff --git a/doc/tcl_extension_intro.md b/doc/tcl_extension_intro.md new file mode 100644 index 0000000000..72065e89d5 --- /dev/null +++ b/doc/tcl_extension_intro.md @@ -0,0 +1,403 @@ +# Introduction to the TCL Extension for the SQLite Extensible Shell # + +This article introduces an extension, +written to be hosted by the SQLite Extensible Shell, +which adds [Tcl](https://www.tcl.tk/about/index.html) features to it. + +## Motivation ## + +An understanding of the motivation for this feature may prove useful +for putting what follows into context. +Over its 2+ decade existence, the sqlite3 CLI shell has evolved, +from its original utility as a test vehicle for the SQLite library, +to a tool used by many people who want to use SQLite databases +without writing a specialized application. While the shell exposes +many of the library features, its original design was suited for +performing short sequences of DB operations, generally known prior +to starting the shell. + +Using the shell for more complex operations can be difficult, +usually requiring use of a general purpose scripting tool +to drive the sqlite3 shell. +This can work well where the task can be decomposed into +"Prepare a sequence of elementary DB operations," +followed by +"then feed the sequence into sqlite3 as its command/SQL input." +However, where the task requires multiple such steps, +with sqlite3 either being run multiple times or +being made to engage in two-way communication with a driving process, +getting the combination to work as intended can be tedious and tricky. + +Those difficulties are enough to make one wish for a better way +to drive the tool, and to muse: If only there was a "Tool Command Language". +Fortunately, there is; it is known as "Tcl". +The sqlite3 shell is a tool, in need of being flexibly commanded, +and Tcl is more than adequate for that purpose, as will be shown. + +Not to be overlooked is that SQLite itself originated as a +library for Tcl, and that its integration with Tcl makes it both +powerful and easy to use in Tcl programs. +In fact, Tcl with SQLite can be a very good substitute for +ad hoc combinations of general purpose scripting tools with the SQLite shell, +and has been +[since year 2000](https://www.sqlite.org/src/timeline?c=2000-05-29+14:26:00). + +Beyond the utility of [Tcl with SQLite](https://www.tcl.tk/about/uses.html), +SQLite (the shell) with Tcl has virtues of its own. +These include instrumentation, +query plan display, ready-to-use data import and export features, +among others familiar to sqlite3 shell users. + +## Basics, Terminology and Getting Started ## + +A variant of the SQLite shell can be built, called "sqlite3x" here, +whose long name is "The SQLite Extensible Shell". +It may be built on Unix-like systems by running "make sqlite3x". +(Build setup for Windows systems will be provided soon.) + +The Tcl extension for sqlite3x, called "tclshext" here, +may be built for Unix-like systems, +after configure is run with a "--enable-tcl" option +(and a "--with-tcl=..." option if necessary), +by invoking make with "tcl_shell_extension" as a target. +It may be necessary to first install the Tcl development package or library +in order for configure to find the Tcl interpreter +and specify how get it linked into the extension. + +To manually get a Tcl-extended-shell started, +(assuming the above-mentioned images were built +and have been placed where the shell and OS loader can find them), +either of these inputs is needed:
+``` + sqlite3x + .load tclshext -shext + or + sqlite3x -cmd '.load tclshext -shext' +``` +Provided this results in another prompt without any error message(s), +the Tcl-extended shell is ready to go. +For brevity, the shell in this state will be called "sqlite3xt". + +## Yet Another Prompt -- Now What? ## + +When sqlite3xt is ready, it acts very much like the sqlite3 shell. +When given sensible inputs (as discussed below in "Parsing"), +that are recognized by sqlite3, +the same outputs and effects will occur as would with sqlite3. +This condition, +where SQL and dot commands recognized by sqlite3 may be input +and acted upon as sqlite3 would, +is referred to below as the "shell execution environment". +For sqlite3xt, that is just the beginning. + +## Execution Environments and Input Parsing Changes ## + +The effect of loading tclshext can be briefly summarized as +the introduction of an alternative "execution environment", +explication of which follows this on the primary "execution environment": + +### shell execution environment ### + +The "shell execution environment", in effect upon sqlite3xt startup, +has these characteristics pertinent here: + +* The leading non-whitespace token of an input line group +is interpreted as a dot command if it begins with a single '.'; + +* if that token is nothing but a '.', the line is a no-op; + +* or, if it begins with anything but a '.', it is collected +as SQL and submitted to the SQL execution (prepare/step) engine; + +* or, if it begins with '#', it is ignored (as a comment). + +* Whitespace-delimited arguments after the leading token +are parsed according to the section on "Dot-command arguments" +[here](https://sqlite.org/cli.html), with one exception as noted next. + +The exception to legacy argument parsing is that open quote +constructs are not auto-closed by the end of an input line. +(This legacy behavior may be reinstated if desired by: entering +".shxopts -parsing" at the prompt; renaming the sqlite3x image +to sqlite3 before executing it; or invoking it with the option +pair '-shxopt' '0'.) +Instead, arguments are collected, potentially across multiple lines, +until an input line ends outside of any open quote construct. +(Input which does not depend on the legacy, auto-close-on-newline +behavior is what the term "sensible inputs" means as used above.) +As examples, this would not be sensible input to sqlite3:
+``` + .print 'Hello. + This was entered as multi-line input.' +``` +as it would result in an error, while this input:
+``` + .print "I'm not fond of closing my quotations. +``` +is acceptable to sqlite3, but deemed not "sensible" here. +When either is input to sqlite3x, a continuation prompt will be issued +(in interactive mode.) + +Of course, this is (mostly) review to those familiar with the sqlite3 shell. + +### Tcl execution environment ### + +The "Tcl execution environment" differs in several important ways +from the familiar execution environment describe above. +How to get into this alternative execution environment is described later. + +In this alernative execution environment, these critical differences exist: + +* An expanded set of command words is available and readily expanded further. + +* The available command words generally do not begin with '.'. + +* When in interactive mode, commands whose +initial token is not defined as a Tcl command, +but can be found as an executable +in directories named in the PATH environment variable, +will be executed in a subprocess +(unless blocked by auto_noexec having been set.) + +* The command word and arguments are collected, +parsed and expanded according to the usual +[rules for the Tcl language](https://www.tcl.tk/about/language.html). +In particular, input line groups are collected until deemed "complete" +by the Tcl parser. This means no open brace, quote or bracket constructs. + +* New [command words can be readily defined] +(https://www.tcl.tk/man/tcl8.6/TclCmd/proc.html), +and [variables can be set](https://www.tcl.tk/man/tcl8.6/TclCmd/set.html), +either of which may affect argument expansion +per the usual Tcl rules. + +This environment will be quite familiar to those who use Tcl. +There are a few differences however. These are:
+ +* A single '.' on an input line which is not inside of an incomplete +Tcl line group is treated as a (momentary) end-of-input by the REPL. + +* The shell's dot commands, while not visible via \[info commands\], +are discoverable by the "unknown" command and will be executed +if their names (with the '.' prefix) would be found and resolved +unambiguously in the shell execution environment. +Commands whose names begin with '.' which are not found unique in the +shell execution environment produce an "invalid command name" error. +Except for that treatment, the unknown command in effect +acts like the stock, Tcl version. +(That stock version remains available as _original_unkown, +to which non-. commands are delegated .) + +* A few, non-stock-Tcl commands are available, in particular: + + + Commands udb and shdb act nearly like commands creatable +by the "sqlite3" Tcl package to represent an open database. They +differ in that they do not accept the "close" subcommand, which +reflects the fact that they exist to allow the shell's current +user database and the shell's own database to be accessed just +like ones created in Tcl via "sqlite3 someDbName itsFilename". + + + Commands sqlite_shell_REPL, get_tcl_group and now_interactive +permit input to be collected in the same manner (and from the same +sources) as the shell's REPL does. Here, "collected in the same manner" +does not include the execution environment switching or SQL execution +that the shell execution environment implements. + + + Command register_adhoc_command permits a newly (or oldly) defined +Tcl command, likely with a leading '.' in its name, to associate +help text with the shell. In this way, .help output can emit help +text, in summary or long forms, for Tcl commands that might be +executable from the shell execution environment. +(More on this below.) + + + Finally, command .. exists for reasons made clear below. +With no arguments, +it does nothing, quietly and successfully, with the empty result. + +* The present implementation does not run the event loop processing +that Tcl often uses for certain functionality (such as sockets.) + +## Switching Execution Environments ## + +The simplest execution environment switching +is effected with lone dots and dot pairs. +Some examples should make this clear. For now, focus on "." or ".." +entered on a line by itself. +(The # comments are not required. The leading prompts reflect +what would be seen in interactive use; they are not to be typed either.)
+``` + sqlite> # Now in shell execution environment, but it is time to do Tcl stuff. + sqlite> .. + tcl% # Now in Tcl execution environment, ready to roll a trivial dot command. + tcl% proc .do {what} { + > .eval $what + > } + tcl% register_adhoc_command {do Does whatever one thing it is told to do + > from the shell execution environment and little else} + tcl% # Time to return to shell execution environment. + tcl% . + sqlite> # See how ad hoc command creation worked. + sqlite> .help do + .do Does whatever one thing it is told to do + from the shell execution environment and little else + sqlite> .do .conn + ACTIVE 0: :memory: + sqlite> .do ".. puts Putting" + Putting + sqlite> # Be sure about being in shell execution environment. + sqlite> . + sqlite> # Oh, the prompt would have sufficed. Now, for some Tcl ... + sqlite> .. + tcl% # Let's see what that .. command does. + tcl% .. + tcl% # Apparently, it either gets to the Tcl environment or stays there. +``` + +The use of lone . and .. to switch environments are easiest to understand. +However, .. , when not alone on an input line enables some other uses of Tcl, +explained in the next section. + +Another effect of loading tclshext is that a new dot command, .tcl , +becomes part of the shell's repertoire. +When entered no argument(s), +from the shell execution environment, +the .tcl command has the same effect as .. by itself. +(footnote: +When run from the Tcl execution environment with no arguments, +it acts as a no-op rather than entering a recursive REPL.) +With arguments, it will read file content into the Tcl interpreter +just as Tcl's source command would. +This may be useful for getting the Tcl execution environment +customized via sqlite3x invocation options such as +"'-cmd' '.tcl my_sqlite3xt_init.tcl'" + +A side benefit of the .tcl command's existence is that +it appears in .help output, which can remind users how +to use the extension's main feature. + +## Momentary Use of Tcl Execution Environment ## + +When .. is not the sole non-whitespace content of an input line group +which has been submitted from the shell excecution environment, +that causes argument collection and expansion to be performed +according to Tcl rules by the Tcl interpreter, +without entering and staying in the Tcl execution environment. +These two variations exist (where "..." is the provide argument set): +``` + sqlite> ..dotcmd ... + or + sqlite> .. tclcmd ... +``` +The former, where no space separates .. from dotcmd, causes the +dot command known as .dotcmd to be executed (if it can be found.) +The latter, where space separates .. from tclcmd, causes the +Tcl command known as tclcmd to be executed (if it can be found.) +In either case, when the execution terminates (or fails with a +"command does not yet exist" or "invalid command name" error), +the shell execution environment remains in effect afterward. + +This temporary use of the Tcl interpreter serves two purposes. +One is to exploit the more powerful capabilities of Tcl for argument processing. +Within the text of arguments as provided, +variables can be accessed, +computations can be done, +and Tcl commands can be invoked to yield results (or produce side effects), +all of which can affect what the expanded arguments finally contain. +The other (miniscule) effect is to avoid the need for extra +input lines to switch to the Tcl execution environment and back +when "one-shot" use of Tcl is all that is needed. +In other words:
+``` + ..dotcmd whatever ... + acts as a shorter form of + .. + .dotcmd whatever ... + . +``` + +Whenever more than one '.' leads an input line group, +when it is submitted in the shell execution environment, +then most of the line group is given to the Tcl interpreter for +processing on a one-shot basis. + +## Cross-Execution Environment Interactions ## + +Study of the above examples might give rise to these questions:
+  "How can '..dotcmd ...' work in the Tcl execution environment?"
+and
+  "Why does '.do anything' work in the shell execution environment?"
+There are two features at work behind those examples working: + +In the Tcl execution environment, (as touched upon above), +when a prospective Tcl command is not found to be defined +in the namespace(s) searched by the interpreter, it is +passed to the Tcl command named "unknown". +In the tclshext-augmented Tcl environment, +that procedure is implemented by C code which treats +commands with a leading '.' specially, +by attempting to find them in the shell's repertoire of dot commands. +If found (unambiguously), they are executed then, +and the result (such as it is) returned to the caller. +So, assuming there is a dot command invocable as .dotcmd, +(which could happen if another extension provided it), +it can be found and executed from the Tcl execution environment +with arguments as collected and expanded by the Tcl interpreter. + +In the shell execution environment, if any extensions have been +loaded when a prospective dot command is not found in the +existing repertoire of dot commands kept by the shell, +the shell's dispatcher gives the (undocumented) .unknown +dot command a chance to execute it. This has no chance +of succeeding unless some extension has overridden the +built-in .unknown implementation. However, tclshext does +override it, and the replacement attempts to find any +dot command name it is given in the Tcl root namespace. +If it can be found, (such as will be true for the above +example invoking .do from the shell execution environment), +then that command is executed as a Tcl command +with whatever arguments were passed to .unknown . + +Because of how this works, there is not often any reason or need to +leave the Tcl execution environment. Dot commands and Tcl commands +can be freely intermixed and executed in that environment. The main +reason one might revert to the shell execution environment would +be to evaluate SQL statements in the usual, sqlite3 way, (by just +typing them followed by ";".) However, SQL can be evaluated without +leaving the Tcl execution environment by use of the recently added +dot command, .eval, which evaluates its arguments in the shell +execution environment. For example:
+``` + tcl% .mode box + tcl% .eval { + > .print "Let's have a one," + > select 1 as one; + > } { + > .print "and a two." + > select 2 as two; + > } + Let's have a one, + ┌─────┐ + │ one │ + ├─────┤ + │ 1 │ + └─────┘ + and a two. + ┌─────┐ + │ two │ + ├─────┤ + │ 2 │ + └─────┘ + tcl% +``` + +It should be noted, for those new to Tcl, that brace-quoting in Tcl +means "Whatever is between these matching braces is the (single) value." +It can be used for nearly any content, except for unmatched braces. + +## Summary, More to Come ## + +The Tcl extension should prove useful to those who like the features +of the sqlite3 shell but sometimes find it diffult to use programmatically. + +A future article will cover using Tk, a graphical user interface +toolkit originally creaated for use with Tcl, with sqlite3xt. diff --git a/manifest b/manifest index 3cd411c81a..856e446142 100644 --- a/manifest +++ b/manifest @@ -1,5 +1,5 @@ -C Create\sparallel\sC\sand\sC++\stest/demo\sshell\sextensions,\swith\sheader\stweaks\sto\smake\sthat\swork.\sFix\sa\sbug\sexposed\swhen\smultiple\sshell\sextensions\swere\sloaded. -D 2022-04-10T09:17:29.899 +C Add\sembedded\sdoc,\sIntro\sto\sTcl\sExtension +D 2022-04-11T08:07:16.927 F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1 F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea F LICENSE.md df5091916dbb40e6e9686186587125e1b2ff51f022cc334e886c19a0e9982724 @@ -42,6 +42,7 @@ F doc/json-enhancements.md e356fc834781f1f1aa22ee300027a270b2c960122468499bf347b F doc/lemon.html efc0cd2345d66905505d98f862e1c571512def0ceb5b016cb658fd4918eb76a3 F doc/pager-invariants.txt 27fed9a70ddad2088750c4a2b493b63853da2710 F doc/shell_extend.html c2ce1584bd976613c984a0b7ea4680ef2d2c0ab0642540d730e0a899648e2294 +F doc/tcl_extension_intro.md bf404245b69db94b619287e3abf44b5e5f33736966c32bd27771bb584901b6d8 F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a F doc/vdbesort-memory.md 4da2639c14cd24a31e0af694b1a8dd37eaf277aff3867e9a8cc14046bc49df56 F doc/vfs-shm.txt e101f27ea02a8387ce46a05be2b1a902a021d37a @@ -1952,8 +1953,8 @@ F vsixtest/vsixtest.tcl 6a9a6ab600c25a91a7acc6293828957a386a8a93 F vsixtest/vsixtest.vcxproj.data 2ed517e100c66dc455b492e1a33350c1b20fbcdc F vsixtest/vsixtest.vcxproj.filters 37e51ffedcdb064aad6ff33b6148725226cd608e F vsixtest/vsixtest_TemporaryKey.pfx e5b1b036facdb453873e7084e1cae9102ccc67a0 -P 861ab023be283782131df5ccd9eff50cf2f06f48f1ea7defd3f08177a3df5889 -R 29bf4bbfab5e25a57abea25dca7857f3 +P 2596e7c439b6c4c89921079a78ba6174e99f766b30fab8763be20c1b631e49ec +R e8438fa69346d1c129bb2ec9bef46aab U larrybr -Z 916c41d1a88108766c11b193884acec2 +Z dacf014d371b12af7a2495c1fcf04998 # Remove this line to create a well-formed Fossil manifest. diff --git a/manifest.uuid b/manifest.uuid index 30250f8904..747cbefcb9 100644 --- a/manifest.uuid +++ b/manifest.uuid @@ -1 +1 @@ -2596e7c439b6c4c89921079a78ba6174e99f766b30fab8763be20c1b631e49ec \ No newline at end of file +33591833326e98882e0fa6225a7a55a7e915063a36bf24bb3c17c35e693d972e \ No newline at end of file -- 2.47.3