From: drh <> Date: Mon, 10 Jan 2022 13:55:08 +0000 (+0000) Subject: New proposal for -> and ->> operators. X-Git-Tag: version-3.38.0~119^2~4 X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=d3c6c3459b5aff110877428770e52df354c63360;p=thirdparty%2Fsqlite.git New proposal for -> and ->> operators. FossilOrigin-Name: 1108e12a2244edc6247050a0e9ad25b912bba6c57c71c51c2bc55167a6955175 --- diff --git a/doc/json-enhancements.md b/doc/json-enhancements.md index 8d83554ad2..d3ab6e0a43 100644 --- a/doc/json-enhancements.md +++ b/doc/json-enhancements.md @@ -3,110 +3,131 @@ This document summaries enhancements to the SQLite JSON support added in early 2022. -## 1.0 New feature summary: +## 1.0 Change summary: 1. New **->** and **->>** operators that work like MySQL and PostgreSQL (PG). - 2. New functions: **json_nextract()** and **json_ntype()**. - 3. JSON functions are built-in rather than being an extension. They + 2. JSON functions are built-in rather than being an extension. They are included by default, but can be omitted using the -DSQLITE_OMIT_JSON compile-time option. -## 2.0 The **json_nextract()** function. -The new **json_nextract()** function works like **json_extract()** with -one exception: if the input text in the first argument is not well-formed -JSON, then json_nextract() returns NULL whereas json_extract() raises -an error. The extra "n" in the name of json_nextract() can be throught -of as meaning "null-if-error". - -A call to json_nextract($JSON,$PATH) is logically equivalent to: - -> ~~~ -CASE WHEN json_valid($JSON) THEN json_extract($JSON,$PATH) END -~~~ - -The json_nextract() function is intended for use in tables where a -column might hold a mixture of datatypes - some rows holding JSON and other -rows holding primitive SQL datatypes such as INT, REAL, and TEXT. The -json_nextract() function makes it easier to write robust queries -against such tables. - -## 3.0 New operators **->** and **->>** +## 2.0 New operators **->** and **->>** The SQLite language adds two new binary operators **->** and **->>**. -The -> operator works like the two-argument version of json_nextract() -and the ->> operator works like the two-argument version of json_extract(). -The left-hand operand of -> and ->> is JSON. The right-hand operand -is a JSON path expression. These operators extract and return a value -from the left-hand JSON that is specified by right-hand path expression. - -The operators work exactly the same if the left-hand side is well-formed -JSON. The only difference is that if the left-hand side is not well-formed -JSON, the ->> raises an error whereas the -> operator simply returns NULL. - -### 3.1 Compatibility with MySQL - -The ->> operator should be compatible with MySQL in the sense that -a ->> operator that works in MySQL should work the same way in SQLite. -But (see below) the SQLite ->> operator is also extended to support PG -syntax so not every use of ->> that wworks in SQLite will work for MySQL. - -The -> operator is *mostly* compatible with MySQL. Key differences -between the SQLite -> operator and the MySQL -> operator are: - - * The SQLite -> operator returns NULL if the left-hand side is - not well-formed JSON whereas MySQL will raise an error. - - * When the JSON path expression on the right-hand side selects a - text value from the JSON, the -> operator in MySQL returns the - string quoted as if for JSON, whereas the SQLite -> operator - returns an unquoted SQL text value. - -This second difference - the handling of text values extracted from JSON - -is also a difference in the json_extract() function between SQLite and -MySQL. Because json_extract() has been in active use for 6 years, and -because the SQLite semantics seem to be more useful, there -are no plans to change json_extract() to make it compatible with MySQL. - -### 3.2 Compatibility with PostgreSQL (PG) - -The ->> operator in PG does not accept a JSON path expression as its -right-hand operand. Instead, PG looks for either a text string X -(which is then interpreted as the path "$.X") or an integer N (which -is then interpreted as "$[N]"). In order to make the SQLite ->> operator -compatible with the PG ->> operator, the SQLite ->> operator has been -extended so that its right-hand operand can be either a text label or -a integer array index, as it is in PG. The SQLite ->> operator also -accepts full JSON path expressions as well. - -The enhancement of accepting JSON path expression that consist of just -a bare object label or array index is unique to the -> and ->> operators. -All other places in the SQLite JSON interface that require JSON path -expressions continue to require well-formed JSON path expressions. -Only -> and ->> accept the PG-compatible abbreviated path expressions. - -The -> operator in SQLite is *mostly* compatible with the -> operator -in PG. The differences are the same as for MySQL. - -## 4.0 The **json_ntype()** function. - -The **json_ntype()** function works like **json_type()** except that when -the argument is not well-formed JSON, the json_ntype() function returns -NULL whereas json_type() raises an error. The extra "n" in the name can -be understood as standing for "null-if-error". - -The json_ntype($JSON) function is logically equivalent to: - -> ~~~ -CASE WHEN json_valid($JSON) THEN json_type($JSON) END -~~~ - -The json_ntype() function can be seen as an enhanced version of -the json_valid() function, that in addition to indicating whether or -not the string is well-formed JSON, also indicates the top-level type -of that JSON. - -## 5.0 JSON moved into the core +Both operators are similar to json_extract(). The left operand is +JSON and the right operand is a JSON path expression (possibly abbreviated +for compatibility with PG - see below). So they are similar to a +two-argument call to json_extract(). + +The difference between -> and ->> (and json_extract()) is as follows: + + * The -> operator always returns JSON. + + * The ->> operator converts the answer into a primitive SQL datatype + such as TEXT, INTEGER, REAL, or NULL. If a JSON object or array + is selected, that object or array is rendered as text. If a JSON + value is selected, that value is converted into its corresponding + SQL type + + * The json_extract() interface returns JSON when a JSON object or + array is selected, or a primitive SQL datatype when a JSON value + is selected. This is different from MySQL, in which json_extract() + always returns JSON, but the difference is retained because it has + worked that way for 6 years and changing it now would likely break + a lot of legacy code. + +In MySQL and PG, the ->> operator always returns TEXT (or NULL) and never +INTEGER or REAL. This is due to limitations in the type handling capabilities +of those systems. In MySQL and PG, the result type a function or operator +may only depend on the type of its arguments, never the value of its arguments. +But the underlying JSON type depends on the value of the JSON path +expression, not the type of the JSON path expression (which is always TEXT). +Hence, the result type of ->> in MySQL and PG is unable to vary according +to the type of the JSON value being extracted. + +The type system in SQLite is more general. Functions in SQLite are able +to return different datatypes depending on the value of their arguments. +So the ->> operator in SQLite is able to return TEXT, INTEGER, REAL, or NULL +depending on the JSON type of the value being extracted. This means that +the behavior of the ->> is slightly different in SQLite versus MySQL and PG +in that it will sometimes return INTEGER and REAL values, depending on its +inputs. It is possible to implement the ->> operator in SQLite so that it +always operates exactly like MySQL and PG and always returns TEXT or NULL, +but I have been unable to think of any situations where returning the +actual JSON value this would cause problems, so I'm including the enhanced +functionality in SQLite. + +The table below attempts to summarize the differences between the +-> and ->> operators and the json_extract() function, for SQLite, MySQL, +and PG. JSON values are shown using their SQL text representation but +in a bold font. + + + +
JSONPATH-> operator
(all)		->> operator
(MySQL/PG) + 		->> operator
(SQLite)		json_extract()
(SQLite) +
**'{"a":123}'** '$.a' **'123'** '123' 123 123 +
**'{"a":4.5}'** '$.a' **'4.5'** '4.5' 4.5 4.5 +
**'{"a":"xyz"}'** '$.a' **'"xyz"'** 'xyz' 'xyz' 'xyz' +
**'{"a":null}'** '$.a' **'null'** NULL NULL NULL +
**'{"a":[6,7,8]}'** '$.a' **'[6,7,8]'** '[6,7,8]' '[6,7,8]' **'[6,7,9]'** +
**'{"a":{"x":9}}'** '$.a' **'{"x":9}'** '{"x":9}' '{"x":9}' **'{"x":9}'** +
**'{"b":999}'** '$.a' NULL NULL NULL NULL +
+ +Important points about the table above: + + * The -> operator always returns either JSON or NULL. + + * The ->> operator never returns JSON. It always returns TEXT or NULL, or in the + case of SQLite, INTEGER or REAL. + + * The MySQL json_extract() function works exactly the same + as the MySQL -> operator. + + * The SQLite json_extract() operator works like -> for JSON objects and + arrays, and like ->> for JSON values. + + * The -> operator works the same for all systems. + + * The only difference in ->> between SQLite and other systems is that + when the JSON value is numeric, SQLite returns a numeric SQL value, + whereas the other systems return a text representation of the numeric + value. + +### 2.1 Abbreviated JSON path expressions for PG compatibility + +The table above always shows the full JSON path expression: '$.a'. But +PG does not accept this syntax. PG only allows a single JSON object label +name or a single integer array index. In order to provide compatibility +with PG, The -> and ->> operators in SQLite are extended to also support +a JSON object label or an integer array index for the right-hand side +operand, in addition to a full JSON path expression. + +Thus, a -> or ->> operator that works on MySQL will work in +SQLite. And a -> or ->> operator that works in PG will work in SQLite. +But because SQLite supports the union of the disjoint capabilities of +MySQL and PG, there will always be -> and ->> operators that work in +SQLite that do not work in one of MySQL and PG. This is an unavoidable +consequence of the different syntax for -> and ->> in MySQL and PG. + +In the following table, assume that "value1" is a JSON object and +"value2" is a JSON array. + + +
SQL expression Works in MySQL?Works in PG?Works in SQLite +
value1->'$.a' yes no yes +
value1->'a' no yes yes +
value2->'$[2]' yes no yes +
value2->2 no yes yes +
+ +The abbreviated JSON path expressions only work for the -> and ->> operators +in SQLite. The json_extract() function, and all other built-in SQLite +JSON functions, continue to require complete JSON path expressions for their +PATH arguments. + +## 3.0 JSON moved into the core The JSON interface is now moved into the SQLite core. diff --git a/manifest b/manifest index 2111bfa40b..2019f703d9 100644 --- a/manifest +++ b/manifest @@ -1,5 +1,5 @@ -C Typo\sfix\sin\sdoc/json-enhancements.md. -D 2022-01-09T20:51:59.795 +C New\sproposal\sfor\s->\sand\s->>\soperators. +D 2022-01-10T13:55:08.463 F .fossil-settings/empty-dirs dbb81e8fc0401ac46a1491ab34a7f2c7c0452f2f06b54ebb845d024ca8283ef1 F .fossil-settings/ignore-glob 35175cdfcf539b2318cb04a9901442804be81cd677d8b889fcc9149c21f239ea F LICENSE.md df5091916dbb40e6e9686186587125e1b2ff51f022cc334e886c19a0e9982724 @@ -38,7 +38,7 @@ F configure a2877fe63cc821af0df41abe70f1f7c4e97cb7e23a42e0a1402e8a2f55a88aa2 x F configure.ac 3ef6eeff4387585bfcab76b0c3f6e15a0618587bb90245dd5d44e4378141bb35 F contrib/sqlitecon.tcl 210a913ad63f9f991070821e599d600bd913e0ad F doc/F2FS.txt c1d4a0ae9711cfe0e1d8b019d154f1c29e0d3abfe820787ba1e9ed7691160fcd -F doc/json-enhancements.md b1731f584a8594660dcee86555e77559dbc447a88c010f010a7f5274bb050459 +F doc/json-enhancements.md 7f67a2e75de23958b9e767a15f6fb6abf918ef53f45f7dbb6d70ec7b55d71810 F doc/lemon.html efc0cd2345d66905505d98f862e1c571512def0ceb5b016cb658fd4918eb76a3 F doc/pager-invariants.txt 27fed9a70ddad2088750c4a2b493b63853da2710 F doc/trusted-schema.md 33625008620e879c7bcfbbfa079587612c434fa094d338b08242288d358c3e8a @@ -1938,8 +1938,8 @@ F vsixtest/vsixtest.tcl 6a9a6ab600c25a91a7acc6293828957a386a8a93 F vsixtest/vsixtest.vcxproj.data 2ed517e100c66dc455b492e1a33350c1b20fbcdc F vsixtest/vsixtest.vcxproj.filters 37e51ffedcdb064aad6ff33b6148725226cd608e F vsixtest/vsixtest_TemporaryKey.pfx e5b1b036facdb453873e7084e1cae9102ccc67a0 -P b8ac938f41eff8e5a23037c0e8b035a65e1b9505eca2a601221c195225595788 -R bb97ccd30a4bea2e78fa0684b28c1c7a +P c3b01d496479b3250a8895c245f79ab43ac469148d163593fea00894db195a37 +R 992262dfca636fa297052ec77a843cce U drh -Z 59da2a31f3cfc387aa42335fd8788184 +Z 83712db3104322990e05c7028e4a9b88 # Remove this line to create a well-formed Fossil manifest. diff --git a/manifest.uuid b/manifest.uuid index 173df7a789..d1f9996545 100644 --- a/manifest.uuid +++ b/manifest.uuid @@ -1 +1 @@ -c3b01d496479b3250a8895c245f79ab43ac469148d163593fea00894db195a37 \ No newline at end of file +1108e12a2244edc6247050a0e9ad25b912bba6c57c71c51c2bc55167a6955175 \ No newline at end of file