From 138bf3bc7f5230462fe215f8e16a15cff751f2f0 Mon Sep 17 00:00:00 2001 From: dan Date: Thu, 19 Feb 2015 18:06:40 +0000 Subject: [PATCH] Update various documentation comments in sqlite3ota.c and sqlite3ota.h. FossilOrigin-Name: 60e0a46b82dd9c704e8aa977d1ccdd73d388422f --- ext/ota/README.txt | 134 -------------------------------------- ext/ota/sqlite3ota.c | 150 +++++++++++++++++++++++++++++++++++-------- ext/ota/sqlite3ota.h | 40 ++++++------ manifest | 15 ++--- manifest.uuid | 2 +- 5 files changed, 150 insertions(+), 191 deletions(-) delete mode 100644 ext/ota/README.txt diff --git a/ext/ota/README.txt b/ext/ota/README.txt deleted file mode 100644 index 42ac29064c..0000000000 --- a/ext/ota/README.txt +++ /dev/null @@ -1,134 +0,0 @@ - -This file contains notes regarding the implementation of the OTA extension. -User documentation is in sqlite3ota.h. - -SQLite Hacks ------------- - - -1) PRAGMA pager_ota_mode=1: - - This pragma sets a flag on the pager associated with the main database only. - In a zipvfs system, this pragma is intercepted by zipvfs and the flag is set - on the lower level pager only. - - The flag can only be set when there is no open transaction and the pager does - not already have an open WAL file. Attempting to do so is an error. - - Once the flag has been set, it is not possible to open a regular WAL file. - If, when the next read-transaction is opened, a *-wal file is found or the - database header flags indicate that it is a wal-mode database, - SQLITE_CANTOPEN is returned. - - Otherwise, if no WAL file or flags are found, the pager opens the *-oal file - and uses it as a write-ahead-log with the *-shm data stored in heap-memory. - - The 8-bytes of "salt" at the start of an *-oal file is a copy of the 8 bytes - starting at offset 24 of the database file header (the change counter and the - number of pages in the file). If the *-oal file already exists when it is - opened, SQLite checks that the salt still matches the database header fields. - If not, it concludes that the database file has been written by a - rollback-mode client since the *-oal wa created and an SQLITE_BUSY_SNAPSHOT - error is returned. No read-transaction can be opened in this case. - - A pager with the pager_ota_mode flag set never runs a checkpoint. - - Other clients see a rollback-mode database on which the pager_ota_mode client - is holding a SHARED lock. There are no locks to arbitrate between multiple - pager_ota_mode connections. If two or more such connections attempt to write - simultaneously, the results are undefined. - -2) PRAGMA pager_ota_mode=2: - - The pager_ota_mode pragma may also be set to 2 if the main database is open - in WAL mode. This prevents SQLite from checkpointing the wal file as part - of sqlite3_close(). - - The effects of setting pager_ota_mode=2 if the db is not in WAL mode are - undefined. - -3) sqlite3_ckpt_open/step/close() - - API for performing (and resuming) incremental checkpoints. - - -The OTA extension ------------------ - -The OTA extension requires that the OTA update be packaged as an SQLite -database. The tables it expects to find are described in sqlite3ota.h. -Essentially, for each table xyz in the target database that the user wishes -to write to, a corresponding data_xyz table is created in the OTA database -and populated with one row for each row to update, insert or delete from -the target table. - -The OTA extension opens the target and OTA update databases using a single -database handle (the target database is "main", and the OTA update database is -attached as "ota"). It executes both the "pager_ota_mode" and "ota_mode" -pragmas described above. For each data_xyz table in then: - - * CREATEs an ota_xyz table in the OTA update database. - - * Loops through the data_xyz table, running the INSERT, UPDATE or DELETE - command on the corresponding target database table. Only the main b-tree - is updated by these statements. Modified pages are appended to the *-oal - file. - - Temporary triggers installed on the target database catch the old.* - values associated with any UPDATEd or DELETEd rows and store them in - the ota_xyz table (in the OTA update database). - - * For each index on the data_xyz table in the target database: - - Loop through a union of the data_xyz and ota_xyz tables in the order - specified by the data_xyz index. In other words, if the index is on - columns (a, b), read rows from the OTA update database using: - - SELECT * FROM data_xyz UNION ALL ota_xyz ORDER BY a, b; - - For each row visited, use an sqlite3_index_writer() VM to update the index - in the target database. - - * DROPs the ota_xyz table. - -At any point in the above, the process may be suspended by the user. In this -case the "ota_state" table is created in the OTA database, containing a single -row indicating the current table/index being processed and the number of updates -already performed on it, and the transaction on the target database is committed -to the *-oal file. The next OTA client will use the contents of the ota_state -table to continue the update from where this one left off. - -Alternatively, if the OTA update is completely applied, the transaction is -committed to the *-oal file and the database connection closed. sqlite3ota.c -then uses a rename() call to move the *-oal file to the corresponding *-wal -path. At that point it is finished - it does not take responsibility for -checkpointing the *-wal file. - - -Problems --------- - -The rename() call might not be portable. And in theory it is unsafe if some -other client starts writing the db file. - -When state is saved, the commit to the *-oal file and the commit to the OTA -update database are not atomic. So if the power fails at the wrong moment they -might get out of sync. As the main database will be committed before the OTA -update database this will likely either just pass unnoticed, or result in -SQLITE_CONSTRAINT errors (due to UNIQUE constraint violations). - -If some client does modify the target database mid OTA update, or some other -error occurs, the OTA extension will keep throwing errors. It's not really -clear how to get out of this state. The system could just by delete the OTA -update database and *-oal file and have the device download the update again -and start over. - -At present, for an UPDATE, both the new.* and old.* records are collected in -the ota_xyz table. And for both UPDATEs and DELETEs all fields are collected. -This means we're probably writing a lot more data to disk when saving the -state of an ongoing update to the OTA update database than is strictly -necessary. - - - - diff --git a/ext/ota/sqlite3ota.c b/ext/ota/sqlite3ota.c index ae4c9dd698..6351dfe852 100644 --- a/ext/ota/sqlite3ota.c +++ b/ext/ota/sqlite3ota.c @@ -9,6 +9,75 @@ ** May you share freely, never taking more than you give. ** ************************************************************************* +** +** +** OVERVIEW +** +** The OTA extension requires that the OTA update be packaged as an +** SQLite database. The tables it expects to find are described in +** sqlite3ota.h. Essentially, for each table xyz in the target database +** that the user wishes to write to, a corresponding data_xyz table is +** created in the OTA database and populated with one row for each row to +** update, insert or delete from the target table. +** +** The update proceeds in three stages: +** +** 1) The database is updated. The modified database pages are written +** to a *-oal file. A *-oal file is just like a *-wal file, except +** that it is named "-oal" instead of "-wal". +** Because regular SQLite clients do not look for file named +** "-oal", they go on using the original database in +** rollback mode while the *-oal file is being generated. +** +** During this stage OTA does not update the database by writing +** directly to the target tables. Instead it creates "imposter" +** tables using the SQLITE_TESTCTRL_IMPOSTER interface that it uses +** to update each b-tree individually. All updates required by each +** b-tree are completed before moving on to the next, and all +** updates are done in sorted key order. +** +** 2) The "-oal" file is moved to the equivalent "-wal" +** location using a call to rename(2). Before doing this the OTA +** module takes an EXCLUSIVE lock on the database file, ensuring +** that there are no other active readers. +** +** Once the EXCLUSIVE lock is released, any other database readers +** detect the new *-wal file and read the database in wal mode. At +** this point they see the new version of the database - including +** the updates made as part of the OTA update. +** +** 3) The new *-wal file is checkpointed. This proceeds in the same way +** as a regular database checkpoint, except that a single frame is +** checkpointed each time sqlite3ota_step() is called. If the OTA +** handle is closed before the entire *-wal file is checkpointed, +** the checkpoint progress is saved in the OTA database and the +** checkpoint can be resumed by another OTA client at some point in +** the future. +** +** POTENTIAL PROBLEMS +** +** The rename() call might not be portable. And OTA is not currently +** syncing the directory after renaming the file. +** +** When state is saved, any commit to the *-oal file and the commit to +** the OTA update database are not atomic. So if the power fails at the +** wrong moment they might get out of sync. As the main database will be +** committed before the OTA update database this will likely either just +** pass unnoticed, or result in SQLITE_CONSTRAINT errors (due to UNIQUE +** constraint violations). +** +** If some client does modify the target database mid OTA update, or some +** other error occurs, the OTA extension will keep throwing errors. It's +** not really clear how to get out of this state. The system could just +** by delete the OTA update database and *-oal file and have the device +** download the update again and start over. +** +** At present, for an UPDATE, both the new.* and old.* records are +** collected in the ota_xyz table. And for both UPDATEs and DELETEs all +** fields are collected. This means we're probably writing a lot more +** data to disk when saving the state of an ongoing update to the OTA +** update database than is strictly necessary. +** */ #include @@ -2403,34 +2472,59 @@ sqlite3_int64 sqlite3ota_progress(sqlite3ota *pOta){ ** Beginning of OTA VFS shim methods. The VFS shim modifies the behaviour ** of a standard VFS in the following ways: ** -** 1. Whenever the first page of a main database file is read or -** written, the value of the change-counter cookie is stored in -** ota_file.iCookie. Similarly, the value of the "write-version" -** database header field is stored in ota_file.iWriteVer. This ensures -** that the values are always trustworthy within an open transaction. -** -** 2. When the ota handle is in OTA_STAGE_OAL or OTA_STAGE_CKPT state, all -** EXCLUSIVE lock attempts on the target database fail. This prevents -** sqlite3_close() from running an automatic checkpoint. Until the -** ota handle reaches OTA_STAGE_DONE - at that point the automatic -** checkpoint may be required to delete the *-wal file. -** -** 3. In OTA_STAGE_OAL, the *-shm file is stored in memory. All xShmLock() -** calls are noops. This is just an optimization. -** -** 4. In OTA_STAGE_OAL mode, when SQLite calls xAccess() to check if a -** *-wal file associated with the target database exists, the following -** special handling applies: -** -** a) if the *-wal file does exist, return SQLITE_CANTOPEN. An OTA -** target database may not be in wal mode already. -** -** b) if the *-wal file does not exist, set the output parameter to -** non-zero (to tell SQLite that it does exist) anyway. -** -** 5. In OTA_STAGE_OAL mode, if SQLite tries to open a *-wal file -** associated with a target database, open the corresponding *-oal file -** instead. +** 1. Whenever the first page of a main database file is read or +** written, the value of the change-counter cookie is stored in +** ota_file.iCookie. Similarly, the value of the "write-version" +** database header field is stored in ota_file.iWriteVer. This ensures +** that the values are always trustworthy within an open transaction. +** +** 2. Whenever an SQLITE_OPEN_WAL file is opened, the (ota_file.pWalFd) +** member variable of the associated database file descriptor is set +** to point to the new file. A mutex protected linked list of all main +** db fds opened using a particular OTA VFS is maintained at +** ota_vfs.pMain to facilitate this. +** +** 3. Using a new file-control "SQLITE_FCNTL_OTA", a main db ota_file +** object can be marked as the target database of an OTA update. This +** turns on the following extra special behaviour: +** +** 3a. If xAccess() is called to check if there exists a *-wal file +** associated with an OTA target database currently in OTA_STAGE_OAL +** stage (preparing the *-oal file), the following special handling +** applies: +** +** * if the *-wal file does exist, return SQLITE_CANTOPEN. An OTA +** target database may not be in wal mode already. +** +** * if the *-wal file does not exist, set the output parameter to +** non-zero (to tell SQLite that it does exist) anyway. +** +** Then, when xOpen() is called to open the *-wal file associated with +** the OTA target in OTA_STAGE_OAL stage, instead of opening the *-wal +** file, the ota vfs opens the corresponding *-oal file instead. +** +** 3b. The *-shm pages returned by xShmMap() for a target db file in +** OTA_STAGE_OAL mode are actually stored in heap memory. This is to +** avoid creating a *-shm file on disk. Additionally, xShmLock() calls +** are no-ops on target database files in OTA_STAGE_OAL mode. This is +** because assert() statements in some VFS implementations fail if +** xShmLock() is called before xShmMap(). +** +** 3c. If an EXCLUSIVE lock is attempted on a target database file in any +** mode except OTA_STAGE_DONE (all work completed and checkpointed), it +** fails with an SQLITE_BUSY error. This is to stop OTA connections +** from automatically checkpointing a *-wal (or *-oal) file from within +** sqlite3_close(). +** +** 3d. In OTA_STAGE_CAPTURE mode, all xRead() calls on the wal file, and +** all xWrite() calls on the target database file perform no IO. +** Instead the frame and page numbers that would be read and written +** are recorded. Additionally, successful attempts to obtain exclusive +** xShmLock() WRITER, CHECKPOINTER and READ0 locks on the target +** database file are recorded. xShmLock() calls to unlock the same +** locks are no-ops (so that once obtained, these locks are never +** relinquished). Finally, calls to xSync() on the target database +** file fail with SQLITE_INTERNAL errors. */ /* diff --git a/ext/ota/sqlite3ota.h b/ext/ota/sqlite3ota.h index f3c53a1790..7d3d1e9494 100644 --- a/ext/ota/sqlite3ota.h +++ b/ext/ota/sqlite3ota.h @@ -24,9 +24,9 @@ ** containing the entry being inserted or deleted must be modified. If the ** working set of leaves is larger than the available cache memory, then a ** single leaf that is modified more than once as part of the transaction -** may be loaded from or written to the persistent media more than once. +** may be loaded from or written to the persistent media multiple times. ** Additionally, because the index updates are likely to be applied in -** random order, access to pages within the databse is also likely to be in +** random order, access to pages within the database is also likely to be in ** random order, which is itself quite inefficient. ** ** One way to improve the situation is to sort the operations on each index @@ -40,10 +40,10 @@ ** Additionally, this extension allows the work involved in writing the ** large transaction to be broken down into sub-transactions performed ** sequentially by separate processes. This is useful if the system cannot -** guarantee that a single update process may run for long enough to apply -** the entire update, for example because the update is running on a mobile -** device that is frequently rebooted. Even after the writer process has -** committed one or more sub-transactions, other database clients continue +** guarantee that a single update process will run for long enough to apply +** the entire update, for example because the update is being applied on a +** mobile device that is frequently rebooted. Even after the writer process +** has committed one or more sub-transactions, other database clients continue ** to read from the original database snapshot. In other words, partially ** applied transactions are not visible to other clients. ** @@ -62,8 +62,8 @@ ** * INSERT statements may not use any default values. ** ** * UPDATE and DELETE statements must identify their target rows by -** PRIMARY KEY values. If the table being written has no PRIMARY KEY -** declaration, affected rows must be identified by rowid. +** PRIMARY KEY values. If the table being written has no PRIMARY KEY, +** affected rows must be identified by rowid. ** ** * UPDATE statements may not modify PRIMARY KEY columns. ** @@ -148,11 +148,11 @@ ** the new values of all columns being update. The text value in the ** "ota_control" column must contain the same number of characters as ** there are columns in the target database table, and must consist entirely -** of "x" and "." characters. For each column that is being updated, -** the corresponding character is set to "x". For those that remain as -** they are, the corresponding character of the ota_control value should -** be set to ".". For example, given the tables above, the update -** statement: +** of 'x' and '.' characters (or in some special cases 'd' - see below). For +** each column that is being updated, the corresponding character is set to +** 'x'. For those that remain as they are, the corresponding character of the +** ota_control value should be set to '.'. For example, given the tables +** above, the update statement: ** ** UPDATE t1 SET c = 'usa' WHERE a = 4; ** @@ -207,10 +207,10 @@ ** ** 4) Calls sqlite3ota_close() to close the OTA update handle. If ** sqlite3ota_step() has been called enough times to completely -** apply the update to the target database, then it is committed -** and made visible to other database clients at this point. -** Otherwise, the state of the OTA update application is saved -** in the OTA database for later resumption. +** apply the update to the target database, then the OTA database +** is marked as fully applied. Otherwise, the state of the OTA +** update application is saved in the OTA database for later +** resumption. ** ** See comments below for more detail on APIs. ** @@ -285,9 +285,9 @@ int sqlite3ota_step(sqlite3ota *pOta); /* ** Close an OTA handle. ** -** If the OTA update has been completely applied, commit it to the target -** database. Otherwise, assuming no error has occurred, save the current -** state of the OTA update appliation to the OTA database. +** If the OTA update has been completely applied, mark the OTA database +** as fully applied. Otherwise, assuming no error has occurred, save the +** current state of the OTA update appliation to the OTA database. ** ** If an error has already occurred as part of an sqlite3ota_step() ** or sqlite3ota_open() call, or if one occurs within this function, an diff --git a/manifest b/manifest index 4d38e28cd7..837a4081da 100644 --- a/manifest +++ b/manifest @@ -1,5 +1,5 @@ -C Merge\slatest\strunk\schanges\swith\sthis\sbranch. -D 2015-02-19T14:41:24.386 +C Update\svarious\sdocumentation\scomments\sin\ssqlite3ota.c\sand\ssqlite3ota.h. +D 2015-02-19T18:06:40.917 F Makefile.arm-wince-mingw32ce-gcc d6df77f1f48d690bd73162294bbba7f59507c72f F Makefile.in 6b9e7677829aa94b9f30949656e27312aefb9a46 F Makefile.linux-gcc 91d710bdc4998cb015f39edf3cb314ec4f4d7e23 @@ -123,7 +123,6 @@ F ext/misc/totype.c 4a167594e791abeed95e0a8db028822b5e8fe512 F ext/misc/vfslog.c fe40fab5c077a40477f7e5eba994309ecac6cc95 F ext/misc/vtshim.c babb0dc2bf116029e3e7c9a618b8a1377045303e F ext/misc/wholenumber.c 784b12543d60702ebdd47da936e278aa03076212 -F ext/ota/README.txt 2ce4ffbb0aaa6731b041c27a7359f9a5f1c69152 F ext/ota/ota.c c11a85af71dccc45976622fe7a51169a481caa91 F ext/ota/ota1.test ba408c5e777c320ef72f328e20cd2ae2a8888cda F ext/ota/ota10.test 85e0f6e7964db5007590c1b299e75211ed4240d4 @@ -138,8 +137,8 @@ F ext/ota/ota9.test d3eee95dd836824d07a22e5efcdb7bf6e869358b F ext/ota/otaA.test ef4bfa8cfd4ed814ae86f7457b64aa2f18c90171 F ext/ota/otafault.test 8c43586c2b96ca16bbce00b5d7e7d67316126db8 F ext/ota/otafault2.test fa202a98ca221faec318f3e5c5f39485b1256561 -F ext/ota/sqlite3ota.c 6c329a3c1f1ca625f51161321724b25c24646f2d -F ext/ota/sqlite3ota.h 1cc7201086fe65a36957740381485a24738c4077 +F ext/ota/sqlite3ota.c a8e19ef2a297627fdc1e18f8679110afeaa5cd4b +F ext/ota/sqlite3ota.h 69106b04616f4e7e565aa4dc2092a2f095212cc2 F ext/ota/test_ota.c 9ec6ea945282f65f67f0e0468dad79a489818f44 F ext/rtree/README 6315c0d73ebf0ec40dedb5aa0e942bc8b54e3761 F ext/rtree/rtree.c 14e6239434d4e3f65d3e90320713f26aa24e167f @@ -1258,7 +1257,7 @@ F tool/vdbe_profile.tcl 67746953071a9f8f2f668b73fe899074e2c6d8c1 F tool/warnings-clang.sh f6aa929dc20ef1f856af04a730772f59283631d4 F tool/warnings.sh 0abfd78ceb09b7f7c27c688c8e3fe93268a13b32 F tool/win/sqlite.vsix deb315d026cc8400325c5863eef847784a219a2f -P a3c1bc5d5e3f4b197f48cbbc240608e94bfc2b45 81f242e338d6122e27aad86986bfd140012c6582 -R d299d5655569d5895ceed70c48c0588b +P 6f5888a5e430feb5d9a50009a2eb103d9945bd22 +R 536b36bcdeeeb4de103e032d97a52335 U dan -Z b1c159c49abb337fc6eddedfd6fb5f7b +Z 4148e3e31feb6bcdc60726ac41a858b3 diff --git a/manifest.uuid b/manifest.uuid index d8db68537e..48aabc2dfe 100644 --- a/manifest.uuid +++ b/manifest.uuid @@ -1 +1 @@ -6f5888a5e430feb5d9a50009a2eb103d9945bd22 \ No newline at end of file +60e0a46b82dd9c704e8aa977d1ccdd73d388422f \ No newline at end of file -- 2.47.3