+++ /dev/null
-
-This file contains notes regarding the implementation of the OTA extension.
-User documentation is in sqlite3ota.h.
-
-SQLite Hacks
-------------
-
-
-1) PRAGMA pager_ota_mode=1:
-
- This pragma sets a flag on the pager associated with the main database only.
- In a zipvfs system, this pragma is intercepted by zipvfs and the flag is set
- on the lower level pager only.
-
- The flag can only be set when there is no open transaction and the pager does
- not already have an open WAL file. Attempting to do so is an error.
-
- Once the flag has been set, it is not possible to open a regular WAL file.
- If, when the next read-transaction is opened, a *-wal file is found or the
- database header flags indicate that it is a wal-mode database,
- SQLITE_CANTOPEN is returned.
-
- Otherwise, if no WAL file or flags are found, the pager opens the *-oal file
- and uses it as a write-ahead-log with the *-shm data stored in heap-memory.
-
- The 8-bytes of "salt" at the start of an *-oal file is a copy of the 8 bytes
- starting at offset 24 of the database file header (the change counter and the
- number of pages in the file). If the *-oal file already exists when it is
- opened, SQLite checks that the salt still matches the database header fields.
- If not, it concludes that the database file has been written by a
- rollback-mode client since the *-oal wa created and an SQLITE_BUSY_SNAPSHOT
- error is returned. No read-transaction can be opened in this case.
-
- A pager with the pager_ota_mode flag set never runs a checkpoint.
-
- Other clients see a rollback-mode database on which the pager_ota_mode client
- is holding a SHARED lock. There are no locks to arbitrate between multiple
- pager_ota_mode connections. If two or more such connections attempt to write
- simultaneously, the results are undefined.
-
-2) PRAGMA pager_ota_mode=2:
-
- The pager_ota_mode pragma may also be set to 2 if the main database is open
- in WAL mode. This prevents SQLite from checkpointing the wal file as part
- of sqlite3_close().
-
- The effects of setting pager_ota_mode=2 if the db is not in WAL mode are
- undefined.
-
-3) sqlite3_ckpt_open/step/close()
-
- API for performing (and resuming) incremental checkpoints.
-
-
-The OTA extension
------------------
-
-The OTA extension requires that the OTA update be packaged as an SQLite
-database. The tables it expects to find are described in sqlite3ota.h.
-Essentially, for each table xyz in the target database that the user wishes
-to write to, a corresponding data_xyz table is created in the OTA database
-and populated with one row for each row to update, insert or delete from
-the target table.
-
-The OTA extension opens the target and OTA update databases using a single
-database handle (the target database is "main", and the OTA update database is
-attached as "ota"). It executes both the "pager_ota_mode" and "ota_mode"
-pragmas described above. For each data_xyz table in then:
-
- * CREATEs an ota_xyz table in the OTA update database.
-
- * Loops through the data_xyz table, running the INSERT, UPDATE or DELETE
- command on the corresponding target database table. Only the main b-tree
- is updated by these statements. Modified pages are appended to the *-oal
- file.
-
- Temporary triggers installed on the target database catch the old.*
- values associated with any UPDATEd or DELETEd rows and store them in
- the ota_xyz table (in the OTA update database).
-
- * For each index on the data_xyz table in the target database:
-
- Loop through a union of the data_xyz and ota_xyz tables in the order
- specified by the data_xyz index. In other words, if the index is on
- columns (a, b), read rows from the OTA update database using:
-
- SELECT * FROM data_xyz UNION ALL ota_xyz ORDER BY a, b;
-
- For each row visited, use an sqlite3_index_writer() VM to update the index
- in the target database.
-
- * DROPs the ota_xyz table.
-
-At any point in the above, the process may be suspended by the user. In this
-case the "ota_state" table is created in the OTA database, containing a single
-row indicating the current table/index being processed and the number of updates
-already performed on it, and the transaction on the target database is committed
-to the *-oal file. The next OTA client will use the contents of the ota_state
-table to continue the update from where this one left off.
-
-Alternatively, if the OTA update is completely applied, the transaction is
-committed to the *-oal file and the database connection closed. sqlite3ota.c
-then uses a rename() call to move the *-oal file to the corresponding *-wal
-path. At that point it is finished - it does not take responsibility for
-checkpointing the *-wal file.
-
-
-Problems
---------
-
-The rename() call might not be portable. And in theory it is unsafe if some
-other client starts writing the db file.
-
-When state is saved, the commit to the *-oal file and the commit to the OTA
-update database are not atomic. So if the power fails at the wrong moment they
-might get out of sync. As the main database will be committed before the OTA
-update database this will likely either just pass unnoticed, or result in
-SQLITE_CONSTRAINT errors (due to UNIQUE constraint violations).
-
-If some client does modify the target database mid OTA update, or some other
-error occurs, the OTA extension will keep throwing errors. It's not really
-clear how to get out of this state. The system could just by delete the OTA
-update database and *-oal file and have the device download the update again
-and start over.
-
-At present, for an UPDATE, both the new.* and old.* records are collected in
-the ota_xyz table. And for both UPDATEs and DELETEs all fields are collected.
-This means we're probably writing a lot more data to disk when saving the
-state of an ongoing update to the OTA update database than is strictly
-necessary.
-
-
-
-
** May you share freely, never taking more than you give.
**
*************************************************************************
+**
+**
+** OVERVIEW
+**
+** The OTA extension requires that the OTA update be packaged as an
+** SQLite database. The tables it expects to find are described in
+** sqlite3ota.h. Essentially, for each table xyz in the target database
+** that the user wishes to write to, a corresponding data_xyz table is
+** created in the OTA database and populated with one row for each row to
+** update, insert or delete from the target table.
+**
+** The update proceeds in three stages:
+**
+** 1) The database is updated. The modified database pages are written
+** to a *-oal file. A *-oal file is just like a *-wal file, except
+** that it is named "<database>-oal" instead of "<database>-wal".
+** Because regular SQLite clients do not look for file named
+** "<database>-oal", they go on using the original database in
+** rollback mode while the *-oal file is being generated.
+**
+** During this stage OTA does not update the database by writing
+** directly to the target tables. Instead it creates "imposter"
+** tables using the SQLITE_TESTCTRL_IMPOSTER interface that it uses
+** to update each b-tree individually. All updates required by each
+** b-tree are completed before moving on to the next, and all
+** updates are done in sorted key order.
+**
+** 2) The "<database>-oal" file is moved to the equivalent "<database>-wal"
+** location using a call to rename(2). Before doing this the OTA
+** module takes an EXCLUSIVE lock on the database file, ensuring
+** that there are no other active readers.
+**
+** Once the EXCLUSIVE lock is released, any other database readers
+** detect the new *-wal file and read the database in wal mode. At
+** this point they see the new version of the database - including
+** the updates made as part of the OTA update.
+**
+** 3) The new *-wal file is checkpointed. This proceeds in the same way
+** as a regular database checkpoint, except that a single frame is
+** checkpointed each time sqlite3ota_step() is called. If the OTA
+** handle is closed before the entire *-wal file is checkpointed,
+** the checkpoint progress is saved in the OTA database and the
+** checkpoint can be resumed by another OTA client at some point in
+** the future.
+**
+** POTENTIAL PROBLEMS
+**
+** The rename() call might not be portable. And OTA is not currently
+** syncing the directory after renaming the file.
+**
+** When state is saved, any commit to the *-oal file and the commit to
+** the OTA update database are not atomic. So if the power fails at the
+** wrong moment they might get out of sync. As the main database will be
+** committed before the OTA update database this will likely either just
+** pass unnoticed, or result in SQLITE_CONSTRAINT errors (due to UNIQUE
+** constraint violations).
+**
+** If some client does modify the target database mid OTA update, or some
+** other error occurs, the OTA extension will keep throwing errors. It's
+** not really clear how to get out of this state. The system could just
+** by delete the OTA update database and *-oal file and have the device
+** download the update again and start over.
+**
+** At present, for an UPDATE, both the new.* and old.* records are
+** collected in the ota_xyz table. And for both UPDATEs and DELETEs all
+** fields are collected. This means we're probably writing a lot more
+** data to disk when saving the state of an ongoing update to the OTA
+** update database than is strictly necessary.
+**
*/
#include <assert.h>
** Beginning of OTA VFS shim methods. The VFS shim modifies the behaviour
** of a standard VFS in the following ways:
**
-** 1. Whenever the first page of a main database file is read or
-** written, the value of the change-counter cookie is stored in
-** ota_file.iCookie. Similarly, the value of the "write-version"
-** database header field is stored in ota_file.iWriteVer. This ensures
-** that the values are always trustworthy within an open transaction.
-**
-** 2. When the ota handle is in OTA_STAGE_OAL or OTA_STAGE_CKPT state, all
-** EXCLUSIVE lock attempts on the target database fail. This prevents
-** sqlite3_close() from running an automatic checkpoint. Until the
-** ota handle reaches OTA_STAGE_DONE - at that point the automatic
-** checkpoint may be required to delete the *-wal file.
-**
-** 3. In OTA_STAGE_OAL, the *-shm file is stored in memory. All xShmLock()
-** calls are noops. This is just an optimization.
-**
-** 4. In OTA_STAGE_OAL mode, when SQLite calls xAccess() to check if a
-** *-wal file associated with the target database exists, the following
-** special handling applies:
-**
-** a) if the *-wal file does exist, return SQLITE_CANTOPEN. An OTA
-** target database may not be in wal mode already.
-**
-** b) if the *-wal file does not exist, set the output parameter to
-** non-zero (to tell SQLite that it does exist) anyway.
-**
-** 5. In OTA_STAGE_OAL mode, if SQLite tries to open a *-wal file
-** associated with a target database, open the corresponding *-oal file
-** instead.
+** 1. Whenever the first page of a main database file is read or
+** written, the value of the change-counter cookie is stored in
+** ota_file.iCookie. Similarly, the value of the "write-version"
+** database header field is stored in ota_file.iWriteVer. This ensures
+** that the values are always trustworthy within an open transaction.
+**
+** 2. Whenever an SQLITE_OPEN_WAL file is opened, the (ota_file.pWalFd)
+** member variable of the associated database file descriptor is set
+** to point to the new file. A mutex protected linked list of all main
+** db fds opened using a particular OTA VFS is maintained at
+** ota_vfs.pMain to facilitate this.
+**
+** 3. Using a new file-control "SQLITE_FCNTL_OTA", a main db ota_file
+** object can be marked as the target database of an OTA update. This
+** turns on the following extra special behaviour:
+**
+** 3a. If xAccess() is called to check if there exists a *-wal file
+** associated with an OTA target database currently in OTA_STAGE_OAL
+** stage (preparing the *-oal file), the following special handling
+** applies:
+**
+** * if the *-wal file does exist, return SQLITE_CANTOPEN. An OTA
+** target database may not be in wal mode already.
+**
+** * if the *-wal file does not exist, set the output parameter to
+** non-zero (to tell SQLite that it does exist) anyway.
+**
+** Then, when xOpen() is called to open the *-wal file associated with
+** the OTA target in OTA_STAGE_OAL stage, instead of opening the *-wal
+** file, the ota vfs opens the corresponding *-oal file instead.
+**
+** 3b. The *-shm pages returned by xShmMap() for a target db file in
+** OTA_STAGE_OAL mode are actually stored in heap memory. This is to
+** avoid creating a *-shm file on disk. Additionally, xShmLock() calls
+** are no-ops on target database files in OTA_STAGE_OAL mode. This is
+** because assert() statements in some VFS implementations fail if
+** xShmLock() is called before xShmMap().
+**
+** 3c. If an EXCLUSIVE lock is attempted on a target database file in any
+** mode except OTA_STAGE_DONE (all work completed and checkpointed), it
+** fails with an SQLITE_BUSY error. This is to stop OTA connections
+** from automatically checkpointing a *-wal (or *-oal) file from within
+** sqlite3_close().
+**
+** 3d. In OTA_STAGE_CAPTURE mode, all xRead() calls on the wal file, and
+** all xWrite() calls on the target database file perform no IO.
+** Instead the frame and page numbers that would be read and written
+** are recorded. Additionally, successful attempts to obtain exclusive
+** xShmLock() WRITER, CHECKPOINTER and READ0 locks on the target
+** database file are recorded. xShmLock() calls to unlock the same
+** locks are no-ops (so that once obtained, these locks are never
+** relinquished). Finally, calls to xSync() on the target database
+** file fail with SQLITE_INTERNAL errors.
*/
/*
** containing the entry being inserted or deleted must be modified. If the
** working set of leaves is larger than the available cache memory, then a
** single leaf that is modified more than once as part of the transaction
-** may be loaded from or written to the persistent media more than once.
+** may be loaded from or written to the persistent media multiple times.
** Additionally, because the index updates are likely to be applied in
-** random order, access to pages within the databse is also likely to be in
+** random order, access to pages within the database is also likely to be in
** random order, which is itself quite inefficient.
**
** One way to improve the situation is to sort the operations on each index
** Additionally, this extension allows the work involved in writing the
** large transaction to be broken down into sub-transactions performed
** sequentially by separate processes. This is useful if the system cannot
-** guarantee that a single update process may run for long enough to apply
-** the entire update, for example because the update is running on a mobile
-** device that is frequently rebooted. Even after the writer process has
-** committed one or more sub-transactions, other database clients continue
+** guarantee that a single update process will run for long enough to apply
+** the entire update, for example because the update is being applied on a
+** mobile device that is frequently rebooted. Even after the writer process
+** has committed one or more sub-transactions, other database clients continue
** to read from the original database snapshot. In other words, partially
** applied transactions are not visible to other clients.
**
** * INSERT statements may not use any default values.
**
** * UPDATE and DELETE statements must identify their target rows by
-** PRIMARY KEY values. If the table being written has no PRIMARY KEY
-** declaration, affected rows must be identified by rowid.
+** PRIMARY KEY values. If the table being written has no PRIMARY KEY,
+** affected rows must be identified by rowid.
**
** * UPDATE statements may not modify PRIMARY KEY columns.
**
** the new values of all columns being update. The text value in the
** "ota_control" column must contain the same number of characters as
** there are columns in the target database table, and must consist entirely
-** of "x" and "." characters. For each column that is being updated,
-** the corresponding character is set to "x". For those that remain as
-** they are, the corresponding character of the ota_control value should
-** be set to ".". For example, given the tables above, the update
-** statement:
+** of 'x' and '.' characters (or in some special cases 'd' - see below). For
+** each column that is being updated, the corresponding character is set to
+** 'x'. For those that remain as they are, the corresponding character of the
+** ota_control value should be set to '.'. For example, given the tables
+** above, the update statement:
**
** UPDATE t1 SET c = 'usa' WHERE a = 4;
**
**
** 4) Calls sqlite3ota_close() to close the OTA update handle. If
** sqlite3ota_step() has been called enough times to completely
-** apply the update to the target database, then it is committed
-** and made visible to other database clients at this point.
-** Otherwise, the state of the OTA update application is saved
-** in the OTA database for later resumption.
+** apply the update to the target database, then the OTA database
+** is marked as fully applied. Otherwise, the state of the OTA
+** update application is saved in the OTA database for later
+** resumption.
**
** See comments below for more detail on APIs.
**
/*
** Close an OTA handle.
**
-** If the OTA update has been completely applied, commit it to the target
-** database. Otherwise, assuming no error has occurred, save the current
-** state of the OTA update appliation to the OTA database.
+** If the OTA update has been completely applied, mark the OTA database
+** as fully applied. Otherwise, assuming no error has occurred, save the
+** current state of the OTA update appliation to the OTA database.
**
** If an error has already occurred as part of an sqlite3ota_step()
** or sqlite3ota_open() call, or if one occurs within this function, an
-C Merge\slatest\strunk\schanges\swith\sthis\sbranch.
-D 2015-02-19T14:41:24.386
+C Update\svarious\sdocumentation\scomments\sin\ssqlite3ota.c\sand\ssqlite3ota.h.
+D 2015-02-19T18:06:40.917
F Makefile.arm-wince-mingw32ce-gcc d6df77f1f48d690bd73162294bbba7f59507c72f
F Makefile.in 6b9e7677829aa94b9f30949656e27312aefb9a46
F Makefile.linux-gcc 91d710bdc4998cb015f39edf3cb314ec4f4d7e23
F ext/misc/vfslog.c fe40fab5c077a40477f7e5eba994309ecac6cc95
F ext/misc/vtshim.c babb0dc2bf116029e3e7c9a618b8a1377045303e
F ext/misc/wholenumber.c 784b12543d60702ebdd47da936e278aa03076212
-F ext/ota/README.txt 2ce4ffbb0aaa6731b041c27a7359f9a5f1c69152
F ext/ota/ota.c c11a85af71dccc45976622fe7a51169a481caa91
F ext/ota/ota1.test ba408c5e777c320ef72f328e20cd2ae2a8888cda
F ext/ota/ota10.test 85e0f6e7964db5007590c1b299e75211ed4240d4
F ext/ota/otaA.test ef4bfa8cfd4ed814ae86f7457b64aa2f18c90171
F ext/ota/otafault.test 8c43586c2b96ca16bbce00b5d7e7d67316126db8
F ext/ota/otafault2.test fa202a98ca221faec318f3e5c5f39485b1256561
-F ext/ota/sqlite3ota.c 6c329a3c1f1ca625f51161321724b25c24646f2d
-F ext/ota/sqlite3ota.h 1cc7201086fe65a36957740381485a24738c4077
+F ext/ota/sqlite3ota.c a8e19ef2a297627fdc1e18f8679110afeaa5cd4b
+F ext/ota/sqlite3ota.h 69106b04616f4e7e565aa4dc2092a2f095212cc2
F ext/ota/test_ota.c 9ec6ea945282f65f67f0e0468dad79a489818f44
F ext/rtree/README 6315c0d73ebf0ec40dedb5aa0e942bc8b54e3761
F ext/rtree/rtree.c 14e6239434d4e3f65d3e90320713f26aa24e167f
F tool/warnings-clang.sh f6aa929dc20ef1f856af04a730772f59283631d4
F tool/warnings.sh 0abfd78ceb09b7f7c27c688c8e3fe93268a13b32
F tool/win/sqlite.vsix deb315d026cc8400325c5863eef847784a219a2f
-P a3c1bc5d5e3f4b197f48cbbc240608e94bfc2b45 81f242e338d6122e27aad86986bfd140012c6582
-R d299d5655569d5895ceed70c48c0588b
+P 6f5888a5e430feb5d9a50009a2eb103d9945bd22
+R 536b36bcdeeeb4de103e032d97a52335
U dan
-Z b1c159c49abb337fc6eddedfd6fb5f7b
+Z 4148e3e31feb6bcdc60726ac41a858b3
-6f5888a5e430feb5d9a50009a2eb103d9945bd22
\ No newline at end of file
+60e0a46b82dd9c704e8aa977d1ccdd73d388422f
\ No newline at end of file
projects / thirdparty / sqlite.git / commitdiff
