`Datatypes In SQLite Version 3 <https://sqlite.org/datatype3.html>`_
-.. _sqlite_concurrency:
-
-Database Locking Behavior / Concurrency
----------------------------------------
-
-SQLite is not designed for a high level of write concurrency. The database
-itself, being a file, is locked completely during write operations within
-transactions, meaning exactly one "connection" (in reality a file handle)
-has exclusive access to the database during this period - all other
-"connections" will be blocked during this time.
-
-The Python DBAPI specification also calls for a connection model that is
-always in a transaction; there is no ``connection.begin()`` method,
-only ``connection.commit()`` and ``connection.rollback()``, upon which a
-new transaction is to be begun immediately. This may seem to imply
-that the SQLite driver would in theory allow only a single filehandle on a
-particular database file at any time; however, there are several
-factors both within SQLite itself as well as within the pysqlite driver
-which loosen this restriction significantly.
-
-However, no matter what locking modes are used, SQLite will still always
-lock the database file once a transaction is started and DML (e.g. INSERT,
-UPDATE, DELETE) has at least been emitted, and this will block
-other transactions at least at the point that they also attempt to emit DML.
-By default, the length of time on this block is very short before it times out
-with an error.
-
-This behavior becomes more critical when used in conjunction with the
-SQLAlchemy ORM. SQLAlchemy's :class:`.Session` object by default runs
-within a transaction, and with its autoflush model, may emit DML preceding
-any SELECT statement. This may lead to a SQLite database that locks
-more quickly than is expected. The locking mode of SQLite and the pysqlite
-driver can be manipulated to some degree, however it should be noted that
-achieving a high degree of write-concurrency with SQLite is a losing battle.
-
-For more information on SQLite's lack of write concurrency by design, please
-see
-`Situations Where Another RDBMS May Work Better - High Concurrency
-<https://www.sqlite.org/whentouse.html>`_ near the bottom of the page.
-
-The following subsections introduce areas that are impacted by SQLite's
-file-based architecture and additionally will usually require workarounds to
-work when using the pysqlite driver.
+.. _sqlite_transactions:
+
+Transactions with SQLite and the sqlite3 driver
+-----------------------------------------------
+
+As a file-based database, SQLite's approach to transactions differs from
+traditional databases in many ways. Additionally, the ``sqlite3`` driver
+standard with Python (as well as the async version ``aiosqlite`` which builds
+on top of it) has several quirks, workarounds, and API features in the
+area of transaction control, all of which generally need to be addressed when
+constructing a SQLAlchemy application that uses SQLite.
+
+Legacy Transaction Mode with the sqlite3 driver
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+The most important aspect of transaction handling with the sqlite3 driver is
+that it defaults (which will continue through Python 3.15 before being
+removed in Python 3.16) to legacy transactional behavior which does
+not strictly follow :pep:`249`. The way in which the driver diverges from the
+PEP is that it does not "begin" a transaction automatically as dictated by
+:pep:`249` except in the case of DML statements, e.g. INSERT, UPDATE, and
+DELETE. Normally, :pep:`249` dictates that a BEGIN must be emitted upon
+the first SQL statement of any kind, so that all subsequent operations will
+be established within a transaction until ``connection.commit()`` has been
+called. The ``sqlite3`` driver, in an effort to be easier to use in
+highly concurrent environments, skips this step for DQL (e.g. SELECT) statements,
+and also skips it for DDL (e.g. CREATE TABLE etc.) statements for more legacy
+reasons. Statements such as SAVEPOINT are also skipped.
+
+In modern versions of the ``sqlite3`` driver as of Python 3.12, this legacy
+mode of operation is referred to as
+`"legacy transaction control" <https://docs.python.org/3/library/sqlite3.html#sqlite3-transaction-control-isolation-level>`_, and is in
+effect by default due to the ``Connection.autocommit`` parameter being set to
+the constant ``sqlite3.LEGACY_TRANSACTION_CONTROL``. Prior to Python 3.12,
+the ``Connection.autocommit`` attribute did not exist.
+
+The implications of legacy transaction mode include:
+
+* **Incorrect support for transactional DDL** - statements like CREATE TABLE, ALTER TABLE,
+ CREATE INDEX etc. will not automatically BEGIN a transaction if one were not
+ started already, leading to the changes by each statement being
+ "autocommitted" immediately unless BEGIN were otherwise emitted first. Very
+ old (pre Python 3.6) versions of SQLite would also force a COMMIT for these
+ operations even if a transaction were present, however this is no longer the
+ case.
+* **SERIALIZABLE behavior not fully functional** - SQLite's transaction isolation
+ behavior is normally consistent with SERIALIZABLE isolation, as it is a file-
+ based system that locks the database file entirely for write operations,
+ preventing COMMIT until all reader transactions (and associated file locks)
+ have completed. However, sqlite3's legacy transaction mode fails to emit BEGIN for SELECT
+ statements, which causes these SELECT statements to no longer be "repeatable",
+ failing one of the consistency guarantees of SERIALIZABLE.
+* **Incorrect behavior for SAVEPOINT** - as the SAVEPOINT statement does not
+ imply a BEGIN, a new SAVEPOINT emitted before a BEGIN will function on its
+ own but fails to participate in the enclosing transaction, meaning a ROLLBACK
+ of the transaction will not rollback elements that were part of a released
+ savepoint.
+
+Legacy transaction mode first existed in order to faciliate working around
+SQLite's file locks. Because SQLite relies upon whole-file locks, it is easy to
+get "database is locked" errors, particularly when newer features like "write
+ahead logging" are disabled. This is a key reason why ``sqlite3``'s legacy
+transaction mode is still the default mode of operation; disabling it will
+produce behavior that is more susceptible to locked database errors. However
+note that **legacy transaction mode will no longer be the default** in a future
+Python version (3.16 as of this writing).
+
+.. _sqlite_enabling_transactions:
+
+Enabling Non-Legacy SQLite Transactional Modes with the sqlite3 or aiosqlite driver
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Current SQLAlchemy support allows either for setting the
+``.Connection.autocommit`` attribute, most directly by using a
+:func:`._sa.create_engine` parameter, or if on an older version of Python where
+the attribute is not available, using event hooks to control the behavior of
+BEGIN.
+
+* **Enabling modern sqlite3 transaction control via the autocommit connect parameter** (Python 3.12 and above)
+
+ To use SQLite in the mode described at `Transaction control via the autocommit attribute <https://docs.python.org/3/library/sqlite3.html#transaction-control-via-the-autocommit-attribute>`_,
+ the most straightforward approach is to set the attribute to its recommended value
+ of ``False`` at the connect level using :paramref:`_sa.create_engine.connect_args``::
+
+ from sqlalchemy import create_engine
+
+ engine = create_engine(
+ "sqlite:///myfile.db", connect_args={"autocommit": False}
+ )
+
+ This parameter is also passed through when using the aiosqlite driver::
+
+ from sqlalchemy.ext.asyncio import create_async_engine
+
+ engine = create_async_engine(
+ "sqlite+aiosqlite:///myfile.db", connect_args={"autocommit": False}
+ )
+
+ The parameter can also be set at the attribute level using the :meth:`.PoolEvents.connect`
+ event hook, however this will only work for sqlite3, as aiosqlite does not yet expose this
+ attribute on its ``Connection`` object::
+
+ from sqlalchemy import create_engine, event
+
+ engine = create_engine("sqlite:///myfile.db")
+
+
+ @event.listens_for(engine, "connect")
+ def do_connect(dbapi_connection, connection_record):
+ # enable autocommit=False mode
+ dbapi_connection.autocommit = False
+
+* **Using SQLAlchemy to emit BEGIN in lieu of SQLite's transaction control** (all Python versions, sqlite3 and aiosqlite)
+
+ For older versions of ``sqlite3`` or for cross-compatiblity with older and
+ newer versions, SQLAlchemy can also take over the job of transaction control.
+ This is achieved by using the :meth:`.ConnectionEvents.begin` hook
+ to emit the "BEGIN" command directly, while also disabling SQLite's control
+ of this command using the :meth:`.PoolEvents.connect` event hook to set the
+ ``Connection.isolation_level`` attribute to ``None``::
+
+
+ from sqlalchemy import create_engine, event
+
+ engine = create_engine("sqlite:///myfile.db")
+
+
+ @event.listens_for(engine, "connect")
+ def do_connect(dbapi_connection, connection_record):
+ # disable sqlite3's emitting of the BEGIN statement entirely.
+ dbapi_connection.isolation_level = None
+
+
+ @event.listens_for(engine, "begin")
+ def do_begin(conn):
+ # emit our own BEGIN. sqlite3 still emits COMMIT/ROLLBACK correctly
+ conn.exec_driver_sql("BEGIN")
+
+ When using the asyncio variant ``aiosqlite``, refer to ``engine.sync_engine``
+ as in the example below::
+
+ from sqlalchemy import create_engine, event
+ from sqlalchemy.ext.asyncio import create_async_engine
+
+ engine = create_async_engine("sqlite+aiosqlite:///myfile.db")
+
+
+ @event.listens_for(engine.sync_engine, "connect")
+ def do_connect(dbapi_connection, connection_record):
+ # disable aiosqlite's emitting of the BEGIN statement entirely.
+ dbapi_connection.isolation_level = None
+
+
+ @event.listens_for(engine.sync_engine, "begin")
+ def do_begin(conn):
+ # emit our own BEGIN. aiosqlite still emits COMMIT/ROLLBACK correctly
+ conn.exec_driver_sql("BEGIN")
.. _sqlite_isolation_level:
-Transaction Isolation Level / Autocommit
-----------------------------------------
-
-SQLite supports "transaction isolation" in a non-standard way, along two
-axes. One is that of the
-`PRAGMA read_uncommitted <https://www.sqlite.org/pragma.html#pragma_read_uncommitted>`_
-instruction. This setting can essentially switch SQLite between its
-default mode of ``SERIALIZABLE`` isolation, and a "dirty read" isolation
-mode normally referred to as ``READ UNCOMMITTED``.
-
-SQLAlchemy ties into this PRAGMA statement using the
-:paramref:`_sa.create_engine.isolation_level` parameter of
-:func:`_sa.create_engine`.
-Valid values for this parameter when used with SQLite are ``"SERIALIZABLE"``
-and ``"READ UNCOMMITTED"`` corresponding to a value of 0 and 1, respectively.
-SQLite defaults to ``SERIALIZABLE``, however its behavior is impacted by
-the pysqlite driver's default behavior.
-
-When using the pysqlite driver, the ``"AUTOCOMMIT"`` isolation level is also
-available, which will alter the pysqlite connection using the ``.isolation_level``
-attribute on the DBAPI connection and set it to None for the duration
-of the setting.
-
-The other axis along which SQLite's transactional locking is impacted is
-via the nature of the ``BEGIN`` statement used. The three varieties
-are "deferred", "immediate", and "exclusive", as described at
-`BEGIN TRANSACTION <https://sqlite.org/lang_transaction.html>`_. A straight
-``BEGIN`` statement uses the "deferred" mode, where the database file is
-not locked until the first read or write operation, and read access remains
-open to other transactions until the first write operation. But again,
-it is critical to note that the pysqlite driver interferes with this behavior
-by *not even emitting BEGIN* until the first write operation.
+Using SQLAlchemy's Driver Level AUTOCOMMIT Feature with SQLite
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
-.. warning::
+SQLAlchemy has a comprehensive database isolation feature with optional
+autocommit support that is introduced in the section :ref:`dbapi_autocommit`.
- SQLite's transactional scope is impacted by unresolved
- issues in the pysqlite driver, which defers BEGIN statements to a greater
- degree than is often feasible. See the section :ref:`pysqlite_serializable`
- or :ref:`aiosqlite_serializable` for techniques to work around this behavior.
+For the ``sqlite3`` and ``aiosqlite`` drivers, SQLAlchemy only includes
+built-in support for "AUTOCOMMIT". Note that this mode is currently incompatible
+with the non-legacy isolation mode hooks documented in the previous
+section at :ref:`sqlite_enabling_transactions`.
-.. seealso::
+To use the ``sqlite3`` driver with SQLAlchemy driver-level autocommit,
+create an engine setting the :paramref:`_sa.create_engine.isolation_level`
+parameter to "AUTOCOMMIT"::
+
+ eng = create_engine("sqlite:///myfile.db", isolation_level="AUTOCOMMIT")
+
+When using the above mode, any event hooks that set the sqlite3 ``Connection.autocommit``
+parameter away from its default of ``sqlite3.LEGACY_TRANSACTION_CONTROL``
+as well as hooks that emit ``BEGIN`` should be disabled.
+
+Additional Reading for SQLite / sqlite3 transaction control
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+Links with important information on SQLite, the sqlite3 driver,
+as well as long historical conversations on how things got to their current state:
+
+* `Isolation in SQLite <https://www.sqlite.org/isolation.html>`_ - on the SQLite website
+* `Transaction control <https://docs.python.org/3/library/sqlite3.html#transaction-control>`_ - describes the sqlite3 autocommit attribute as well
+ as the legacy isolation_level attribute.
+* `sqlite3 SELECT does not BEGIN a transaction, but should according to spec <https://github.com/python/cpython/issues/54133>`_ - imported Python standard library issue on github
+* `sqlite3 module breaks transactions and potentially corrupts data <https://github.com/python/cpython/issues/54949>`_ - imported Python standard library issue on github
- :ref:`dbapi_autocommit`
INSERT/UPDATE/DELETE...RETURNING
---------------------------------
.. versionadded:: 2.0 Added support for SQLite RETURNING
-SAVEPOINT Support
-----------------------------
-
-SQLite supports SAVEPOINTs, which only function once a transaction is
-begun. SQLAlchemy's SAVEPOINT support is available using the
-:meth:`_engine.Connection.begin_nested` method at the Core level, and
-:meth:`.Session.begin_nested` at the ORM level. However, SAVEPOINTs
-won't work at all with pysqlite unless workarounds are taken.
-
-.. warning::
-
- SQLite's SAVEPOINT feature is impacted by unresolved
- issues in the pysqlite and aiosqlite drivers, which defer BEGIN statements
- to a greater degree than is often feasible. See the sections
- :ref:`pysqlite_serializable` and :ref:`aiosqlite_serializable`
- for techniques to work around this behavior.
-
-Transactional DDL
-----------------------------
-
-The SQLite database supports transactional :term:`DDL` as well.
-In this case, the pysqlite driver is not only failing to start transactions,
-it also is ending any existing transaction when DDL is detected, so again,
-workarounds are required.
-
-.. warning::
-
- SQLite's transactional DDL is impacted by unresolved issues
- in the pysqlite driver, which fails to emit BEGIN and additionally
- forces a COMMIT to cancel any transaction when DDL is encountered.
- See the section :ref:`pysqlite_serializable`
- for techniques to work around this behavior.
.. _sqlite_foreign_keys:
Serializable isolation / Savepoints / Transactional DDL
-------------------------------------------------------
-In the section :ref:`sqlite_concurrency`, we refer to the pysqlite
-driver's assortment of issues that prevent several features of SQLite
-from working correctly. The pysqlite DBAPI driver has several
-long-standing bugs which impact the correctness of its transactional
-behavior. In its default mode of operation, SQLite features such as
-SERIALIZABLE isolation, transactional DDL, and SAVEPOINT support are
-non-functional, and in order to use these features, workarounds must
-be taken.
+A newly revised version of this important section is now available
+at the top level of the SQLAlchemy SQLite documentation, in the section
+:ref:`sqlite_transactions`.
-The issue is essentially that the driver attempts to second-guess the user's
-intent, failing to start transactions and sometimes ending them prematurely, in
-an effort to minimize the SQLite databases's file locking behavior, even
-though SQLite itself uses "shared" locks for read-only activities.
-
-SQLAlchemy chooses to not alter this behavior by default, as it is the
-long-expected behavior of the pysqlite driver; if and when the pysqlite
-driver attempts to repair these issues, that will be more of a driver towards
-defaults for SQLAlchemy.
-
-The good news is that with a few events, we can implement transactional
-support fully, by disabling pysqlite's feature entirely and emitting BEGIN
-ourselves. This is achieved using two event listeners::
-
- from sqlalchemy import create_engine, event
-
- engine = create_engine("sqlite:///myfile.db")
-
-
- @event.listens_for(engine, "connect")
- def do_connect(dbapi_connection, connection_record):
- # disable pysqlite's emitting of the BEGIN statement entirely.
- # also stops it from emitting COMMIT before any DDL.
- dbapi_connection.isolation_level = None
-
-
- @event.listens_for(engine, "begin")
- def do_begin(conn):
- # emit our own BEGIN
- conn.exec_driver_sql("BEGIN")
-
-.. warning:: When using the above recipe, it is advised to not use the
- :paramref:`.Connection.execution_options.isolation_level` setting on
- :class:`_engine.Connection` and :func:`_sa.create_engine`
- with the SQLite driver,
- as this function necessarily will also alter the ".isolation_level" setting.
-
-
-Above, we intercept a new pysqlite connection and disable any transactional
-integration. Then, at the point at which SQLAlchemy knows that transaction
-scope is to begin, we emit ``"BEGIN"`` ourselves.
-
-When we take control of ``"BEGIN"``, we can also control directly SQLite's
-locking modes, introduced at
-`BEGIN TRANSACTION <https://sqlite.org/lang_transaction.html>`_,
-by adding the desired locking mode to our ``"BEGIN"``::
-
- @event.listens_for(engine, "begin")
- def do_begin(conn):
- conn.exec_driver_sql("BEGIN EXCLUSIVE")
-
-.. seealso::
-
- `BEGIN TRANSACTION <https://sqlite.org/lang_transaction.html>`_ -
- on the SQLite site
-
- `sqlite3 SELECT does not BEGIN a transaction <https://bugs.python.org/issue9924>`_ -
- on the Python bug tracker
-
- `sqlite3 module breaks transactions and potentially corrupts data <https://bugs.python.org/issue10740>`_ -
- on the Python bug tracker
.. _pysqlite_udfs:
projects / thirdparty / sqlalchemy / sqlalchemy.git / commitdiff
