From: Mike Bayer Date: Sun, 12 Jul 2026 22:11:21 +0000 (-0400) Subject: rewrite SQLite in-memory database docs for concurrency X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=702a7f11dd8a9d3dae30048aa3d2fc0197330d88;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git rewrite SQLite in-memory database docs for concurrency Rewrote the "Using a Memory Database in Multiple Threads" section in the pysqlite dialect docs to lead with the fundamental constraint that a :memory: database is scoped to a single connection, and is not suitable for concurrent use without shared cache or full serialization. Added a new "Using a Shared-Cache Memory Database" subsection recommending the file::memory:?cache=shared&uri=true URI approach, which gives each checkout its own DBAPI connection with independent transaction state while sharing one in-memory database. Documented process-global scoping and named databases for isolation. Demoted the StaticPool approach to a secondary subsection with a prominent warning about its single-connection limitation and silent data loss under concurrent sessions. Added a corresponding "Using a Memory Database with Multiple Coroutines" section to the aiosqlite dialect docs, cross-referencing the pysqlite shared-cache documentation. References: #13428, #6987 Change-Id: Ic07c5a2a564e8eca19e596267f61051a2aaa0258 --- diff --git a/lib/sqlalchemy/dialects/sqlite/aiosqlite.py b/lib/sqlalchemy/dialects/sqlite/aiosqlite.py index 5be6aa3a63..7b9657bbee 100644 --- a/lib/sqlalchemy/dialects/sqlite/aiosqlite.py +++ b/lib/sqlalchemy/dialects/sqlite/aiosqlite.py @@ -76,6 +76,36 @@ based on the kind of SQLite database that's requested: may be used by specifying it via the :paramref:`_sa.create_engine.poolclass` parameter. +.. _aiosqlite_memory: + +Using a Memory Database with Multiple Coroutines +------------------------------------------------- + +The default :class:`.StaticPool` used for ``:memory:`` databases forces all +coroutines to share a single DBAPI connection. Because SQLite maintains only +one transaction state per connection, concurrent coroutines can interfere +with each other — a ``ROLLBACK`` in one coroutine will also discard +uncommitted work from any other coroutine using the same engine. + +For async workloads where multiple :class:`.AsyncSession` or +:class:`.AsyncConnection` objects may be active simultaneously, use SQLite's +shared-cache URI mode instead. This gives each checkout its own DBAPI +connection with independent transaction state while still sharing one +in-memory database:: + + engine = create_async_engine( + "sqlite+aiosqlite:///file::memory:?cache=shared&uri=true" + ) + +Because this URL form is treated as a file-based database by the dialect, +:class:`.AsyncAdaptedQueuePool` is used automatically and no additional +configuration is needed. + +See the pysqlite documentation at +:ref:`pysqlite_uri_shared_cache` for full details on shared-cache memory +databases, including how to use named databases to maintain multiple +independent in-memory databases within the same process. + """ # noqa from __future__ import annotations diff --git a/lib/sqlalchemy/dialects/sqlite/pysqlite.py b/lib/sqlalchemy/dialects/sqlite/pysqlite.py index 01ec141ad2..008919ad4a 100644 --- a/lib/sqlalchemy/dialects/sqlite/pysqlite.py +++ b/lib/sqlalchemy/dialects/sqlite/pysqlite.py @@ -207,8 +207,8 @@ processing. .. _pysqlite_threading_pooling: -Threading/Pooling Behavior ---------------------------- +Concurrency/Threading/Pooling Behavior +-------------------------------------- The ``sqlite3`` DBAPI by default prohibits the use of a particular connection in a thread which is not the one in which it was created. As SQLite has @@ -259,15 +259,84 @@ connection reuse implemented by :class:`.QueuePool`. However, it still may be beneficial to use this class if the application is experiencing issues with files being locked. -Using a Memory Database in Multiple Threads -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +Using a Memory Database in Multiple Threads or Coroutines +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ + +A ``:memory:`` SQLite database exists only within the scope of a single +DBAPI connection. It is not possible for two separate ``sqlite3`` +connection objects to access the same ``:memory:`` database unless SQLite's +shared-cache feature is enabled. Without shared cache, each connection +creates its own independent in-memory database. + +This means a ``:memory:`` database is **not suitable** for use with +multiple concurrent threads or coroutines unless either: + +* All workers are fully serialized (mutexed) against each other + such that only one is using the database at a time, or +* SQLite's shared-cache URI feature is used to allow multiple + independent connections to access the same in-memory database. + +The same considerations apply when using the :ref:`aiosqlite ` +dialect, which wraps ``pysqlite`` connections in an async interface — +without shared cache, each connection still creates its own independent +in-memory database. + +The recommended approach for multithreaded or async in-memory +use is the shared-cache URI feature, described below. + +.. _pysqlite_uri_shared_cache: + +Using a Shared-Cache Memory Database +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ + +SQLite's +`URI shared-cache `_ +mode allows multiple independent DBAPI connections, each with +their own transaction state, to access the same in-memory +database. This is enabled by using a ``file:`` URI with +``cache=shared`` and passing ``uri=true`` in the query +string:: + + engine = create_engine( + "sqlite:///file::memory:?cache=shared&uri=true" + ) + +For async use with aiosqlite, use +:func:`_asyncio.create_async_engine`:: + + engine = create_async_engine( + "sqlite+aiosqlite:///file::memory:?cache=shared&uri=true" + ) + +Because this URL form is treated as a file-based database by the +dialect, :class:`.QueuePool` is used automatically and +``check_same_thread`` defaults to ``False``, so no additional pool +or connect_args configuration is needed. Each checkout from the +pool is a distinct DBAPI connection with its own transaction state, +and the in-memory database persists as long as at least one +connection remains open. + +The shared-cache database is scoped by the filename component of +the URI. ``file::memory:`` (empty name) is process-global — all +engines in the process that use this URI share the same database. +To maintain multiple independent in-memory databases within the +same process, supply a distinct name for each:: + + engine_a = create_engine( + "sqlite:///file:db_a?mode=memory&cache=shared&uri=true" + ) + engine_b = create_engine( + "sqlite:///file:db_b?mode=memory&cache=shared&uri=true" + ) + +Using StaticPool for Single-Connection Memory Databases +~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ -To use a ``:memory:`` database in a multithreaded scenario, the same -connection object must be shared among threads, since the database exists -only within the scope of that connection. The -:class:`.StaticPool` implementation will maintain a single connection -globally, and the ``check_same_thread`` flag can be passed to Pysqlite -as ``False``:: +An older approach for sharing a ``:memory:`` database among threads +is to force all checkouts to return the same DBAPI connection using +:class:`.StaticPool`. **This approach does not support any form of +concurrency** and is only useful when all access to the engine is +fully serialized, such as in single-threaded test suites:: from sqlalchemy.pool import StaticPool @@ -277,8 +346,18 @@ as ``False``:: poolclass=StaticPool, ) -Note that using a ``:memory:`` database in multiple threads requires a recent -version of SQLite. +.. warning:: + + Because :class:`.StaticPool` maintains a single DBAPI connection, + all :class:`.Session` or :class:`.Connection` objects that use + this engine share that one underlying connection and its single + SQLite transaction state. A ``ROLLBACK`` issued by one session + (e.g. during error handling) will also roll back uncommitted work + from any other session, and concurrent ``COMMIT`` / ``ROLLBACK`` + calls can interfere with each other unpredictably. This approach + is only appropriate when access to the engine is fully serialized, + such as in single-threaded test suites. For concurrent workloads, + use the shared-cache URI approach described above. Using Temporary Tables with SQLite ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^