rewrite SQLite in-memory database docs for concurrency

Rewrote the "Using a Memory Database in Multiple Threads" section
in the pysqlite dialect docs to lead with the fundamental constraint
that a :memory: database is scoped to a single connection, and is
not suitable for concurrent use without shared cache or full
serialization.

Added a new "Using a Shared-Cache Memory Database" subsection
recommending the file::memory:?cache=shared&uri=true URI approach,
which gives each checkout its own DBAPI connection with independent
transaction state while sharing one in-memory database.  Documented
process-global scoping and named databases for isolation.

Demoted the StaticPool approach to a secondary subsection with a
prominent warning about its single-connection limitation and silent
data loss under concurrent sessions.

Added a corresponding "Using a Memory Database with Multiple
Coroutines" section to the aiosqlite dialect docs, cross-referencing
the pysqlite shared-cache documentation.

References: #13428, #6987
Change-Id: Ic07c5a2a564e8eca19e596267f61051a2aaa0258

diff --git a/lib/sqlalchemy/dialects/sqlite/aiosqlite.py b/lib/sqlalchemy/dialects/sqlite/aiosqlite.py
index 5be6aa3a6384e07e1a79b6aea2adf03558f58da9..7b9657bbeee78a47baf178af19a8565649bc98aa 100644 (file)
--- a/lib/sqlalchemy/dialects/sqlite/aiosqlite.py
+++ b/lib/sqlalchemy/dialects/sqlite/aiosqlite.py
@@ -76,6 +76,36 @@ based on the kind of SQLite database that's requested:
     may be used by specifying it via the
     :paramref:`_sa.create_engine.poolclass` parameter.
 
+.. _aiosqlite_memory:
+
+Using a Memory Database with Multiple Coroutines
+-------------------------------------------------
+
+The default :class:`.StaticPool` used for ``:memory:`` databases forces all
+coroutines to share a single DBAPI connection.  Because SQLite maintains only
+one transaction state per connection, concurrent coroutines can interfere
+with each other — a ``ROLLBACK`` in one coroutine will also discard
+uncommitted work from any other coroutine using the same engine.
+
+For async workloads where multiple :class:`.AsyncSession` or
+:class:`.AsyncConnection` objects may be active simultaneously, use SQLite's
+shared-cache URI mode instead.  This gives each checkout its own DBAPI
+connection with independent transaction state while still sharing one
+in-memory database::
+
+    engine = create_async_engine(
+        "sqlite+aiosqlite:///file::memory:?cache=shared&uri=true"
+    )
+
+Because this URL form is treated as a file-based database by the dialect,
+:class:`.AsyncAdaptedQueuePool` is used automatically and no additional
+configuration is needed.
+
+See the pysqlite documentation at
+:ref:`pysqlite_uri_shared_cache` for full details on shared-cache memory
+databases, including how to use named databases to maintain multiple
+independent in-memory databases within the same process.
+
 """  # noqa
 
 from __future__ import annotations

diff --git a/lib/sqlalchemy/dialects/sqlite/pysqlite.py b/lib/sqlalchemy/dialects/sqlite/pysqlite.py
index 01ec141ad2cf9101d43badc8f827ceb510701d79..008919ad4a0b1bca95ff18b1ad98b74e66fda241 100644 (file)
--- a/lib/sqlalchemy/dialects/sqlite/pysqlite.py
+++ b/lib/sqlalchemy/dialects/sqlite/pysqlite.py
@@ -207,8 +207,8 @@ processing.
 
 .. _pysqlite_threading_pooling:
 
-Threading/Pooling Behavior
----------------------------
+Concurrency/Threading/Pooling Behavior
+--------------------------------------
 
 The ``sqlite3`` DBAPI by default prohibits the use of a particular connection
 in a thread which is not the one in which it was created.  As SQLite has
@@ -259,15 +259,84 @@ connection reuse implemented by :class:`.QueuePool`.  However, it still
 may be beneficial to use this class if the application is experiencing
 issues with files being locked.
 
-Using a Memory Database in Multiple Threads
-^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+Using a Memory Database in Multiple Threads or Coroutines
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+A ``:memory:`` SQLite database exists only within the scope of a single
+DBAPI connection.  It is not possible for two separate ``sqlite3``
+connection objects to access the same ``:memory:`` database unless SQLite's
+shared-cache feature is enabled.  Without shared cache, each connection
+creates its own independent in-memory database.
+
+This means a ``:memory:`` database is **not suitable** for use with
+multiple concurrent threads or coroutines unless either:
+
+* All workers are fully serialized (mutexed) against each other
+  such that only one is using the database at a time, or
+* SQLite's shared-cache URI feature is used to allow multiple
+  independent connections to access the same in-memory database.
+
+The same considerations apply when using the :ref:`aiosqlite <aiosqlite>`
+dialect, which wraps ``pysqlite`` connections in an async interface —
+without shared cache, each connection still creates its own independent
+in-memory database.
+
+The recommended approach for multithreaded or async in-memory
+use is the shared-cache URI feature, described below.
+
+.. _pysqlite_uri_shared_cache:
+
+Using a Shared-Cache Memory Database
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+SQLite's
+`URI shared-cache <https://www.sqlite.org/sharedcache.html>`_
+mode allows multiple independent DBAPI connections, each with
+their own transaction state, to access the same in-memory
+database.  This is enabled by using a ``file:`` URI with
+``cache=shared`` and passing ``uri=true`` in the query
+string::
+
+    engine = create_engine(
+        "sqlite:///file::memory:?cache=shared&uri=true"
+    )
+
+For async use with aiosqlite, use
+:func:`_asyncio.create_async_engine`::
+
+    engine = create_async_engine(
+        "sqlite+aiosqlite:///file::memory:?cache=shared&uri=true"
+    )
+
+Because this URL form is treated as a file-based database by the
+dialect, :class:`.QueuePool` is used automatically and
+``check_same_thread`` defaults to ``False``, so no additional pool
+or connect_args configuration is needed.  Each checkout from the
+pool is a distinct DBAPI connection with its own transaction state,
+and the in-memory database persists as long as at least one
+connection remains open.
+
+The shared-cache database is scoped by the filename component of
+the URI.  ``file::memory:`` (empty name) is process-global — all
+engines in the process that use this URI share the same database.
+To maintain multiple independent in-memory databases within the
+same process, supply a distinct name for each::
+
+    engine_a = create_engine(
+        "sqlite:///file:db_a?mode=memory&cache=shared&uri=true"
+    )
+    engine_b = create_engine(
+        "sqlite:///file:db_b?mode=memory&cache=shared&uri=true"
+    )
+
+Using StaticPool for Single-Connection Memory Databases
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
 
-To use a ``:memory:`` database in a multithreaded scenario, the same
-connection object must be shared among threads, since the database exists
-only within the scope of that connection.   The
-:class:`.StaticPool` implementation will maintain a single connection
-globally, and the ``check_same_thread`` flag can be passed to Pysqlite
-as ``False``::
+An older approach for sharing a ``:memory:`` database among threads
+is to force all checkouts to return the same DBAPI connection using
+:class:`.StaticPool`.  **This approach does not support any form of
+concurrency** and is only useful when all access to the engine is
+fully serialized, such as in single-threaded test suites::
 
     from sqlalchemy.pool import StaticPool
 
@@ -277,8 +346,18 @@ as ``False``::
         poolclass=StaticPool,
     )
 
-Note that using a ``:memory:`` database in multiple threads requires a recent
-version of SQLite.
+.. warning::
+
+    Because :class:`.StaticPool` maintains a single DBAPI connection,
+    all :class:`.Session` or :class:`.Connection` objects that use
+    this engine share that one underlying connection and its single
+    SQLite transaction state.  A ``ROLLBACK`` issued by one session
+    (e.g. during error handling) will also roll back uncommitted work
+    from any other session, and concurrent ``COMMIT`` / ``ROLLBACK``
+    calls can interfere with each other unpredictably.  This approach
+    is only appropriate when access to the engine is fully serialized,
+    such as in single-threaded test suites.  For concurrent workloads,
+    use the shared-cache URI approach described above.
 
 Using Temporary Tables with SQLite
 ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^