--- /dev/null
+.. change::
+ :tags: deprecated, sqlite
+ :tickets: 13433
+
+ Deprecated the selection of a single-connection pool class, i.e.
+ :class:`.SingletonThreadPool` for pysqlite or :class:`.StaticPool` for
+ aiosqlite, based on the presence of the ``mode=memory`` query string
+ argument in a SQLite URL. Pool selection for SQLite is intended to be
+ based on the database name alone, where only ``:memory:`` or an empty
+ database name indicate a memory database; interpreting the query string
+ additionally requires that assumptions be made regarding whether or not
+ the resulting database can be shared among multiple connections. In a
+ future release, such URLs will make use of :class:`.QueuePool` or
+ :class:`.AsyncAdaptedQueuePool` as would any other URL. This notably
+ includes the shared cache form
+ ``sqlite:///file:mydb?mode=memory&cache=shared&uri=true``, for which a
+ queue pool is in fact the appropriate class, as a shared cache database
+ supports multiple concurrent connections, whereas a single-connection
+ pool causes such connections to share one transaction state.
+ Applications that rely upon the present behavior should indicate the
+ intended pool using the :paramref:`_sa.create_engine.poolclass`
+ parameter. Pull request courtesy Itachi-0xAI.
+
+.. change::
+ :tags: bug, sqlite
+ :tickets: 13433
+
+ Added a warning for query string arguments that are passed to a SQLite
+ URL without the ``uri=true`` argument also being present, and which are
+ not accepted by the ``sqlite3`` driver itself. SQLite URI arguments
+ such as ``mode`` or ``cache`` take effect only when URI mode is in use;
+ without it they were previously discarded silently, so that a URL such
+ as ``sqlite:///file:mydb?mode=memory`` would connect to a file on disk
+ named ``file:mydb``. Arguments intended for the driver itself may be
+ passed using the :paramref:`_sa.create_engine.connect_args` parameter.
+
+.. change::
+ :tags: bug, documentation, sqlite
+ :tickets: 13433
+
+ Corrected the SQLite documentation regarding shared cache memory
+ databases, which incorrectly indicated that the named form
+ ``sqlite:///file:mydb?mode=memory&cache=shared&uri=true`` makes use of
+ :class:`.QueuePool`; a single-connection pool is used for this form.
+ Documentation has also been added noting that a shared cache database
+ exists only for as long as at least one connection to it remains open,
+ so that ordinary pool operations such as
+ :meth:`_engine.Engine.dispose` or use of
+ :paramref:`_sa.create_engine.pool_recycle` will discard its contents.
that was in progress is now in an invalid state, and must be explicitly rolled
back in order to remove it from the :class:`_engine.Connection`.
+.. _error_sqmp:
+
+Selection of the <pool> pool class based on the 'mode=memory' query string argument is deprecated
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The SQLite dialects select a connection pool class automatically, based on
+whether the URL refers to a file database or to a memory database. A memory
+database is indicated by the database name ``:memory:``, or by an empty
+database name, and receives a single-connection pool class, being
+:class:`.SingletonThreadPool` for pysqlite or :class:`.StaticPool` for
+aiosqlite. All other database names receive :class:`.QueuePool` or, for
+aiosqlite, :class:`.AsyncAdaptedQueuePool`.
+
+Historically, a URL that includes ``mode=memory`` within the query string
+also received a single-connection pool class. This behavior is deprecated
+as of SQLAlchemy 2.1 and will be removed in a future release, as it requires
+that the dialect make assumptions regarding whether or not the resulting
+database can be shared among multiple connections, based on query string
+arguments whose meaning is determined by SQLite rather than by SQLAlchemy.
+
+The behavior is notably incorrect for the shared cache form::
+
+ # a shared cache database supports multiple concurrent connections,
+ # however a single-connection pool is presently selected
+ engine = create_engine("sqlite:///file:mydb?mode=memory&cache=shared&uri=true")
+
+For this URL, :class:`.QueuePool` is the appropriate pool class, as each
+connection to a shared cache database has its own transaction state. A
+single-connection pool instead causes all :class:`.Session` or
+:class:`_engine.Connection` objects to share one transaction state, so that a
+``ROLLBACK`` emitted by one will discard uncommitted work belonging to
+another.
+
+To resolve the warning, indicate the intended pool class explicitly, which
+also disables automatic selection entirely::
+
+ from sqlalchemy.pool import QueuePool
+
+ engine = create_engine(
+ "sqlite:///file:mydb?mode=memory&cache=shared&uri=true",
+ poolclass=QueuePool,
+ )
+
+For a single-connection memory database, the plain ``:memory:`` form may be
+used instead, which continues to select a single-connection pool::
+
+ engine = create_engine("sqlite://")
+
+.. seealso::
+
+ :ref:`pysqlite_threading_pooling`
+
+ :ref:`pysqlite_uri_shared_cache`
+
+ :ref:`pysqlite_shared_cache_lifespan`
+
+.. _error_squa:
+
+Query string argument(s) <names> are not accepted by the pysqlite driver and are being ignored
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+SQLite accepts a range of options within the database name itself, using a
+`URI filename <https://www.sqlite.org/uri.html>`_ such as
+``file:mydb?mode=ro``. These options are interpreted by SQLite only when URI
+filenames are enabled, which for the ``sqlite3`` driver requires that
+``uri=true`` be passed to its ``connect()`` function.
+
+Within a SQLAlchemy URL, query string arguments are delivered to the
+``sqlite3`` driver, with those not recognized by the driver being appended to
+the SQLite URI filename. This latter step takes place only when ``uri=true``
+is also present; otherwise the database name is treated as an ordinary
+filename and the remaining arguments have no effect at all::
+
+ # 'mode' has no effect; connects to a file named "file:mydb"
+ engine = create_engine("sqlite:///file:mydb?mode=memory")
+
+ # 'mode' is passed to SQLite as part of the URI filename
+ engine = create_engine("sqlite:///file:mydb?mode=memory&uri=true")
+
+The warning indicates arguments that fall into the first case above. To
+resolve it, add ``uri=true`` to the URL if the arguments are intended for
+SQLite, or remove them if they are not. Arguments intended for the
+``sqlite3`` driver itself may alternatively be passed using the
+:paramref:`_sa.create_engine.connect_args` parameter.
+
+.. seealso::
+
+ :ref:`pysqlite_uri_connections`
+
.. _error_dbapi:
DBAPI Errors
may be used by specifying it via the
:paramref:`_sa.create_engine.poolclass` parameter.
+As with the pysqlite dialect, this selection is made based on the database
+name alone, and the ``mode=memory`` query string argument is deprecated as
+a means of influencing it; see :ref:`pysqlite_threading_pooling` for
+background.
+
.. _aiosqlite_memory:
Using a Memory Database with Multiple Coroutines
:class:`.AsyncAdaptedQueuePool` is used automatically and no additional
configuration is needed.
+Note that a shared-cache database is discarded once its last connection is
+closed, so that operations such as :meth:`_asyncio.AsyncEngine.dispose` or
+the use of :paramref:`_sa.create_engine.pool_recycle` will destroy its
+contents; see :ref:`pysqlite_shared_cache_lifespan` for background and for
+how to hold such a database open.
+
See the pysqlite documentation at
:ref:`pysqlite_uri_shared_cache` for full details on shared-cache memory
databases, including how to use named databases to maintain multiple
if cls._is_url_file_db(url):
return pool.AsyncAdaptedQueuePool
else:
+ cls._warn_memory_mode_pool_selection(
+ url, pool.StaticPool, pool.AsyncAdaptedQueuePool
+ )
return pool.StaticPool
def is_disconnect(
may be used by specifying it via the
:paramref:`_sa.create_engine.poolclass` parameter.
+This selection is made based on the database name alone. Where a
+particular pool class is desired, it should be stated explicitly using the
+:paramref:`_sa.create_engine.poolclass` parameter, in which case no
+selection takes place at all.
+
+.. deprecated:: 2.1
+
+ A URL that passes ``mode=memory`` in the query string is currently also
+ given a single-connection pool class. This behavior is deprecated and
+ will be removed in a future release, at which point such URLs will
+ receive :class:`.QueuePool` like any other. This affects URLs such as
+ ``sqlite:///file:mydb?mode=memory&cache=shared&uri=true``, for which
+ :class:`.QueuePool` is in fact the appropriate class, as a shared cache
+ database supports multiple concurrent connections; see
+ :ref:`pysqlite_uri_shared_cache`. Applications relying on the present
+ behavior should state the pool class explicitly.
+
Disabling Connection Pooling for File Databases
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
dialect, :class:`.QueuePool` is used automatically and
``check_same_thread`` defaults to ``False``, so no additional pool
or connect_args configuration is needed. Each checkout from the
-pool is a distinct DBAPI connection with its own transaction state,
-and the in-memory database persists as long as at least one
-connection remains open.
+pool is a distinct DBAPI connection with its own transaction state.
The shared-cache database is scoped by the filename component of
the URI. ``file::memory:`` (empty name) is process-global — all
engines in the process that use this URI share the same database.
To maintain multiple independent in-memory databases within the
-same process, supply a distinct name for each::
+same process, supply a distinct name for each. A named database of
+this kind requires ``mode=memory``, which presently causes a
+single-connection pool to be selected; :class:`.QueuePool` should
+therefore be requested explicitly::
+
+ from sqlalchemy.pool import QueuePool
engine_a = create_engine(
- "sqlite:///file:db_a?mode=memory&cache=shared&uri=true"
+ "sqlite:///file:db_a?mode=memory&cache=shared&uri=true",
+ poolclass=QueuePool,
)
engine_b = create_engine(
- "sqlite:///file:db_b?mode=memory&cache=shared&uri=true"
+ "sqlite:///file:db_b?mode=memory&cache=shared&uri=true",
+ poolclass=QueuePool,
)
+For :func:`_asyncio.create_async_engine`, use
+:class:`.AsyncAdaptedQueuePool` in the same way.
+
+.. deprecated:: 2.1
+
+ Selection of a single-connection pool class based on ``mode=memory``
+ is deprecated; a future release will use :class:`.QueuePool` for these
+ URLs, at which point stating
+ :paramref:`_sa.create_engine.poolclass` explicitly will no longer be
+ necessary. See :ref:`pysqlite_threading_pooling`.
+
+.. _pysqlite_shared_cache_lifespan:
+
+Lifespan of a Shared-Cache Memory Database
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+A shared-cache in-memory database exists only for as long as at least one
+connection to it remains open; when the last connection is closed, the
+database and all of its contents are discarded. A subsequent connection
+using the same URI then opens a new, empty database, which typically
+surfaces as ``no such table`` errors.
+
+:class:`.QueuePool` retains connections that have been returned to it, so
+in a default configuration the database will normally persist once the
+first connection has been established. This is a consequence of pool
+behavior rather than a guarantee, however, and the database will be
+discarded by ordinary pool operations including:
+
+* :meth:`_engine.Engine.dispose`, which closes all connections currently
+ in the pool
+* :paramref:`_sa.create_engine.pool_recycle`, as a recycled connection is
+ closed before its replacement is opened
+* connection invalidation, including that performed by
+ :paramref:`_sa.create_engine.pool_pre_ping`
+* use of :class:`.NullPool`, which closes each connection as it is
+ returned
+
+Where the database must survive independently of pool activity, hold a
+single connection open for as long as the database is needed::
+
+ engine = create_engine("sqlite:///file::memory:?cache=shared&uri=true")
+
+ # keep the database alive for the lifetime of the engine
+ keepalive = engine.connect()
+
+The same consideration applies to the :ref:`aiosqlite <aiosqlite>`
+dialect, using :meth:`_asyncio.AsyncEngine.connect`.
+
Using StaticPool for Single-Connection Memory Databases
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
else:
return False
+ @classmethod
+ def _warn_memory_mode_pool_selection(
+ cls,
+ url: URL,
+ current_pool: type[pool.Pool],
+ future_pool: type[pool.Pool],
+ ) -> None:
+ """Warn when the ``mode=memory`` query string argument is what
+ caused a single-connection pool class to be selected.
+
+ See :ticket:`13433`.
+
+ """
+ if url.query.get("mode", None) != "memory":
+ return
+
+ util.warn_deprecated(
+ "Selection of the %s pool class based on the 'mode=memory' "
+ "query string argument is deprecated; a future release will "
+ "use %s for this URL. Indicate the intended pool class "
+ "using the create_engine.poolclass parameter."
+ % (current_pool.__name__, future_pool.__name__),
+ "2.1",
+ code="sqmp",
+ )
+
@classmethod
def get_pool_class(cls, url: URL) -> type[pool.Pool]:
if cls._is_url_file_db(url):
return pool.QueuePool
else:
+ cls._warn_memory_mode_pool_selection(
+ url, pool.SingletonThreadPool, pool.QueuePool
+ )
return pool.SingletonThreadPool
def _get_server_version_info(
)
)
else:
+ # without uri=True, the SQLite URI query string is not in
+ # play at all, so anything left over here is silently
+ # discarded; warn rather than have it appear to take effect
+ ignored = sorted(
+ set(opts).difference(key for key, _ in pysqlite_args)
+ )
+ if ignored:
+ util.warn(
+ "Query string argument(s) %s are not accepted by the "
+ "pysqlite driver and are being ignored; SQLite URI "
+ "arguments require that 'uri=true' also be present "
+ "in the URL."
+ % (", ".join("'%s'" % key for key in ignored),),
+ code="squa",
+ )
+
filename = url.database or ":memory:"
if filename != ":memory:":
filename = os.path.abspath(filename)
deprecated_since: Optional[str] = None
"Indicates the version that started raising this deprecation warning"
+ _what_are_we = "warning"
+
class Base20DeprecationWarning(SADeprecationWarning):
"""Issued for usage of APIs specifically deprecated or legacy in
from sqlalchemy import text
from sqlalchemy import types as sqltypes
from sqlalchemy import UniqueConstraint
+from sqlalchemy.dialects.sqlite import aiosqlite as aiosqlite_dialect
from sqlalchemy.dialects.sqlite import base as sqlite
from sqlalchemy.dialects.sqlite import pysqlite as pysqlite_dialect
from sqlalchemy.engine.url import make_url
from sqlalchemy.testing import combinations
from sqlalchemy.testing import engines
from sqlalchemy.testing import eq_
+from sqlalchemy.testing import expect_deprecated
from sqlalchemy.testing import expect_raises
from sqlalchemy.testing import expect_warnings
from sqlalchemy.testing import fixtures
assert "méil" in result.keys()
assert "\u6e2c\u8a66" in result.keys()
- def test_pool_class(self):
- e = create_engine("sqlite+pysqlite://")
- assert e.pool.__class__ is pool.SingletonThreadPool
+ @combinations(
+ ("sqlite+pysqlite://", pool.SingletonThreadPool),
+ ("sqlite+pysqlite:///:memory:", pool.SingletonThreadPool),
+ # changed as of 2.0 #7490
+ ("sqlite+pysqlite:///foo.db", pool.QueuePool),
+ ("sqlite+pysqlite:///file:foo.db?uri=true", pool.QueuePool),
+ ("sqlite+pysqlite:///file:foo.db?mode=rwc&uri=true", pool.QueuePool),
+ (
+ "sqlite+pysqlite:///file::memory:?cache=shared&uri=true",
+ pool.QueuePool,
+ ),
+ argnames="url, expected",
+ )
+ def test_pool_class(self, url, expected):
+ e = create_engine(url)
+ assert e.pool.__class__ is expected
- e = create_engine("sqlite+pysqlite:///:memory:")
- assert e.pool.__class__ is pool.SingletonThreadPool
+ @combinations(
+ (
+ "sqlite+pysqlite:///file:foo.db?mode=memory&uri=true",
+ pysqlite_dialect.dialect,
+ "SingletonThreadPool",
+ "QueuePool",
+ ),
+ (
+ "sqlite+pysqlite:///file:foo.db?"
+ "mode=memory&cache=shared&uri=true",
+ pysqlite_dialect.dialect,
+ "SingletonThreadPool",
+ "QueuePool",
+ ),
+ (
+ "sqlite+aiosqlite:///file:foo.db?"
+ "mode=memory&cache=shared&uri=true",
+ aiosqlite_dialect.dialect,
+ "StaticPool",
+ "AsyncAdaptedQueuePool",
+ ),
+ argnames="url, dialect_cls, current_pool, future_pool",
+ )
+ def test_memory_mode_pool_deprecated(
+ self, url, dialect_cls, current_pool, future_pool
+ ):
+ """test #13433
+
+ the ``mode=memory`` query string argument selecting a
+ single-connection pool class is deprecated, including for a
+ shared cache database, where the queue pool is in fact the
+ appropriate class.
+
+ """
+
+ with expect_deprecated(
+ "Selection of the %s pool class based on the 'mode=memory' "
+ "query string argument is deprecated; a future release will "
+ "use %s for this URL." % (current_pool, future_pool)
+ ):
+ pool_cls = dialect_cls.get_pool_class(make_url(url))
+
+ eq_(pool_cls.__name__, current_pool)
+
+ def test_memory_mode_pool_deprecated_no_uri(self):
+ """test #13433
+
+ without ``uri=true``, the ``mode`` argument is not passed to the
+ driver at all, so that both warnings are emitted.
+
+ """
+
+ with (
+ expect_warnings(
+ "Query string argument\\(s\\) 'mode' are not accepted by "
+ "the pysqlite driver and are being ignored"
+ ),
+ expect_deprecated(
+ "Selection of the SingletonThreadPool pool class based on "
+ "the 'mode=memory' query string argument is deprecated"
+ ),
+ ):
+ e = create_engine("sqlite+pysqlite:///file:foo.db?mode=memory")
- e = create_engine(
- "sqlite+pysqlite:///file:foo.db?mode=memory&uri=true"
- )
assert e.pool.__class__ is pool.SingletonThreadPool
- e = create_engine("sqlite+pysqlite:///foo.db")
- # changed as of 2.0 #7490
- assert e.pool.__class__ is pool.QueuePool
+ @combinations(
+ ("sqlite:///foo.db?charset=utf8", "'charset'"),
+ ("sqlite:///foo.db?cache=shared&nolock=1", "'cache', 'nolock'"),
+ argnames="url, expected",
+ )
+ def test_connect_args_ignored(self, url, expected):
+ """test #13433
+
+ query string arguments that are not accepted by the driver are
+ silently discarded when ``uri=true`` is not present; warn instead.
+
+ """
+
+ d = pysqlite_dialect.dialect()
+ with expect_warnings(
+ "Query string argument\\(s\\) %s are not accepted by the "
+ "pysqlite driver and are being ignored" % expected
+ ):
+ d.create_connect_args(make_url(url))
@combinations(
(
],
)
- @testing.requires.sqlite
def test_plugin_url_registration(self):
- from sqlalchemy.dialects import sqlite
-
global MyEnginePlugin
def side_effect(url, kw):
MyEnginePlugin = Mock(side_effect=side_effect, update_url=update_url)
plugins.register("engineplugin", __name__, "MyEnginePlugin")
+ registry.register("mockdialect", __name__, "MockDialect")
e = create_engine(
- "sqlite:///?plugin=engineplugin&foo=bar&myplugin_arg=bat",
+ "mockdialect://?plugin=engineplugin&foo=bar&myplugin_arg=bat",
logging_name="foob",
)
- eq_(e.dialect.name, "sqlite")
eq_(e.logging_name, "bar")
# plugin args are removed from URL.
eq_(e.url.query, {"foo": "bar"})
- assert isinstance(e.dialect, sqlite.dialect)
+ assert isinstance(e.dialect, MockDialect)
eq_(
MyEnginePlugin.mock_calls,
[
call(
url.make_url(
- "sqlite:///?plugin=engineplugin"
+ "mockdialect://?plugin=engineplugin"
"&foo=bar&myplugin_arg=bat"
),
{},
),
- call.handle_dialect_kwargs(sqlite.dialect, mock.ANY),
+ call.handle_dialect_kwargs(MockDialect, mock.ANY),
call.handle_pool_kwargs(mock.ANY, {"dialect": e.dialect}),
call.engine_created(e),
],
)
- @testing.requires.sqlite
def test_plugin_multiple_url_registration(self):
- from sqlalchemy.dialects import sqlite
-
global MyEnginePlugin1
global MyEnginePlugin2
plugins.register("engineplugin1", __name__, "MyEnginePlugin1")
plugins.register("engineplugin2", __name__, "MyEnginePlugin2")
+ registry.register("mockdialect", __name__, "MockDialect")
url_str = (
- "sqlite:///?plugin=engineplugin1&foo=bar&myplugin1_arg=bat"
+ "mockdialect://?plugin=engineplugin1&foo=bar&myplugin1_arg=bat"
"&plugin=engineplugin2&myplugin2_arg=hoho"
)
e = create_engine(
url_str,
logging_name="foob",
)
- eq_(e.dialect.name, "sqlite")
eq_(e.logging_name, "bar")
# plugin args are removed from URL.
eq_(e.url.query, {"foo": "bar"})
- assert isinstance(e.dialect, sqlite.dialect)
+ assert isinstance(e.dialect, MockDialect)
eq_(
MyEnginePlugin1.mock_calls,
[
call(url.make_url(url_str), {}),
- call.handle_dialect_kwargs(sqlite.dialect, mock.ANY),
+ call.handle_dialect_kwargs(MockDialect, mock.ANY),
call.handle_pool_kwargs(mock.ANY, {"dialect": e.dialect}),
call.engine_created(e),
],
MyEnginePlugin2.mock_calls,
[
call(url.make_url(url_str), {}),
- call.handle_dialect_kwargs(sqlite.dialect, mock.ANY),
+ call.handle_dialect_kwargs(MockDialect, mock.ANY),
call.handle_pool_kwargs(mock.ANY, {"dialect": e.dialect}),
call.engine_created(e),
],
)
- @testing.requires.sqlite
def test_plugin_arg_registration(self):
- from sqlalchemy.dialects import sqlite
-
global MyEnginePlugin
def side_effect(url, kw):
MyEnginePlugin = Mock(side_effect=side_effect, update_url=update_url)
plugins.register("engineplugin", __name__, "MyEnginePlugin")
+ registry.register("mockdialect", __name__, "MockDialect")
e = create_engine(
- "sqlite:///?foo=bar",
+ "mockdialect://?foo=bar",
logging_name="foob",
plugins=["engineplugin"],
myplugin_arg="bat",
)
- eq_(e.dialect.name, "sqlite")
eq_(e.logging_name, "bar")
- assert isinstance(e.dialect, sqlite.dialect)
+ assert isinstance(e.dialect, MockDialect)
eq_(
MyEnginePlugin.mock_calls,
[
- call(url.make_url("sqlite:///?foo=bar"), {}),
- call.handle_dialect_kwargs(sqlite.dialect, mock.ANY),
+ call(url.make_url("mockdialect://?foo=bar"), {}),
+ call.handle_dialect_kwargs(MockDialect, mock.ANY),
call.handle_pool_kwargs(mock.ANY, {"dialect": e.dialect}),
call.engine_created(e),
],
projects / thirdparty / sqlalchemy / sqlalchemy.git / commitdiff
