--- /dev/null
+.. change::
+ :tags: usecase, orm
+ :tickets: 9015
+
+ The behavior of "joining an external transaction into a Session" has been
+ revised and improved, allowing explicit control over how the
+ :class:`_orm.Session` will accommodate an incoming
+ :class:`_engine.Connection` that already has a transaction and possibly a
+ savepoint already established. The new parameter
+ :paramref:`_orm.Session.join_transaction_mode` includes a series of option
+ values which can accommodate the existing transaction in several ways, most
+ importantly allowing a :class:`_orm.Session` to operate in a fully
+ transactional style using savepoints exclusively, while leaving the
+ externally initiated transaction non-committed and active under all
+ circumstances, allowing test suites to rollback all changes that take place
+ within tests.
+
+ Additionally, revised the :meth:`_orm.Session.close` method to fully close
+ out savepoints that may still be present, which also allows the
+ "external transaction" recipe to proceed without warnings if the
+ :class:`_orm.Session` did not explicitly end its own SAVEPOINT
+ transactions.
+
+ .. seealso::
+
+ :ref:`change_9015`
+
not expected to have significant effects on backwards compatibility.
+.. _change_9015:
+
+New transaction join modes for ``Session``
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+The behavior of "joining an external transaction into a Session" has been
+revised and improved, allowing explicit control over how the
+:class:`_orm.Session` will accommodate an incoming :class:`_engine.Connection`
+that already has a transaction and possibly a savepoint already established.
+The new parameter :paramref:`_orm.Session.join_transaction_mode` includes a
+series of option values which can accommodate the existing transaction in
+several ways, most importantly allowing a :class:`_orm.Session` to operate in a
+fully transactional style using savepoints exclusively, while leaving the
+externally initiated transaction non-committed and active under all
+circumstances, allowing test suites to rollback all changes that take place
+within tests.
+
+The primary improvement this allows is that the recipe documented at
+:ref:`session_external_transaction`, which also changed from SQLAlchemy 1.3
+to 1.4, is now simplified to no longer require explicit use of an event
+handler or any mention of an explicit savepoint; by using
+``join_transaction_mode="create_savepoint"``, the :class:`_orm.Session` will
+never affect the state of an incoming transaction, and will instead create a
+savepoint (i.e. "nested transaction") as its root transaction.
+
+The following illustrates part of the example given at
+:ref:`session_external_transaction`; see that section for a full example::
+
+ class SomeTest(TestCase):
+ def setUp(self):
+ # connect to the database
+ self.connection = engine.connect()
+
+ # begin a non-ORM transaction
+ self.trans = self.connection.begin()
+
+ # bind an individual Session to the connection, selecting
+ # "create_savepoint" join_transaction_mode
+ self.session = Session(
+ bind=self.connection, join_transaction_mode="create_savepoint"
+ )
+
+ def tearDown(self):
+ self.session.close()
+
+ # rollback non-ORM transaction
+ self.trans.rollback()
+
+ # return connection to the Engine
+ self.connection.close()
+
+The default mode selected for :paramref:`_orm.Session.join_transaction_mode`
+is ``"conditional_savepoint"``, which uses ``"create_savepoint"`` behavior
+if the given :class:`_engine.Connection` is itself already on a savepoint.
+If the given :class:`_engine.Connection` is in a transaction but not a
+savepoint, the :class:`_orm.Session` will propagate "rollback" calls
+but not "commit" calls, but will not begin a new savepoint on its own. This
+behavior is chosen by default for its maximum compatibility with
+older SQLAlchemy versions as well as that it does not start a new SAVEPOINT
+unless the given driver is already making use of SAVEPOINT, as support
+for SAVEPOINT varies not only with specific backend and driver but also
+configurationally.
+
+The following illustrates a case that worked in SQLAlchemy 1.3, stopped working
+in SQLAlchemy 1.4, and is now restored in SQLAlchemy 2.0::
+
+ engine = create_engine("...")
+
+ # setup outer connection with a transaction and a SAVEPOINT
+ conn = engine.connect()
+ trans = conn.begin()
+ nested = conn.begin_nested()
+
+ # bind a Session to that connection and operate upon it, including
+ # a commit
+ session = Session(conn)
+ session.connection()
+ session.commit()
+ session.close()
+
+ # assert both SAVEPOINT and transaction remain active
+ assert nested.is_active
+ nested.rollback()
+ trans.rollback()
+
+Where above, a :class:`_orm.Session` is joined to a :class:`_engine.Connection`
+that has a savepoint started on it; the state of these two units remains
+unchanged after the :class:`_orm.Session` has worked with the transaction. In
+SQLAlchemy 1.3, the above case worked because the :class:`_orm.Session` would
+begin a "subtransaction" upon the :class:`_engine.Connection`, which would
+allow the outer savepoint / transaction to remain unaffected for simple cases
+as above. Since subtransactions were deprecated in 1.4 and are now removed in
+2.0, this behavior was no longer available. The new default behavior improves
+upon the behavior of "subtransactions" by using a real, second SAVEPOINT
+instead, so that even calls to :meth:`_orm.Session.rollback` prevent the
+:class:`_orm.Session` from "breaking out" into the externally initiated
+SAVEPOINT or transaction.
+
+New code that is joining a transaction-started :class:`_engine.Connection` into
+a :class:`_orm.Session` should however select a
+:paramref:`_orm.Session.join_transaction_mode` explicitly, so that the desired
+behavior is explicitly defined.
+
+:ticket:`9015`
+
+
.. _Cython: https://cython.org/
.. _change_8567:
including the ability to call :meth:`.Session.commit`, where afterwards the
entire database interaction is rolled back.
-.. versionchanged:: 1.4 This section introduces a new version of the
- "join into an external transaction" recipe that will work equally well
- for both :term:`2.0 style` and :term:`1.x style` engines and sessions.
- The recipe here from previous versions such as 1.3 will also continue to
- work for 1.x engines and sessions.
-
+.. versionchanged:: 2.0 The "join into an external transaction" recipe is
+ newly improved again in 2.0; event handlers to "reset" the nested
+ transaction are no longer required.
The recipe works by establishing a :class:`_engine.Connection` within a
-transaction and optionally a SAVEPOINT, then passing it to a :class:`_orm.Session` as the
-"bind". The :class:`_orm.Session` detects that the given :class:`_engine.Connection`
-is already in a transaction and will not run COMMIT on it if the transaction
-is in fact an outermost transaction. Then when the test tears down, the
-transaction is rolled back so that any data changes throughout the test
-are reverted::
+transaction and optionally a SAVEPOINT, then passing it to a
+:class:`_orm.Session` as the "bind"; the
+:paramref:`_orm.Session.join_transaction_mode` parameter is passed with the
+setting ``"create_savepoint"``, which indicates that new SAVEPOINTs should be
+created in order to implement BEGIN/COMMIT/ROLLBACK for the
+:class:`_orm.Session`, which will leave the external transaction in the same
+state in which it was passed.
+
+When the test tears down, the external transaction is rolled back so that any
+data changes throughout the test are reverted::
from sqlalchemy.orm import sessionmaker
from sqlalchemy import create_engine
# begin a non-ORM transaction
self.trans = self.connection.begin()
- # bind an individual Session to the connection
- self.session = Session(bind=self.connection)
-
- ### optional ###
-
- # if the database supports SAVEPOINT (SQLite needs special
- # config for this to work), starting a savepoint
- # will allow tests to also use rollback within tests
-
- self.nested = self.connection.begin_nested()
-
- @event.listens_for(self.session, "after_transaction_end")
- def end_savepoint(session, transaction):
- if not self.nested.is_active:
- self.nested = self.connection.begin_nested()
-
- ### ^^^ optional ^^^ ###
+ # bind an individual Session to the connection, selecting
+ # "create_savepoint" join_transaction_mode
+ self.session = Session(
+ bind=self.connection, join_transaction_mode="create_savepoint"
+ )
def test_something(self):
# use the session in tests.
self.session.commit()
def test_something_with_rollbacks(self):
- # if the SAVEPOINT steps are taken, then a test can also
- # use session.rollback() and continue working with the database
self.session.add(Bar())
self.session.flush()
_SessionBindKey = Union[Type[Any], "Mapper[Any]", "Table", str]
_SessionBind = Union["Engine", "Connection"]
+JoinTransactionMode = Literal[
+ "conditional_savepoint",
+ "rollback_only",
+ "control_fully",
+ "create_savepoint",
+]
+
class _ConnectionCallableProto(Protocol):
"""a callable that returns a :class:`.Connection` given an instance.
transaction: Transaction
if self.session.twophase and self._parent is None:
+ # TODO: shouldn't we only be here if not
+ # conn.in_transaction() ?
+ # if twophase is set and conn.in_transaction(), validate
+ # that it is in fact twophase.
transaction = conn.begin_twophase()
elif self.nested:
transaction = conn.begin_nested()
elif conn.in_transaction():
- # if given a future connection already in a transaction, don't
- # commit that transaction unless it is a savepoint
- if conn.in_nested_transaction():
- transaction = conn._get_required_nested_transaction()
+ join_transaction_mode = self.session.join_transaction_mode
+
+ if join_transaction_mode == "conditional_savepoint":
+ if conn.in_nested_transaction():
+ join_transaction_mode = "create_savepoint"
+ else:
+ join_transaction_mode = "rollback_only"
+
+ if join_transaction_mode in (
+ "control_fully",
+ "rollback_only",
+ ):
+ if conn.in_nested_transaction():
+ transaction = conn._get_required_nested_transaction()
+ else:
+ transaction = conn._get_required_transaction()
+ if join_transaction_mode == "rollback_only":
+ should_commit = False
+ elif join_transaction_mode == "create_savepoint":
+ transaction = conn.begin_nested()
else:
- transaction = conn._get_required_transaction()
- should_commit = False
+ assert False, join_transaction_mode
else:
transaction = conn.begin()
except:
_StateChangeStates.ANY, SessionTransactionState.CLOSED
)
def close(self, invalidate: bool = False) -> None:
+
if self.nested:
self.session._nested_transaction = (
self._previous_nested_transaction
self.session._transaction = self._parent
- if self._parent is None:
- for connection, transaction, should_commit, autoclose in set(
- self._connections.values()
- ):
- if invalidate:
- connection.invalidate()
- if should_commit and transaction.is_active:
- transaction.close()
- if autoclose:
- connection.close()
+ for connection, transaction, should_commit, autoclose in set(
+ self._connections.values()
+ ):
+ if invalidate and self._parent is None:
+ connection.invalidate()
+ if should_commit and transaction.is_active:
+ transaction.close()
+ if autoclose and self._parent is None:
+ connection.close()
self._state = SessionTransactionState.CLOSED
sess = self.session
expire_on_commit: bool
enable_baked_queries: bool
twophase: bool
+ join_transaction_mode: JoinTransactionMode
_query_cls: Type[Query[Any]]
def __init__(
info: Optional[_InfoType] = None,
query_cls: Optional[Type[Query[Any]]] = None,
autocommit: Literal[False] = False,
+ join_transaction_mode: JoinTransactionMode = "conditional_savepoint",
):
r"""Construct a new Session.
:param autocommit: the "autocommit" keyword is present for backwards
compatibility but must remain at its default value of ``False``.
+ :param join_transaction_mode: Describes the transactional behavior to
+ take when a given bind is a :class:`_engine.Connection` that
+ has already begun a transaction outside the scope of this
+ :class:`_orm.Session`; in other words the
+ :meth:`_engine.Connection.in_transaction()` method returns True.
+
+ The following behaviors only take effect when the :class:`_orm.Session`
+ **actually makes use of the connection given**; that is, a method
+ such as :meth:`_orm.Session.execute`, :meth:`_orm.Session.connection`,
+ etc. are actually invoked:
+
+ * ``"conditional_savepoint"`` - this is the default. if the given
+ :class:`_engine.Connection` is begun within a transaction but
+ does not have a SAVEPOINT, then ``"rollback_only"`` is used.
+ If the :class:`_engine.Connection` is additionally within
+ a SAVEPOINT, in other words
+ :meth:`_engine.Connection.in_nested_transaction()` method returns
+ True, then ``"create_savepoint"`` is used.
+
+ ``"conditional_savepoint"`` behavior attempts to make use of
+ savepoints in order to keep the state of the existing transaction
+ unchanged, but only if there is already a savepoint in progress;
+ otherwise, it is not assumed that the backend in use has adequate
+ support for SAVEPOINT, as availability of this feature varies.
+ ``"conditional_savepoint"`` also seeks to establish approximate
+ backwards compatibility with previous :class:`_orm.Session`
+ behavior, for applications that are not setting a specific mode. It
+ is recommended that one of the explicit settings be used.
+
+ * ``"create_savepoint"`` - the :class:`_orm.Session` will use
+ :meth:`_engine.Connection.begin_nested()` in all cases to create
+ its own transaction. This transaction by its nature rides
+ "on top" of any existing transaction that's opened on the given
+ :class:`_engine.Connection`; if the underlying database and
+ the driver in use has full, non-broken support for SAVEPOINT, the
+ external transaction will remain unaffected throughout the
+ lifespan of the :class:`_orm.Session`.
+
+ The ``"create_savepoint"`` mode is the most useful for integrating
+ a :class:`_orm.Session` into a test suite where an externally
+ initiated transaction should remain unaffected; however, it relies
+ on proper SAVEPOINT support from the underlying driver and
+ database.
+
+ .. tip:: When using SQLite, the SQLite driver included through
+ Python 3.11 does not handle SAVEPOINTs correctly in all cases
+ without workarounds. See the section
+ :ref:`pysqlite_serializable` for details on current workarounds.
+
+ * ``"control_fully"`` - the :class:`_orm.Session` will take
+ control of the given transaction as its own;
+ :meth:`_orm.Session.commit` will call ``.commit()`` on the
+ transaction, :meth:`_orm.Session.rollback` will call
+ ``.rollback()`` on the transaction, :meth:`_orm.Session.close` will
+ call ``.rollback`` on the transaction.
+
+ .. tip:: This mode of use is equivalent to how SQLAlchemy 1.4 would
+ handle a :class:`_engine.Connection` given with an existing
+ SAVEPOINT (i.e. :meth:`_engine.Connection.begin_nested`); the
+ :class:`_orm.Session` would take full control of the existing
+ SAVEPOINT.
+
+ * ``"rollback_only"`` - the :class:`_orm.Session` will take control
+ of the given transaction for ``.rollback()`` calls only;
+ ``.commit()`` calls will not be propagated to the given
+ transaction. ``.close()`` calls will have no effect on the
+ given transaction.
+
+ .. tip:: This mode of use is equivalent to how SQLAlchemy 1.4 would
+ handle a :class:`_engine.Connection` given with an existing
+ regular database transaction (i.e.
+ :meth:`_engine.Connection.begin`); the :class:`_orm.Session`
+ would propagate :meth:`_orm.Session.rollback` calls to the
+ underlying transaction, but not :meth:`_orm.Session.commit` or
+ :meth:`_orm.Session.close` calls.
+
+ .. versionadded:: 2.0.0b5
+
+
""" # noqa
# considering allowing the "autocommit" keyword to still be accepted
self.autoflush = autoflush
self.expire_on_commit = expire_on_commit
self.enable_baked_queries = enable_baked_queries
+ if (
+ join_transaction_mode
+ and join_transaction_mode
+ not in JoinTransactionMode.__args__ # type: ignore
+ ):
+ raise sa_exc.ArgumentError(
+ f"invalid selection for join_transaction_mode: "
+ f'"{join_transaction_mode}"'
+ )
+ self.join_transaction_mode = join_transaction_mode
self.twophase = twophase
self._query_cls = query_cls if query_cls else query.Query
watchdog.symmetric_difference(self._class_methods),
)
+ def test_join_transaction_mode(self):
+ with expect_raises_message(
+ sa.exc.ArgumentError,
+ 'invalid selection for join_transaction_mode: "bogus"',
+ ):
+ Session(join_transaction_mode="bogus")
+
def test_unmapped_instance(self):
class Unmapped:
pass
+from __future__ import annotations
+
import contextlib
+import random
+from typing import Optional
+from typing import TYPE_CHECKING
from sqlalchemy import Column
from sqlalchemy import event
from sqlalchemy.testing import is_
from sqlalchemy.testing import is_not
from sqlalchemy.testing import mock
+from sqlalchemy.testing.config import Variation
from sqlalchemy.testing.fixtures import fixture_session
from sqlalchemy.testing.util import gc_collect
from test.orm._fixtures import FixtureTest
+if TYPE_CHECKING:
+ from sqlalchemy import NestedTransaction
+ from sqlalchemy import Transaction
+
class SessionTransactionTest(fixtures.RemovesEvents, FixtureTest):
run_inserts = None
trans.commit()
assert len(sess.query(User).all()) == 1
+ @testing.variation(
+ "join_transaction_mode",
+ [
+ "none",
+ "conditional_savepoint",
+ "create_savepoint",
+ "control_fully",
+ "rollback_only",
+ ],
+ )
+ @testing.variation("operation", ["commit", "close", "rollback", "nothing"])
+ @testing.variation("external_state", ["none", "transaction", "savepoint"])
+ def test_join_transaction_modes(
+ self,
+ connection_no_trans,
+ join_transaction_mode,
+ operation,
+ external_state: testing.Variation,
+ ):
+ """test new join_transaction modes added in #9015"""
+
+ connection = connection_no_trans
+
+ t1: Optional[Transaction]
+ s1: Optional[NestedTransaction]
+
+ if external_state.none:
+ t1 = s1 = None
+ elif external_state.transaction:
+ t1 = connection.begin()
+ s1 = None
+ elif external_state.savepoint:
+ t1 = connection.begin()
+ s1 = connection.begin_nested()
+ else:
+ external_state.fail()
+
+ if join_transaction_mode.none:
+ sess = Session(connection)
+ else:
+ sess = Session(
+ connection, join_transaction_mode=join_transaction_mode.name
+ )
+
+ sess.connection()
+
+ if operation.close:
+ sess.close()
+ elif operation.commit:
+ sess.commit()
+ elif operation.rollback:
+ sess.rollback()
+ elif operation.nothing:
+ pass
+ else:
+ operation.fail()
+
+ if external_state.none:
+ if operation.nothing:
+ assert connection.in_transaction()
+ else:
+ assert not connection.in_transaction()
+
+ elif external_state.transaction:
+
+ assert t1 is not None
+
+ if (
+ join_transaction_mode.none
+ or join_transaction_mode.conditional_savepoint
+ or join_transaction_mode.rollback_only
+ ):
+ if operation.rollback:
+ assert t1._deactivated_from_connection
+ assert not t1.is_active
+ else:
+ assert not t1._deactivated_from_connection
+ assert t1.is_active
+ elif join_transaction_mode.create_savepoint:
+ assert not t1._deactivated_from_connection
+ assert t1.is_active
+ elif join_transaction_mode.control_fully:
+ if operation.nothing:
+ assert not t1._deactivated_from_connection
+ assert t1.is_active
+ else:
+ assert t1._deactivated_from_connection
+ assert not t1.is_active
+ else:
+ join_transaction_mode.fail()
+
+ if t1.is_active:
+ t1.rollback()
+ elif external_state.savepoint:
+ assert s1 is not None
+ assert t1 is not None
+
+ assert not t1._deactivated_from_connection
+ assert t1.is_active
+
+ if join_transaction_mode.rollback_only:
+ if operation.rollback:
+ assert s1._deactivated_from_connection
+ assert not s1.is_active
+ else:
+ assert not s1._deactivated_from_connection
+ assert s1.is_active
+ elif join_transaction_mode.control_fully:
+ if operation.nothing:
+ assert not s1._deactivated_from_connection
+ assert s1.is_active
+ else:
+ assert s1._deactivated_from_connection
+ assert not s1.is_active
+ else:
+ if operation.nothing:
+ # session is still open in the sub-savepoint,
+ # so we are not activated on connection
+ assert s1._deactivated_from_connection
+
+ # but we are still an active savepoint
+ assert s1.is_active
+
+ # close session, then we're good
+ sess.close()
+
+ assert not s1._deactivated_from_connection
+ assert s1.is_active
+
+ if s1.is_active:
+ s1.rollback()
+ if t1.is_active:
+ t1.rollback()
+ else:
+ external_state.fail()
+
def test_subtransaction_on_external_commit(self, connection_no_trans):
users, User = self.tables.users, self.classes.User
eq_(result, count)
+class CtxManagerJoinIntoAnExternalTransactionFixture(
+ JoinIntoAnExternalTransactionFixture
+):
+ @testing.requires.compat_savepoints
+ def test_something_with_context_managers(self):
+ A = self.A
+
+ a1 = A()
+
+ with self.session.begin():
+ self.session.add(a1)
+ self.session.flush()
+
+ self._assert_count(1)
+ self.session.rollback()
+
+ self._assert_count(0)
+
+ a1 = A()
+ with self.session.begin():
+ self.session.add(a1)
+
+ self._assert_count(1)
+
+ a2 = A()
+
+ with self.session.begin():
+ self.session.add(a2)
+ self.session.flush()
+ self._assert_count(2)
+
+ self.session.rollback()
+ self._assert_count(1)
+
+ @testing.requires.compat_savepoints
+ def test_super_abusive_nesting(self):
+ session = self.session
+
+ for i in range(random.randint(5, 30)):
+ choice = random.randint(1, 3)
+ if choice == 1:
+ if session.in_transaction():
+ session.begin_nested()
+ else:
+ session.begin()
+ elif choice == 2:
+ session.rollback()
+ elif choice == 3:
+ session.commit()
+
+ session.connection()
+
+ # remaining nested / etc. are cleanly cleared out
+ session.close()
+
+
class NewStyleJoinIntoAnExternalTransactionTest(
- JoinIntoAnExternalTransactionFixture, fixtures.MappedTest
+ CtxManagerJoinIntoAnExternalTransactionFixture, fixtures.MappedTest
):
- """A new recipe for "join into an external transaction" that works
- for both legacy and future engines/sessions
+ """test the 1.4 join to an external transaction fixture.
+
+ In 1.4, this works for both legacy and future engines/sessions
"""
if self.trans.is_active:
self.trans.rollback()
- @testing.requires.compat_savepoints
- def test_something_with_context_managers(self):
- A = self.A
- a1 = A()
+@testing.combinations(
+ *Variation.generate_cases(
+ "join_mode",
+ [
+ "create_savepoint",
+ "conditional_w_savepoint",
+ "create_savepoint_w_savepoint",
+ ],
+ ),
+ argnames="join_mode",
+ id_="s",
+)
+class ReallyNewJoinIntoAnExternalTransactionTest(
+ CtxManagerJoinIntoAnExternalTransactionFixture, fixtures.MappedTest
+):
+ """2.0 only recipe for "join into an external transaction" that works
+ without event handlers
- with self.session.begin():
- self.session.add(a1)
- self.session.flush()
+ """
- self._assert_count(1)
- self.session.rollback()
+ def setup_session(self):
+ self.trans = self.connection.begin()
- self._assert_count(0)
+ if (
+ self.join_mode.conditional_w_savepoint
+ or self.join_mode.create_savepoint_w_savepoint
+ ):
+ self.nested = self.connection.begin_nested()
- a1 = A()
- with self.session.begin():
- self.session.add(a1)
+ class A:
+ pass
- self._assert_count(1)
+ clear_mappers()
+ self.mapper_registry.map_imperatively(A, self.table)
+ self.A = A
- a2 = A()
+ self.session = Session(
+ self.connection,
+ join_transaction_mode="create_savepoint"
+ if (
+ self.join_mode.create_savepoint
+ or self.join_mode.create_savepoint_w_savepoint
+ )
+ else "conditional_savepoint",
+ )
- with self.session.begin():
- self.session.add(a2)
- self.session.flush()
- self._assert_count(2)
+ def teardown_session(self):
+ self.session.close()
- self.session.rollback()
- self._assert_count(1)
+ if (
+ self.join_mode.conditional_w_savepoint
+ or self.join_mode.create_savepoint_w_savepoint
+ ):
+ assert not self.nested._deactivated_from_connection
+ assert self.nested.is_active
+ self.nested.rollback()
+
+ assert not self.trans._deactivated_from_connection
+ assert self.trans.is_active
+ self.trans.rollback()
class LegacyJoinIntoAnExternalTransactionTest(
JoinIntoAnExternalTransactionFixture,
fixtures.MappedTest,
):
+ """test the 1.3 join to an external transaction fixture"""
+
def setup_session(self):
# begin a non-ORM transaction
self.trans = self.connection.begin()
projects / thirdparty / sqlalchemy / sqlalchemy.git / commitdiff
