--- /dev/null
+.. change::
+ :tags: usecase, engine
+ :tickets: 12784
+
+ Added new parameter :paramref:`.create_engine.skip_autocommit_rollback`
+ which provides for a per-dialect feature of preventing the DBAPI
+ ``.rollback()`` from being called under any circumstances, if the
+ connection is detected as being in "autocommit" mode. This improves upon
+ a critical performance issue identified in MySQL dialects where the network
+ overhead of the ``.rollback()`` call remains prohibitive even if autocommit
+ mode is set.
+
+ .. seealso::
+
+ :ref:`dbapi_autocommit_skip_rollback`
:ref:`dbapi_autocommit_understanding`, that "autocommit" isolation level like
any other isolation level does **not** affect the "transactional" behavior of
the :class:`_engine.Connection` object, which continues to call upon DBAPI
- ``.commit()`` and ``.rollback()`` methods (they just have no effect under
+ ``.commit()`` and ``.rollback()`` methods (they just have no net effect under
autocommit), and for which the ``.begin()`` method assumes the DBAPI will
start a transaction implicitly (which means that SQLAlchemy's "begin" **does
not change autocommit mode**).
set at this level. This because the option must be set on a DBAPI connection
on a per-transaction basis.
+.. _dbapi_autocommit_engine:
+
Setting Isolation Level or DBAPI Autocommit for an Engine
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
be set to use a ``"REPEATABLE READ"`` isolation level setting for all
subsequent operations.
+.. tip::
+
+ Prefer to set frequently used isolation levels engine wide as illustrated
+ above compared to using per-engine or per-connection execution options for
+ maximum performance.
+
.. _dbapi_autocommit_multiple:
Maintaining Multiple Isolation Levels for a Single Engine
~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The isolation level may also be set per engine, with a potentially greater
-level of flexibility, using either the
-:paramref:`_sa.create_engine.execution_options` parameter to
+level of flexibility but with a small per-connection performance overhead,
+using either the :paramref:`_sa.create_engine.execution_options` parameter to
:func:`_sa.create_engine` or the :meth:`_engine.Engine.execution_options`
method, the latter of which will create a copy of the :class:`.Engine` that
shares the dialect and connection pool of the original engine, but has its own
The isolation level setting, regardless of which one it is, is unconditionally
reverted when a connection is returned to the connection pool.
+.. note::
+
+ The execution options approach, whether used engine wide or per connection,
+ incurs a small performance penalty as isolation level instructions
+ are sent on connection acquire as well as connection release. Consider
+ the engine-wide isolation setting at :ref:`dbapi_autocommit_engine` so
+ that connections are configured at the target isolation level permanently
+ as they are pooled.
.. seealso::
It is important to note that "autocommit" mode
persists even when the :meth:`_engine.Connection.begin` method is called;
-the DBAPI will not emit any BEGIN to the database, nor will it emit
-COMMIT when :meth:`_engine.Connection.commit` is called. This usage is also
+the DBAPI will not emit any BEGIN to the database. When
+:meth:`_engine.Connection.commit` is called, the DBAPI may still emit the
+"COMMIT" instruction, but this is a no-op at the database level. This usage is also
not an error scenario, as it is expected that the "autocommit" isolation level
may be applied to code that otherwise was written assuming a transactional context;
the "isolation level" is, after all, a configurational detail of the transaction
INFO sqlalchemy.engine.Engine BEGIN (implicit)
...
- INFO sqlalchemy.engine.Engine COMMIT using DBAPI connection.commit(), DBAPI should ignore due to autocommit mode
+ INFO sqlalchemy.engine.Engine COMMIT using DBAPI connection.commit(), has no effect due to autocommit mode
At the same time, even though we are using "DBAPI autocommit", SQLAlchemy's
transactional semantics, that is, the in-Python behavior of :meth:`_engine.Connection.begin`
:class:`_engine.Connection` where DBAPI-autocommit mode can be changed
independently without indicating any code changes elsewhere.
+.. _dbapi_autocommit_skip_rollback:
+
+Fully preventing ROLLBACK calls under autocommit
+^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
+
+.. versionadded:: 2.0.43
+
+A common use case is to use AUTOCOMMIT isolation mode to improve performance,
+and this is a particularly common practice on MySQL / MariaDB databases.
+When seeking this pattern, it should be preferred to set AUTOCOMMIT engine
+wide using the :paramref:`.create_engine.isolation_level` so that pooled
+connections are permanently set in autocommit mode. The SQLAlchemy connection
+pool as well as the :class:`.Connection` will still seek to invoke the DBAPI
+``.rollback()`` method upon connection :term:`reset`, as their behavior
+remains agonstic of the isolation level that's configured on the connection.
+As this rollback still incurs a network round trip under most if not all
+DBAPI drivers, this additional network trip may be disabled using the
+:paramref:`.create_engine.skip_autocommit_rollback` parameter, which will
+apply a rule at the basemost portion of the dialect that invokes DBAPI
+``.rollback()`` to first check if the connection is configured in autocommit,
+using a method of detection that does not itself incur network overhead::
+
+ autocommit_engine = create_engine(
+ "mysql+mysqldb://scott:tiger@mysql80/test",
+ skip_autocommit_rollback=True,
+ isolation_level="AUTOCOMMIT",
+ )
+
+When DBAPI connections are returned to the pool by the :class:`.Connection`,
+whether the :class:`.Connection` or the pool attempts to reset the
+"transaction", the underlying DBAPI ``.rollback()`` method will be blocked
+based on a positive test of "autocommit".
+
+If the dialect in use does not support a no-network means of detecting
+autocommit, the dialect will raise ``NotImplementedError`` when a connection
+release is attempted.
+
Changing Between Isolation Levels
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
method of the DBAPI connection when the connection is returned to the pool.
This is so that any existing transactional state is removed from the
connection, which includes not just uncommitted data but table and row locks as
-well. For most DBAPIs, the call to ``rollback()`` is inexpensive, and if the
-DBAPI has already completed a transaction, the method should be a no-op.
+well. For most DBAPIs, the call to ``rollback()`` is relatively inexpensive.
+
+The "reset on return" feature takes place when a connection is :term:`released`
+back to the connection pool. In modern SQLAlchemy, this reset on return
+behavior is shared between the :class:`.Connection` and the :class:`.Pool`,
+where the :class:`.Connection` itself, if it releases its transaction upon close,
+considers ``.rollback()`` to have been called, and instructs the pool to skip
+this step.
Disabling Reset on Return for non-transactional connections
:ref:`autocommit <dbapi_autocommit_understanding>` or when using a database
that has no ACID capabilities such as the MyISAM engine of MySQL, the
reset-on-return behavior can be disabled, which is typically done for
-performance reasons. This can be affected by using the
+performance reasons.
+
+As of SQLAlchemy 2.0.43, the :paramref:`.create_engine.skip_autocommit_rollback`
+parameter of :func:`.create_engine` provides the most complete means of
+preventing ROLLBACK from being emitted while under autocommit mode, as it
+blocks the DBAPI ``.rollback()`` method from being called by the dialect
+completely::
+
+ autocommit_engine = create_engine(
+ "mysql+mysqldb://scott:tiger@mysql80/test",
+ skip_autocommit_rollback=True,
+ isolation_level="AUTOCOMMIT",
+ )
+
+Detail on this pattern is at :ref:`dbapi_autocommit_skip_rollback`.
+
+The :class:`_pool.Pool` itself also has a parameter that can control its
+"reset on return" behavior, noting that in modern SQLAlchemy this is not
+the only path by which the DBAPI transaction is released, which is the
:paramref:`_pool.Pool.reset_on_return` parameter of :class:`_pool.Pool`, which
is also available from :func:`_sa.create_engine` as
:paramref:`_sa.create_engine.pool_reset_on_return`, passing a value of ``None``.
-This is illustrated in the example below, in conjunction with the
-:paramref:`.create_engine.isolation_level` parameter setting of
-``AUTOCOMMIT``::
+This pattern looks as below::
- non_acid_engine = create_engine(
- "mysql://scott:tiger@host/db",
+ autocommit_engine = create_engine(
+ "mysql+mysqldb://scott:tiger@mysql80/test",
pool_reset_on_return=None,
isolation_level="AUTOCOMMIT",
)
-The above engine won't actually perform ROLLBACK when connections are returned
-to the pool; since AUTOCOMMIT is enabled, the driver will also not perform
-any BEGIN operation.
-
+The above pattern will still see ROLLBACKs occur however as the :class:`.Connection`
+object implicitly starts transaction blocks in the SQLAlchemy 2.0 series,
+which still emit ROLLBACK independently of the pool's reset sequence.
Custom Reset-on-Return Schemes
^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
else:
dbapi_connection.autocommit = False
super().set_isolation_level(dbapi_connection, level)
+
+ def detect_autocommit_setting(
+ self, dbapi_conn: interfaces.DBAPIConnection
+ ) -> bool:
+ return bool(dbapi_conn.autocommit)
def autocommit(self, value: Any) -> None:
self.await_(self._connection.autocommit(value))
+ def get_autocommit(self) -> bool:
+ return self._connection.get_autocommit() # type: ignore
+
def terminate(self) -> None:
# it's not awaitable.
self._connection.close()
def autocommit(self, value: Any) -> None:
self.await_(self._connection.autocommit(value))
+ def get_autocommit(self) -> bool:
+ return self._connection.get_autocommit() # type: ignore
+
def terminate(self) -> None:
# it's not awaitable.
self._connection.close()
"AUTOCOMMIT",
)
+ def detect_autocommit_setting(self, dbapi_conn: DBAPIConnection) -> bool:
+ return bool(dbapi_conn.autocommit)
+
def set_isolation_level(
self, dbapi_connection: DBAPIConnection, level: IsolationLevel
) -> None:
"AUTOCOMMIT",
)
+ def detect_autocommit_setting(self, dbapi_conn: DBAPIConnection) -> bool:
+ return bool(dbapi_conn.autocommit)
+
def set_isolation_level(
self, dbapi_connection: DBAPIConnection, level: IsolationLevel
) -> None:
"AUTOCOMMIT",
)
+ def detect_autocommit_setting(self, dbapi_conn: DBAPIConnection) -> bool:
+ return dbapi_conn.get_autocommit() # type: ignore[no-any-return]
+
def set_isolation_level(
self, dbapi_connection: DBAPIConnection, level: IsolationLevel
) -> None:
with dbapi_connection.cursor() as cursor:
cursor.execute(f"ALTER SESSION SET ISOLATION_LEVEL={level}")
+ def detect_autocommit_setting(self, dbapi_conn) -> bool:
+ return bool(dbapi_conn.autocommit)
+
def _detect_decimal_char(self, connection):
# we have the option to change this setting upon connect,
# or just look at what it is upon connect and convert.
def _do_autocommit(self, connection, value):
connection.autocommit = value
+ def detect_autocommit_setting(self, dbapi_connection):
+ return bool(dbapi_connection.autocommit)
+
def do_ping(self, dbapi_connection):
before_autocommit = dbapi_connection.autocommit
def set_isolation_level(self, dbapi_connection, level):
dbapi_connection.set_isolation_level(self._isolation_lookup[level])
+ def detect_autocommit_setting(self, dbapi_conn) -> bool:
+ return bool(dbapi_conn.autocommit)
+
def set_readonly(self, connection, value):
connection.readonly = value
cursor.execute("COMMIT")
cursor.close()
+ def detect_autocommit_setting(self, dbapi_conn) -> bool:
+ return bool(dbapi_conn.autocommit)
+
def set_readonly(self, connection, value):
cursor = connection.cursor()
try:
dbapi_connection.isolation_level = ""
return super().set_isolation_level(dbapi_connection, level)
+ def detect_autocommit_setting(self, dbapi_connection):
+ return dbapi_connection.isolation_level is None
+
def on_connect(self):
def regexp(a, b):
if b is None:
if self._still_open_and_dbapi_connection_is_valid:
if self._echo:
if self._is_autocommit_isolation():
- self._log_info(
- "ROLLBACK using DBAPI connection.rollback(), "
- "DBAPI should ignore due to autocommit mode"
- )
+ if self.dialect.skip_autocommit_rollback:
+ self._log_info(
+ "ROLLBACK will be skipped by "
+ "skip_autocommit_rollback"
+ )
+ else:
+ self._log_info(
+ "ROLLBACK using DBAPI connection.rollback(); "
+ "set skip_autocommit_rollback to prevent fully"
+ )
else:
self._log_info("ROLLBACK")
try:
if self._is_autocommit_isolation():
self._log_info(
"COMMIT using DBAPI connection.commit(), "
- "DBAPI should ignore due to autocommit mode"
+ "has no effect due to autocommit mode"
)
else:
self._log_info("COMMIT")
:ref:`connections_toplevel`
+ :param skip_autocommit_rollback: When True, the dialect will
+ unconditionally skip all calls to the DBAPI ``connection.rollback()``
+ method if the DBAPI connection is confirmed to be in "autocommit" mode.
+ The availability of this feature is dialect specific; if not available,
+ a ``NotImplementedError`` is raised by the dialect when rollback occurs.
+
+ .. seealso::
+
+ :ref:`dbapi_autocommit_skip_rollback`
+
+ .. versionadded:: 2.0.43
+
:param connect_args: a dictionary of options which will be
passed directly to the DBAPI's ``connect()`` method as
additional keyword arguments. See the example
:ref:`pool_reset_on_return`
+ :ref:`dbapi_autocommit_skip_rollback` - a more modern approach
+ to using connections with no transactional instructions
+
:param pool_timeout=30: number of seconds to wait before giving
up on getting a connection from the pool. This is only used
with :class:`~sqlalchemy.pool.QueuePool`. This can be a float but is
# Linting.NO_LINTING constant
compiler_linting: Linting = int(compiler.NO_LINTING), # type: ignore
server_side_cursors: bool = False,
+ skip_autocommit_rollback: bool = False,
**kwargs: Any,
):
if server_side_cursors:
self.dbapi = dbapi
+ self.skip_autocommit_rollback = skip_autocommit_rollback
+
if paramstyle is not None:
self.paramstyle = paramstyle
elif self.dbapi is not None:
pass
def do_rollback(self, dbapi_connection):
+ if self.skip_autocommit_rollback and self.detect_autocommit_setting(
+ dbapi_connection
+ ):
+ return
dbapi_connection.rollback()
def do_commit(self, dbapi_connection):
default_isolation_level: Optional[IsolationLevel]
"""the isolation that is implicitly present on new connections"""
+ skip_autocommit_rollback: bool
+ """Whether or not the :paramref:`.create_engine.skip_autocommit_rollback
+ parameter was set.
+
+ .. versionadded:: 2.0.43
+
+ """
+
# create_engine() -> isolation_level currently goes here
_on_connect_isolation_level: Optional[IsolationLevel]
raise NotImplementedError()
+ def detect_autocommit_setting(self, dbapi_conn: DBAPIConnection) -> bool:
+ """Detect the current autocommit setting for a DBAPI connection.
+
+ :param dbapi_connection: a DBAPI connection object
+ :return: True if autocommit is enabled, False if disabled
+ :rtype: bool
+
+ This method inspects the given DBAPI connection to determine
+ whether autocommit mode is currently enabled. The specific
+ mechanism for detecting autocommit varies by database dialect
+ and DBAPI driver, however it should be done **without** network
+ round trips.
+
+ .. note::
+
+ Not all dialects support autocommit detection. Dialects
+ that do not support this feature will raise
+ :exc:`NotImplementedError`.
+
+ """
+ raise NotImplementedError(
+ "This dialect cannot detect autocommit on a DBAPI connection"
+ )
+
def get_default_isolation_level(
self, dbapi_conn: DBAPIConnection
) -> IsolationLevel:
"""target dialect supports 'AUTOCOMMIT' as an isolation_level"""
return exclusions.closed()
+ @property
+ def skip_autocommit_rollback(self):
+ """target dialect supports the detect_autocommit_setting() method and
+ uses the default implementation of do_rollback()"""
+
+ return exclusions.closed()
+
@property
def isolation_level(self):
"""target dialect supports general isolation level settings.
from .. import fixtures
from .. import is_not_none
from .. import is_true
+from .. import mock
from .. import ne_
from .. import provide_metadata
from ..assertions import expect_raises
test_needs_acid=True,
)
- def _test_conn_autocommits(self, conn, autocommit):
+ def _test_conn_autocommits(self, conn, autocommit, ensure_table=False):
+ if ensure_table:
+ self.tables.some_table.create(conn, checkfirst=True)
+ conn.commit()
+
trans = conn.begin()
conn.execute(
self.tables.some_table.insert(), {"id": 1, "data": "some data"}
)
self._test_conn_autocommits(conn, False)
+ @testing.requires.skip_autocommit_rollback
+ @testing.variation("autocommit_setting", ["false", "engine", "option"])
+ @testing.variation("block_rollback", [True, False])
+ def test_autocommit_block(
+ self, testing_engine, autocommit_setting, block_rollback
+ ):
+ kw = {}
+ if bool(block_rollback):
+ kw["skip_autocommit_rollback"] = True
+ if autocommit_setting.engine:
+ kw["isolation_level"] = "AUTOCOMMIT"
+
+ engine = testing_engine(options=kw)
+
+ conn = engine.connect()
+ if autocommit_setting.option:
+ conn.execution_options(isolation_level="AUTOCOMMIT")
+ self._test_conn_autocommits(
+ conn,
+ autocommit_setting.engine or autocommit_setting.option,
+ ensure_table=True,
+ )
+ with mock.patch.object(
+ conn.connection, "rollback", wraps=conn.connection.rollback
+ ) as check_rollback:
+ conn.close()
+ if autocommit_setting.false or not block_rollback:
+ eq_(check_rollback.mock_calls, [mock.call()])
+ else:
+ eq_(check_rollback.mock_calls, [])
+
@testing.requires.independent_readonly_connections
@testing.variation("use_dialect_setting", [True, False])
def test_dialect_autocommit_is_restored(
@testing.fixture()
def logging_engine(self, testing_engine):
- kw = {"echo": True, "future": True}
+ kw = {"echo": True}
e = testing_engine(options=kw)
e.connect().close()
return e
@testing.fixture()
def autocommit_iso_logging_engine(self, testing_engine):
- kw = {"echo": True, "future": True, "isolation_level": "AUTOCOMMIT"}
+ kw = {"echo": True, "isolation_level": "AUTOCOMMIT"}
e = testing_engine(options=kw)
e.connect().close()
return e
[
"BEGIN (implicit; DBAPI should not "
"BEGIN due to autocommit mode)",
- "COMMIT using DBAPI connection.commit(), DBAPI "
- "should ignore due to autocommit mode",
+ "COMMIT using DBAPI connection.commit(), "
+ "has no effect due to autocommit mode",
]
)
[
"BEGIN (implicit; DBAPI should not "
"BEGIN due to autocommit mode)",
- "COMMIT using DBAPI connection.commit(), DBAPI "
- "should ignore due to autocommit mode",
+ "COMMIT using DBAPI connection.commit(), "
+ "has no effect due to autocommit mode",
]
)
+ @testing.variation("block_rollback", [True, False])
def test_commit_as_you_go_block_rollback_autocommit(
- self, logging_engine, assert_buf
+ self, testing_engine, assert_buf, block_rollback
):
- with logging_engine.connect().execution_options(
- isolation_level="AUTOCOMMIT"
- ) as conn:
+
+ kw = {
+ "echo": True,
+ "isolation_level": "AUTOCOMMIT",
+ "skip_autocommit_rollback": bool(block_rollback),
+ }
+ logging_engine = testing_engine(options=kw)
+ logging_engine.connect().close()
+
+ with logging_engine.connect() as conn:
conn.begin()
conn.rollback()
- assert_buf(
- [
- "BEGIN (implicit; DBAPI should not "
- "BEGIN due to autocommit mode)",
- "ROLLBACK using DBAPI connection.rollback(), DBAPI "
- "should ignore due to autocommit mode",
- ]
- )
+ if block_rollback:
+ assert_buf(
+ [
+ "BEGIN (implicit; DBAPI should not "
+ "BEGIN due to autocommit mode)",
+ "ROLLBACK will be skipped by skip_autocommit_rollback",
+ ]
+ )
+ else:
+ assert_buf(
+ [
+ "BEGIN (implicit; DBAPI should not "
+ "BEGIN due to autocommit mode)",
+ "ROLLBACK using DBAPI connection.rollback(); "
+ "set skip_autocommit_rollback to prevent fully",
+ ]
+ )
def test_logging_compatibility(
self, plain_assert_buf, plain_logging_engine
in self.get_isolation_levels(config)["supported"]
)
+ @property
+ def skip_autocommit_rollback(self):
+ return exclusions.skip_if(
+ ["mssql+pymssql"],
+ "DBAPI has no means of testing the autocommit status of a "
+ "connection",
+ )
+
@property
def row_triggers(self):
"""Target must support standard statement-running EACH ROW triggers."""
projects / thirdparty / sqlalchemy / sqlalchemy.git / commitdiff
