From: Mike Bayer Date: Sat, 27 Jun 2026 15:11:49 +0000 (-0400) Subject: rework migration docs a bit, fix issues X-Git-Tag: rel_2_1_0b3~4 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=01a7ac1a59ccc5ef8ccd3cc3eb6b45ff94757a16;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git rework migration docs a bit, fix issues Change-Id: I0ef6e304e32ee8add3ebc2e325cc7d13e39e38aa --- diff --git a/doc/build/changelog/changelog_21.rst b/doc/build/changelog/changelog_21.rst index ad096cf55e..6cfffde07d 100644 --- a/doc/build/changelog/changelog_21.rst +++ b/doc/build/changelog/changelog_21.rst @@ -876,6 +876,10 @@ :meth:`_orm.Session.get`, :meth:`_orm.Session.scalars`) for that session instance. + .. seealso:: + + :ref:`change_12659` + .. change:: :tags: change, postgresql :tickets: 10594, 12690 diff --git a/doc/build/changelog/migration_21.rst b/doc/build/changelog/migration_21.rst index 154d370bf3..473deeecfa 100644 --- a/doc/build/changelog/migration_21.rst +++ b/doc/build/changelog/migration_21.rst @@ -31,7 +31,7 @@ Asyncio "greenlet" dependency no longer installs by default SQLAlchemy 1.4 and 2.0 used a complex expression to determine if the ``greenlet`` dependency, needed by the :ref:`asyncio ` extension, could be installed from pypi using a pre-built wheel instead -of having to build from source. This because the source build of ``greenlet`` +of having to build from source. This is because the source build of ``greenlet`` is not always trivial on some platforms. Disadvantages to this approach included that SQLAlchemy needed to track @@ -47,73 +47,36 @@ need to be aware of this extra installation dependency. :ticket:`10197` -New Features and Improvements - ORM -==================================== - - - -.. _change_9809: - -Session autoflush behavior simplified to be unconditional ---------------------------------------------------------- - -Session autoflush behavior has been simplified to unconditionally flush the -session each time an execution takes place, regardless of whether an ORM -statement or Core statement is being executed. This change eliminates the -previous conditional logic that only flushed when ORM-related statements -were detected. - -Previously, the session would only autoflush when executing ORM queries:: - - # 2.0 behavior - autoflush only occurred for ORM statements - session.add(User(name="new user")) - - # This would trigger autoflush - users = session.execute(select(User)).scalars().all() - - # This would NOT trigger autoflush - result = session.execute(text("SELECT * FROM users")) - -In 2.1, autoflush occurs for all statement executions:: - - # 2.1 behavior - autoflush occurs for all executions - session.add(User(name="new user")) - - # Both of these now trigger autoflush - users = session.execute(select(User)).scalars().all() - result = session.execute(text("SELECT * FROM users")) - -This change provides more consistent and predictable session behavior across -all types of SQL execution. - -:ticket:`9809` +ORM - New Features +================== +.. _change_12659: .. _change_13346: -Session-level execution options applied to Connection at procurement time + +Session-level execution options added -------------------------------------------------------------------------- -Building on the session-level execution options feature introduced in -:ticket:`12659`, the :paramref:`_orm.Session.execution_options` parameter -now applies its options to the :class:`_engine.Connection` when it is first -procured for a transaction, in addition to being merged into explicit query -executions as before. This means that execution options such as -``schema_translate_map`` as well as custom user-defined options now take -effect for **all** operations within the session, including: +The :class:`_orm.Session`, :class:`_orm.sessionmaker`, +:class:`_orm.scoped_session`, :class:`_ext.asyncio.AsyncSession`, and +:class:`_ext.asyncio.async_sessionmaker` constructors now accept an +:paramref:`_orm.Session.execution_options` parameter, which establishes +a dictionary of execution options that are applied across all operations +within that session instance. These options are propagated both to +explicit query executions such as :meth:`_orm.Session.execute` and +:meth:`_orm.Session.scalars`, and to the :class:`_engine.Connection` +when it is first procured for a transaction. This means that execution +options such as ``schema_translate_map`` as well as custom user-defined +options take effect for **all** operations within the session, including: * Flush operations (INSERT/UPDATE/DELETE emitted by the unit of work) * Event hooks such as :meth:`_events.ConnectionEvents.before_cursor_execute` * Eager loader queries -Previously, session-level execution options were only applied to explicit calls -such as :meth:`_orm.Session.execute`, which meant that -``schema_translate_map`` set on the :class:`_orm.Session` would not take effect -for flush operations. The prior workaround was to set ``schema_translate_map`` -on the :class:`_engine.Engine` itself, which remains supported. - -The new behavior allows ``schema_translate_map`` to be set directly on the -:class:`_orm.Session`:: +For example, ``schema_translate_map`` may be applied to a +:class:`_orm.Session` such that it takes effect for both queries and +flushes:: session = Session( engine, @@ -126,7 +89,7 @@ The new behavior allows ``schema_translate_map`` to be set directly on the results = session.scalars(select(MyObject)).all() -Custom execution options that are consumed in event hooks such as +Custom execution options consumed in event hooks such as :meth:`_events.ConnectionEvents.before_cursor_execute` are also available during flush operations:: @@ -144,8 +107,151 @@ during flush operations:: session.add(SomeObject()) session.flush() # before_cursor_execute sees my_audit_flag=True +:ticket:`12659` + :ticket:`13346` +.. _change_12496: + +New Hybrid DML hook features +---------------------------- + +To complement the existing :meth:`.hybrid_property.update_expression` decorator, +a new decorator :meth:`.hybrid_property.bulk_dml` is added, which works +specifically with parameter dictionaries passed to :meth:`_orm.Session.execute` +when dealing with ORM-enabled :func:`_dml.insert` or :func:`_dml.update`:: + + from typing import MutableMapping + from dataclasses import dataclass + + + @dataclass + class Point: + x: int + y: int + + + class Location(Base): + __tablename__ = "location" + + id: Mapped[int] = mapped_column(primary_key=True) + x: Mapped[int] + y: Mapped[int] + + @hybrid_property + def coordinates(self) -> Point: + return Point(self.x, self.y) + + @coordinates.inplace.bulk_dml + @classmethod + def _coordinates_bulk_dml( + cls, mapping: MutableMapping[str, Any], value: Point + ) -> None: + mapping["x"] = value.x + mapping["y"] = value.y + +Additionally, a new helper :func:`_sql.from_dml_column` is added, which may be +used with the :meth:`.hybrid_property.update_expression` hook to indicate +reuse of a column expression from elsewhere in the UPDATE statement's SET +clause:: + + from sqlalchemy import from_dml_column + + + class Product(Base): + __tablename__ = "product" + + id: Mapped[int] = mapped_column(primary_key=True) + price: Mapped[float] + tax_rate: Mapped[float] + + @hybrid_property + def total_price(self) -> float: + return self.price * (1 + self.tax_rate) + + @total_price.inplace.update_expression + @classmethod + def _total_price_update_expression(cls, value: Any) -> List[Tuple[Any, Any]]: + return [(cls.price, value / (1 + from_dml_column(cls.tax_rate)))] + +In the above example, if the ``tax_rate`` column is also indicated in the +SET clause of the UPDATE, that expression will be used for the ``total_price`` +expression rather than making use of the previous value of the ``tax_rate`` +column: + +.. sourcecode:: pycon+sql + + >>> from sqlalchemy import update + >>> print(update(Product).values({Product.tax_rate: 0.08, Product.total_price: 125.00})) + {printsql}UPDATE product SET tax_rate=:tax_rate, price=(:param_1 / (:tax_rate + :param_2)) + +When the target column is omitted, :func:`_sql.from_dml_column` falls back to +using the original column expression: + +.. sourcecode:: pycon+sql + + >>> from sqlalchemy import update + >>> print(update(Product).values({Product.total_price: 125.00})) + {printsql}UPDATE product SET price=(:param_1 / (tax_rate + :param_2)) + + +.. seealso:: + + :ref:`hybrid_bulk_update` + +:ticket:`12496` + +.. _change_9832: + +New RegistryEvents System for ORM Mapping Customization +-------------------------------------------------------- + +SQLAlchemy 2.1 introduces :class:`.RegistryEvents`, providing for event +hooks that are specific to a :class:`_orm.registry`. These events include +:meth:`_orm.RegistryEvents.before_configured` and :meth:`_orm.RegistryEvents.after_configured` +to complement the same-named events that can be established on a +:class:`_orm.Mapper`, as well as :meth:`_orm.RegistryEvents.resolve_type_annotation` +that allows programmatic access to the ORM Annotated Declarative type resolution +process. Examples are provided illustrating how to define resolution schemes +for any kind of type hierarchy in an automated fashion, including :pep:`695` +type aliases. + +E.g.:: + + from typing import Any + + from sqlalchemy import event + from sqlalchemy.orm import DeclarativeBase + from sqlalchemy.orm import registry as RegistryType + from sqlalchemy.orm import TypeResolve + from sqlalchemy.types import TypeEngine + + + class Base(DeclarativeBase): + pass + + + @event.listens_for(Base, "resolve_type_annotation") + def resolve_custom_type(resolve_type: TypeResolve) -> TypeEngine[Any] | None: + if resolve_type.resolved_type is MyCustomType: + return MyCustomSQLType() + else: + return None + + + @event.listens_for(Base, "after_configured") + def after_base_configured(registry: RegistryType) -> None: + print(f"Registry {registry} fully configured") + +.. seealso:: + + :ref:`orm_declarative_resolve_type_event` - Complete documentation on using + the :meth:`.RegistryEvents.resolve_type_annotation` event + + :class:`.RegistryEvents` - Complete API reference for all registry events + +:ticket:`9832` + .. _change_10050: @@ -176,6 +282,12 @@ lambdas which do the same:: :ticket:`10050` + + + +ORM - Behavioral Changes and Improvements +========================================= + .. _change_12168: ORM Mapped Dataclasses no longer populate implicit ``default``, collection-based ``default_factory`` in ``__dict__`` @@ -337,7 +449,7 @@ default_factory for collection-based relationships internally uses DONT_SET A late add to the behavioral change brings equivalent behavior to the use of the :paramref:`_orm.relationship.default_factory` parameter with -collection-based relationships. This attribute is `documented ` +collection-based relationships. This attribute is :ref:`documented ` as being limited to exactly the collection class that's stated on the left side of the annotation, which is now enforced at mapper configuration time:: @@ -386,108 +498,53 @@ This change includes the following API changes: :ticket:`12168` -.. _change_12496: +.. _change_9809: -New Hybrid DML hook features ----------------------------- +Session autoflush behavior simplified to be unconditional +--------------------------------------------------------- -To complement the existing :meth:`.hybrid_property.update_expression` decorator, -a new decorator :meth:`.hybrid_property.bulk_dml` is added, which works -specifically with parameter dictionaries passed to :meth:`_orm.Session.execute` -when dealing with ORM-enabled :func:`_dml.insert` or :func:`_dml.update`:: +Session autoflush behavior has been simplified to unconditionally flush the +session each time an execution takes place, regardless of whether an ORM +statement or Core statement is being executed. This change eliminates the +previous conditional logic that only flushed when ORM-related statements +were detected. - from typing import MutableMapping - from dataclasses import dataclass +Previously, the session would only autoflush when executing ORM queries:: + # 2.0 behavior - autoflush only occurred for ORM statements + session.add(User(name="new user")) - @dataclass - class Point: - x: int - y: int + # This would trigger autoflush + users = session.execute(select(User)).scalars().all() + # This would NOT trigger autoflush + result = session.execute(text("SELECT * FROM users")) - class Location(Base): - __tablename__ = "location" +In 2.1, autoflush occurs for all statement executions:: - id: Mapped[int] = mapped_column(primary_key=True) - x: Mapped[int] - y: Mapped[int] + # 2.1 behavior - autoflush occurs for all executions + session.add(User(name="new user")) - @hybrid_property - def coordinates(self) -> Point: - return Point(self.x, self.y) + # Both of these now trigger autoflush + users = session.execute(select(User)).scalars().all() + result = session.execute(text("SELECT * FROM users")) - @coordinates.inplace.bulk_dml - @classmethod - def _coordinates_bulk_dml( - cls, mapping: MutableMapping[str, Any], value: Point - ) -> None: - mapping["x"] = value.x - mapping["y"] = value.y +This change provides more consistent and predictable session behavior across +all types of SQL execution. -Additionally, a new helper :func:`_sql.from_dml_column` is added, which may be -used with the :meth:`.hybrid_property.update_expression` hook to indicate -reuse of a column expression from elsewhere in the UPDATE statement's SET -clause:: +:ticket:`9809` - from sqlalchemy import from_dml_column +.. _change_12570: +New rules for None-return for ORM Composites +-------------------------------------------- - class Product(Base): - __tablename__ = "product" - - id: Mapped[int] = mapped_column(primary_key=True) - price: Mapped[float] - tax_rate: Mapped[float] - - @hybrid_property - def total_price(self) -> float: - return self.price * (1 + self.tax_rate) - - @total_price.inplace.update_expression - @classmethod - def _total_price_update_expression(cls, value: Any) -> List[Tuple[Any, Any]]: - return [(cls.price, value / (1 + from_dml_column(cls.tax_rate)))] - -In the above example, if the ``tax_rate`` column is also indicated in the -SET clause of the UPDATE, that expression will be used for the ``total_price`` -expression rather than making use of the previous value of the ``tax_rate`` -column: - -.. sourcecode:: pycon+sql - - >>> from sqlalchemy import update - >>> print(update(Product).values({Product.tax_rate: 0.08, Product.total_price: 125.00})) - {printsql}UPDATE product SET tax_rate=:tax_rate, price=(:param_1 / (:tax_rate + :param_2)) - -When the target column is omitted, :func:`_sql.from_dml_column` falls back to -using the original column expression: - -.. sourcecode:: pycon+sql - - >>> from sqlalchemy import update - >>> print(update(Product).values({Product.total_price: 125.00})) - {printsql}UPDATE product SET price=(:param_1 / (tax_rate + :param_2)) - - -.. seealso:: - - :ref:`hybrid_bulk_update` - -:ticket:`12496` - - -.. _change_12570: - -New rules for None-return for ORM Composites --------------------------------------------- - -ORM composite attributes configured using :func:`_orm.composite` can now -specify whether or not they should return ``None`` using a new parameter -:paramref:`_orm.composite.return_none_on`. By default, a composite -attribute now returns a non-None object in all cases, whereas previously -under 2.0, a ``None`` value would be returned for a pending object with -``None`` values for all composite columns. +ORM composite attributes configured using :func:`_orm.composite` can now +specify whether or not they should return ``None`` using a new parameter +:paramref:`_orm.composite.return_none_on`. By default, a composite +attribute now returns a non-None object in all cases, whereas previously +under 2.0, a ``None`` value would be returned for a pending object with +``None`` values for all composite columns. Given a composite mapping:: @@ -592,59 +649,9 @@ Annotated Declarative setting from taking place. :ticket:`12570` -.. _change_9832: - -New RegistryEvents System for ORM Mapping Customization --------------------------------------------------------- - -SQLAlchemy 2.1 introduces :class:`.RegistryEvents`, providing for event -hooks that are specific to a :class:`_orm.registry`. These events include -:meth:`_orm.RegistryEvents.before_configured` and :meth:`_orm.RegistryEvents.after_configured` -to complement the same-named events that can be established on a -:class:`_orm.Mapper`, as well as :meth:`_orm.RegistryEvents.resolve_type_annotation` -that allows programmatic access to the ORM Annotated Declarative type resolution -process. Examples are provided illustrating how to define resolution schemes -for any kind of type hierarchy in an automated fashion, including :pep:`695` -type aliases. - -E.g.:: - - from typing import Any - - from sqlalchemy import event - from sqlalchemy.orm import DeclarativeBase - from sqlalchemy.orm import registry as RegistryType - from sqlalchemy.orm import TypeResolve - from sqlalchemy.types import TypeEngine - - - class Base(DeclarativeBase): - pass - - - @event.listens_for(Base, "resolve_type_annotation") - def resolve_custom_type(resolve_type: TypeResolve) -> TypeEngine[Any] | None: - if resolve_type.resolved_type is MyCustomType: - return MyCustomSQLType() - else: - return None - - - @event.listens_for(Base, "after_configured") - def after_base_configured(registry: RegistryType) -> None: - print(f"Registry {registry} fully configured") - -.. seealso:: - - :ref:`orm_declarative_resolve_type_event` - Complete documentation on using - the :meth:`.RegistryEvents.resolve_type_annotation` event - - :class:`.RegistryEvents` - Complete API reference for all registry events - -:ticket:`9832` -New Features and Improvements - Core -===================================== +Core - New Features +=================== .. _change_12548: @@ -710,274 +717,181 @@ literal values will share the same cache key, providing optimal performance. :ticket:`12548` -.. _change_10635: +.. _change_4950: -``Row`` now represents individual column types directly without ``Tuple`` --------------------------------------------------------------------------- +CREATE VIEW and CREATE TABLE AS SELECT Support +---------------------------------------------- -SQLAlchemy 2.0 implemented a broad array of :pep:`484` typing throughout -all components, including a new ability for row-returning statements such -as :func:`_sql.select` to maintain track of individual column types, which -were then passed through the execution phase onto the :class:`_engine.Result` -object and then to the individual :class:`_engine.Row` objects. Described -at :ref:`change_result_typing_20`, this approach solved several issues -with statement / row typing, but some remained unsolvable. In 2.1, one -of those issues, that the individual column types needed to be packaged -into a ``typing.Tuple``, is now resolved using new :pep:`646` integration, -which allows for tuple-like types that are not actually typed as ``Tuple``. +SQLAlchemy 2.1 adds support for the SQL ``CREATE VIEW`` and +``CREATE TABLE ... AS SELECT`` constructs, as well as the ``SELECT ... INTO`` +variant for selected backends. Both DDL statements generate a table +or table-like construct based on the structure and rows represented by a +SELECT statement. The constructs are available via the :class:`.CreateView` +and :class:`_schema.CreateTableAs` DDL classes, as well as the +:meth:`_sql.SelectBase.into` convenience method. -In SQLAlchemy 2.0, a statement such as:: +Both constructs work in exactly the same way, including that a :class:`.Table` +object is automatically generated from a given :class:`.Select`. DDL +can then be emitted by executing the construct directly or by allowing the +:meth:`.MetaData.create_all` or :meth:`.Table.create` sequences to emit the +correct DDL. - stmt = select(column("x", Integer), column("y", String)) +E.g. using :class:`.CreateView`:: -Would be typed as:: + >>> from sqlalchemy import Table, Column, Integer, String, MetaData + >>> from sqlalchemy import CreateView, select + >>> + >>> metadata_obj = MetaData() + >>> user_table = Table( + ... "user_account", + ... metadata_obj, + ... Column("id", Integer, primary_key=True), + ... Column("name", String(30)), + ... Column("fullname", String), + ... ) + >>> view = CreateView( + ... select(user_table).where(user_table.c.name.like("%spongebob%")), + ... "spongebob_view", + ... metadata=metadata_obj, + ... ) - Select[Tuple[int, str]] -In 2.1, it's now typed as:: +The above ``CreateView`` construct will emit CREATE VIEW when executed directly, +or when a DDL create operation is run. When using :meth:`.MetaData.create_all`, +the view is created after all dependent tables have been created: - Select[int, str] +.. sourcecode:: pycon+sql -When executing ``stmt``, the :class:`_engine.Result` and :class:`_engine.Row` -objects will be typed as ``Result[int, str]`` and ``Row[int, str]``, respectively. -The prior workaround using :attr:`_engine.Row._t` to type as a real ``Tuple`` -is no longer needed and projects can migrate off this pattern. + >>> from sqlalchemy import create_engine + >>> e = create_engine("sqlite://", echo=True) + >>> metadata_obj.create_all(e) + {opensql}BEGIN (implicit) -Mypy users will need to make use of **Mypy 1.7 or greater** for pep-646 -integration to be available. + CREATE TABLE user_account ( + id INTEGER NOT NULL, + name VARCHAR(30), + fullname VARCHAR, + PRIMARY KEY (id) + ) -Limitations -^^^^^^^^^^^ + CREATE VIEW spongebob_view AS + SELECT user_account.id, user_account.name, user_account.fullname + FROM user_account + WHERE user_account.name LIKE '%spongebob%' -Not yet solved by pep-646 or any other pep is the ability for an arbitrary -number of expressions within :class:`_sql.Select` and others to be mapped to -row objects, without stating each argument position explicitly within typing -annotations. To work around this issue, SQLAlchemy makes use of automated -"stub generation" tools to generate hardcoded mappings of different numbers of -positional arguments to constructs like :func:`_sql.select` to resolve to -individual ``Unpack[]`` expressions (in SQLAlchemy 2.0, this generation -produced ``Tuple[]`` annotations instead). This means that there are arbitrary -limits on how many specific column expressions will be typed within the -:class:`_engine.Row` object, without restoring to ``Any`` for remaining -expressions; for :func:`_sql.select`, it's currently ten expressions, and -for DML expressions like :func:`_dml.insert` that use :meth:`_dml.Insert.returning`, -it's eight. If and when a new pep that provides a ``Map`` operator -to pep-646 is proposed, this limitation can be lifted. [1]_ Originally, it was -mistakenly assumed that this limitation prevented pep-646 from being usable at all, -however, the ``Unpack`` construct does in fact replace everything that -was done using ``Tuple`` in 2.0. + COMMIT -An additional limitation for which there is no proposed solution is that -there's no way for the name-based attributes on :class:`_engine.Row` to be -automatically typed, so these continue to be typed as ``Any`` (e.g. ``row.x`` -and ``row.y`` for the above example). With current language features, -this could only be fixed by having an explicit class-based construct that -allows one to compose an explicit :class:`_engine.Row` with explicit fields -up front, which would be verbose and not automatic. +The view is usable in SQL expressions via the :attr:`.CreateView.table` attribute: -.. [1] https://github.com/python/typing/discussions/1001#discussioncomment-1897813 +.. sourcecode:: pycon+sql -:ticket:`10635` + >>> with e.connect() as conn: + ... conn.execute(select(view.table)) + {opensql}BEGIN (implicit) + SELECT spongebob_view.id, spongebob_view.name, spongebob_view.fullname + FROM spongebob_view + + ROLLBACK -.. _change_13085: +:class:`_schema.CreateTableAs` works in the same way, emitting ``CREATE TABLE AS``:: -Better type checker integration for Core froms, like Table ----------------------------------------------------------- + >>> from sqlalchemy import CreateTableAs + >>> select_stmt = select(user_table.c.id, user_table.c.name).where( + ... user_table.c.name == "squidward" + ... ) + >>> create_table_as = CreateTableAs(select_stmt, "squidward_users") -SQLAlchemy 2.1 changes :class:`_schema.Table`, along with most -:class:`_sql.FromClause` subclasses, to be generic on the column collection, -providing the option for better static type checking support. -By declaring the columns using a :class:`_schema.TypedColumns` subclass and -providing it to the :class:`_schema.Table` instance, IDEs and type checkers -can infer the exact types of columns when accessing them via the -:attr:`_schema.Table.c` attribute, enabling better autocomplete and type validation. +In this case, :class:`.CreateTableAs` was not given a :class:`.MetaData` collection. +While a :class:`.MetaData` collection will be created automatically in this case, +the actual ``CREATE TABLE AS`` statement can also be generated by directly +executing the object: -Example usage:: +.. sourcecode:: pycon+sql - from sqlalchemy import Table, TypedColumns, Column, Integer - from sqlalchemy import MetaData, Named, SmallInteger, select + >>> with e.begin() as conn: + ... conn.execute(create_table_as) + {opensql}BEGIN (implicit) + CREATE TABLE squidward_users AS SELECT user_account.id, user_account.name + FROM user_account + WHERE user_account.name = 'squidward' + COMMIT +Like before, the :class:`.Table` is accessible from :attr:`.CreateTableAs.table`: - class user_cols(TypedColumns): - # the name will be set to ``id``, type is inferred as Column[int] - # from the Integer SQL type. - id = Column(Integer, primary_key=True) +.. sourcecode:: pycon+sql - # not null String column is generated - name: Named[str] - - # nullable Integer column, the SQL type is manually set SmallInteger - age: Named[int | None] = Column(SmallInteger) - - # optional, used to infer the select types when selecting the table - __row_pos__: tuple[int, str, int | None] - - - metadata = MetaData() - user = Table("user", metadata, user_cols) - - # Type checkers now understand the column types when selecting single columns - stmt = select(user.c.id, user.c.name) # Inferred as Select[int, str] - - # and also when selecting the whole table, when __row_pos__ is present - stmt = select(user) # Inferred as Select[int, str, int | None] - -The optional :attr:`sqlalchemy.sql._annotated_cols.HasRowPos.__row_pos__` annotation -is used to infer the types of a select when selecting the table directly. - -Columns can be declared in :class:`.TypedColumns` subclasses by instantiating -them directly, like ``id``, by using only a type annotations, like ``name``, letting -the :class:`_schema.Table` infer SQL type and nullability, or by mixing the two, like ``age``, -to provide explicit column options while inferring nullability and/or SQL type. - -Other :class:`_sql.FromClause`, like :class:`_sql.Join`, :class:`_sql.CTE`, etc, can be made -generic using the :meth:`_sql.FromClause.with_cols` method:: - - # using with_cols the ``c`` collection of the cte has typed columns - cte = user.select().cte().with_cols(user_cols) - -ORM Integration -^^^^^^^^^^^^^^^ - -This functionality also offers some integration with the ORM, by using -:class:`_orm.MappedColumn` annotated attributes in the ORM model and -:func:`_orm.as_typed_table` to get an annotated :class:`_sql.FromClause`:: - - from sqlalchemy import TypedColumns - from sqlalchemy.orm import DeclarativeBase, mapped_column - from sqlalchemy.orm import MappedColumn, as_typed_table - - - class Base(DeclarativeBase): - pass - - - class A(Base): - __tablename__ = "a" - __typed_cols__: "a_cols" - - id: MappedColumn[int] = mapped_column(primary_key=True) - data: MappedColumn[str] - - - class a_cols(A, TypedColumns): - pass + >>> with e.connect() as conn: + ... conn.execute(select(create_table_as.table)) + {opensql}BEGIN (implicit) + SELECT squidward_users.id, squidward_users.name + FROM squidward_users + + ROLLBACK +.. seealso:: - # table_a is annotated as FromClause[a_cols], and is just A.__table__ - table_a = as_typed_table(A) + :ref:`metadata_create_view` - in :ref:`metadata_toplevel` -For proper typing integration :class:`_orm.MappedColumn` should be used -to annotate the single columns, since it's a more specific annotation than -the usual :class:`_orm.Mapped` used for ORM attributes. + :ref:`metadata_create_table_as` - in :ref:`metadata_toplevel` -:ticket:`13085` + :class:`_schema.CreateView` - DDL construct for CREATE VIEW -.. _change_8601: + :class:`_schema.CreateTableAs` - DDL construct for CREATE TABLE AS -``filter_by()`` now searches across all FROM clause entities -------------------------------------------------------------- + :meth:`_sql.SelectBase.into` - convenience method on SELECT and UNION + statements -The :meth:`_sql.Select.filter_by` method, available for both Core -:class:`_sql.Select` objects and ORM-enabled select statements, has been -enhanced to search for attribute names across **all entities present in the -FROM clause** of the statement, rather than only looking at the last joined -entity or first FROM entity. +:ticket:`4950` -This resolves a long-standing issue where the behavior of -:meth:`_sql.Select.filter_by` was sensitive to the order of operations. For -example, calling :meth:`_sql.Select.with_only_columns` after setting up joins -would reset which entity was searched, causing :meth:`_sql.Select.filter_by` -to fail even though the joined entity was still part of the FROM clause. +.. _change_8130: -Example - previously failing case now works:: +Explicit USING support for DELETE (MySQL, PostgreSQL) +------------------------------------------------------ - from sqlalchemy import select, MetaData, Table, Column, Integer, String, ForeignKey +The :meth:`_sql.Delete.using` method has been added, allowing explicit +``USING`` expressions to be specified in DELETE statements. This is +useful for backend-specific multiple-table DELETE forms where the secondary +FROM clause needs to be stated explicitly, such as joined DELETE on +MySQL/MariaDB and PostgreSQL. - metadata = MetaData() +Previously, multi-table DELETE was supported by inferring extra FROM entries +from the WHERE clause, which works for simple cases. The new +:meth:`_sql.Delete.using` method allows more complex expressions such as +explicit joins to be stated:: - users = Table( - "users", - metadata, - Column("id", Integer, primary_key=True), - Column("name", String(50)), - ) + from sqlalchemy import delete, table, column - addresses = Table( - "addresses", - metadata, - Column("id", Integer, primary_key=True), - Column("user_id", ForeignKey("users.id")), - Column("email", String(100)), - ) + user_table = table("users", column("id"), column("name")) + address_table = table("addresses", column("id"), column("user_id"), column("email")) - # This now works in 2.1 - previously raised an error stmt = ( - select(users) - .join(addresses) - .with_only_columns(users.c.id) # changes selected columns - .filter_by(email="foo@bar.com") # searches addresses table successfully + delete(user_table) + .using( + user_table.outerjoin( + address_table, + user_table.c.id == address_table.c.user_id, + ) + ) + .where(address_table.c.email == "patrick@aol.com") ) -Ambiguous Attribute Names -^^^^^^^^^^^^^^^^^^^^^^^^^^ - -When an attribute name exists in more than one entity in the FROM clause, -:meth:`_sql.Select.filter_by` now raises :class:`_exc.AmbiguousColumnError`, -indicating that :meth:`_sql.Select.filter` should be used instead with -explicit column references:: - - # Both users and addresses have 'id' column - stmt = select(users).join(addresses) - - # Raises AmbiguousColumnError in 2.1 - stmt = stmt.filter_by(id=5) - - # Use filter() with explicit qualification instead - stmt = stmt.filter(addresses.c.id == 5) - -The same behavior applies to ORM entities:: - - from sqlalchemy.orm import Session - - stmt = select(User).join(Address) - - # If both User and Address have an 'id' attribute, this raises - # AmbiguousColumnError - stmt = stmt.filter_by(id=5) - - # Use filter() with explicit entity qualification - stmt = stmt.filter(Address.id == 5) - -Legacy Query Use is Unchanged -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - -The change to :meth:`.Select.filter_by` has **not** been applied to the -:meth:`.Query.filter_by` method of :class:`.Query`; as :class:`.Query` is -a legacy API, its behavior hasn't changed. +On MySQL/MariaDB, the above renders as: -Migration Path -^^^^^^^^^^^^^^ +.. sourcecode:: sql -Code that was previously working should continue to work without modification -in the vast majority of cases. The only breaking changes would be: + DELETE FROM users USING users LEFT OUTER JOIN addresses + ON users.id = addresses.user_id + WHERE addresses.email = %s -1. **Ambiguous names that were previously accepted**: If your code had joins - where :meth:`_sql.Select.filter_by` happened to use an ambiguous column - name but it worked because it searched only one entity, this will now - raise :class:`_exc.AmbiguousColumnError`. The fix is to use - :meth:`_sql.Select.filter` with explicit column qualification. +On PostgreSQL, a similar form is rendered using the PostgreSQL-specific +``DELETE .. USING`` syntax. -2. **Different entity selection**: In rare cases where the old behavior of - selecting the "last joined" or "first FROM" entity was being relied upon, - :meth:`_sql.Select.filter_by` might now find the attribute in a different - entity. Review any :meth:`_sql.Select.filter_by` calls in complex - multi-entity queries. +.. seealso:: -It's hoped that in most cases, this change will make -:meth:`_sql.Select.filter_by` more intuitive to use. + :ref:`tutorial_multi_table_deletes` - updated tutorial section for + multi-table deletes -:ticket:`8601` +:ticket:`8130` .. _change_new_syntax_ext: @@ -1041,7 +955,7 @@ This can then be used in a select using the :meth:`.Select.ext` method: ... .select_from(sa.table("tbl")) ... .ext(into_outfile("myfile.txt")) ... ) - >>> print(sql) + >>> print(stmt) {printsql}SELECT a FROM tbl INTO OUTFILE 'myfile.txt'{stop} @@ -1058,57 +972,144 @@ that supersede the previous implementations. :ticket:`12195` :ticket:`12342` +.. _change_13085: -.. _change_11234: +New type checker integration structures for Core FROM clauses, like Table +------------------------------------------------------------------------- -URL stringify and parse now supports URL escaping for the "database" portion ----------------------------------------------------------------------------- +SQLAlchemy 2.1 changes :class:`_schema.Table`, along with most +:class:`_sql.FromClause` subclasses, to be generic on the column collection, +providing the option for better static type checking support. +By declaring the columns using a :class:`_schema.TypedColumns` subclass and +providing it to the :class:`_schema.Table` instance, IDEs and type checkers +can infer the exact types of columns when accessing them via the +:attr:`_schema.Table.c` attribute, enabling better autocomplete and type validation. -A URL that includes URL-escaped characters in the database portion will -now parse with conversion of those escaped characters:: +Example usage:: - >>> from sqlalchemy import make_url - >>> u = make_url("driver://user:pass@host/database%3Fname") - >>> u.database - 'database?name' + from sqlalchemy import Table, TypedColumns, Column, Integer + from sqlalchemy import MetaData, Named, SmallInteger, select -Previously, such characters would not be unescaped:: - >>> # pre-2.1 behavior - >>> from sqlalchemy import make_url - >>> u = make_url("driver://user:pass@host/database%3Fname") - >>> u.database - 'database%3Fname' + class user_cols(TypedColumns): + # the name will be set to ``id``, type is inferred as Column[int] + # from the Integer SQL type. + id = Column(Integer, primary_key=True) -This change also applies to the stringify side; most special characters in -the database name will be URL escaped, omitting a few such as plus signs and -slashes:: + # not null String column is generated + name: Named[str] - >>> from sqlalchemy import URL - >>> u = URL.create("driver", database="a?b=c") - >>> str(u) - 'driver:///a%3Fb%3Dc' + # nullable Integer column, the SQL type is manually set SmallInteger + age: Named[int | None] = Column(SmallInteger) -Where the above URL correctly round-trips to itself:: + # optional, used to infer the select types when selecting the table + __row_pos__: tuple[int, str, int | None] - >>> make_url(str(u)) - driver:///a%3Fb%3Dc - >>> make_url(str(u)).database == u.database - True + metadata = MetaData() + user = Table("user", metadata, user_cols) -Whereas previously, special characters applied programmatically would not -be escaped in the result, leading to a URL that does not represent the -original database portion. Below, `b=c` is part of the query string and -not the database portion:: + # Type checkers now understand the column types when selecting single columns + stmt = select(user.c.id, user.c.name) # Inferred as Select[int, str] - >>> # pre-2.1 behavior - >>> from sqlalchemy import URL - >>> u = URL.create("driver", database="a?b=c") - >>> str(u) - 'driver:///a?b=c' + # and also when selecting the whole table, when __row_pos__ is present + stmt = select(user) # Inferred as Select[int, str, int | None] + +The optional :attr:`sqlalchemy.sql._annotated_cols.HasRowPos.__row_pos__` annotation +is used to infer the types of a select when selecting the table directly. + +Columns can be declared in :class:`.TypedColumns` subclasses by instantiating +them directly, like ``id``, by using only a type annotation, like ``name``, letting +the :class:`_schema.Table` infer SQL type and nullability, or by mixing the two, like ``age``, +to provide explicit column options while inferring nullability and/or SQL type. + +Other :class:`_sql.FromClause`, like :class:`_sql.Join`, :class:`_sql.CTE`, etc, can be made +generic using the :meth:`_sql.FromClause.with_cols` method:: + + # using with_cols the ``c`` collection of the cte has typed columns + cte = user.select().cte().with_cols(user_cols) + +ORM Integration +^^^^^^^^^^^^^^^ + +This functionality also offers some integration with the ORM, by using +:class:`_orm.MappedColumn` annotated attributes in the ORM model and +:func:`_orm.as_typed_table` to get an annotated :class:`_sql.FromClause`:: + + from sqlalchemy import TypedColumns + from sqlalchemy.orm import DeclarativeBase, mapped_column + from sqlalchemy.orm import MappedColumn, as_typed_table + + + class Base(DeclarativeBase): + pass + + + class A(Base): + __tablename__ = "a" + __typed_cols__: "a_cols" + + id: MappedColumn[int] = mapped_column(primary_key=True) + data: MappedColumn[str] + + + class a_cols(A, TypedColumns): + pass + + + # table_a is annotated as FromClause[a_cols], and is just A.__table__ + table_a = as_typed_table(A) + +For proper typing integration :class:`_orm.MappedColumn` should be used +to annotate the single columns, since it's a more specific annotation than +the usual :class:`_orm.Mapped` used for ORM attributes. + +:ticket:`13085` + +.. _change_12596: + +Non-integer RANGE window frame clauses now supported +----------------------------------------------------- + +The :func:`_sql.over` clause now supports non-integer values in the +:paramref:`_sql.over.range_` parameter through the new :class:`_sql.FrameClause` +construct. Previously, only integer values were allowed in RANGE clauses, which +limited their use to integer-based ordering columns. + +With this change, applications can now use RANGE with other data types such +as floating-point numbers, dates, and intervals. The new :class:`_sql.FrameClause` +construct provides explicit control over frame boundaries using the +:class:`_sql.FrameClauseType` enum:: + + from datetime import timedelta + from sqlalchemy import FrameClause, FrameClauseType + + # Example: date-based RANGE with a 7-day window + func.sum(my_table.c.amount).over( + order_by=my_table.c.date, + range_=FrameClause( + start=timedelta(days=7), + end=None, + start_frame_type=FrameClauseType.PRECEDING, + end_frame_type=FrameClauseType.CURRENT, + ), + ) + +For backwards compatibility, the traditional tuple-based syntax continues to +work with integer values:: + + # This continues to work unchanged + func.row_number().over(order_by=table.c.col, range_=(None, 10)) + +However, attempting to use non-integer values in the tuple syntax will now +raise an error, directing users to use :class:`_sql.FrameClause` instead. + +:ticket:`12596` + + +Core - Behavioral Changes and Improvements +========================================== -:ticket:`11234` .. _change_7066: @@ -1154,130 +1155,276 @@ period but will emit deprecation warnings. :ticket:`7066` +.. _change_10635: -.. _change_4950: +``Row`` now represents individual column types directly without ``Tuple`` +-------------------------------------------------------------------------- -CREATE VIEW and CREATE TABLE AS SELECT Support ----------------------------------------------- +SQLAlchemy 2.0 implemented a broad array of :pep:`484` typing throughout +all components, including a new ability for row-returning statements such +as :func:`_sql.select` to maintain track of individual column types, which +were then passed through the execution phase onto the :class:`_engine.Result` +object and then to the individual :class:`_engine.Row` objects. Described +at :ref:`change_result_typing_20`, this approach solved several issues +with statement / row typing, but some remained unsolvable. In 2.1, one +of those issues, that the individual column types needed to be packaged +into a ``typing.Tuple``, is now resolved using new :pep:`646` integration, +which allows for tuple-like types that are not actually typed as ``Tuple``. -SQLAlchemy 2.1 adds support for the SQL ``CREATE VIEW`` and -``CREATE TABLE ... AS SELECT`` constructs, as well as the ``SELECT ... INTO`` -variant for selected backends. Both DDL statements generate a table -or table-like construct based on the structure and rows represented by a -SELECT statement. The constructs are available via the :class:`.CreateView` -and :class:`_schema.CreateTableAs` DDL classes, as well as the -:meth:`_sql.SelectBase.into` convenience method. +In SQLAlchemy 2.0, a statement such as:: -Both constructs work in exactly the same way, including that a :class:`.Table` -object is automatically generated from a given :class:`.Select`. DDL -can then be emitted by executing the construct directly or by allowing the -:meth:`.MetaData.create_all` or :meth:`.Table.create` sequences to emit the -correct DDL. + stmt = select(column("x", Integer), column("y", String)) -E.g. using :class:`.CreateView`:: +Would be typed as:: - >>> from sqlalchemy import Table, Column, Integer, String, MetaData - >>> from sqlalchemy import CreateView, select - >>> - >>> metadata_obj = MetaData() - >>> user_table = Table( - ... "user_account", - ... metadata_obj, - ... Column("id", Integer, primary_key=True), - ... Column("name", String(30)), - ... Column("fullname", String), - ... ) - >>> view = CreateView( - ... select(user_table).where(user_table.c.name.like("%spongebob%")), - ... "spongebob_view", - ... metadata=metadata_obj, - ... ) + Select[Tuple[int, str]] + +In 2.1, it's now typed as:: + + Select[int, str] + +When executing ``stmt``, the :class:`_engine.Result` and :class:`_engine.Row` +objects will be typed as ``Result[int, str]`` and ``Row[int, str]``, respectively. +The prior workaround using :attr:`_engine.Row._t` to type as a real ``Tuple`` +is no longer needed and projects can migrate off this pattern. + +Mypy users will need to make use of **Mypy 1.7 or greater** for pep-646 +integration to be available. + +Limitations +^^^^^^^^^^^ + +Not yet solved by pep-646 or any other pep is the ability for an arbitrary +number of expressions within :class:`_sql.Select` and others to be mapped to +row objects, without stating each argument position explicitly within typing +annotations. To work around this issue, SQLAlchemy makes use of automated +"stub generation" tools to generate hardcoded mappings of different numbers of +positional arguments to constructs like :func:`_sql.select` to resolve to +individual ``Unpack[]`` expressions (in SQLAlchemy 2.0, this generation +produced ``Tuple[]`` annotations instead). This means that there are arbitrary +limits on how many specific column expressions will be typed within the +:class:`_engine.Row` object, without resorting to ``Any`` for remaining +expressions; for :func:`_sql.select`, it's currently ten expressions, and +for DML expressions like :func:`_dml.insert` that use :meth:`_dml.Insert.returning`, +it's eight. If and when a new pep that provides a ``Map`` operator +to pep-646 is proposed, this limitation can be lifted. [1]_ Originally, it was +mistakenly assumed that this limitation prevented pep-646 from being usable at all, +however, the ``Unpack`` construct does in fact replace everything that +was done using ``Tuple`` in 2.0. + +An additional limitation for which there is no proposed solution is that +there's no way for the name-based attributes on :class:`_engine.Row` to be +automatically typed, so these continue to be typed as ``Any`` (e.g. ``row.x`` +and ``row.y`` for the above example). With current language features, +this could only be fixed by having an explicit class-based construct that +allows one to compose an explicit :class:`_engine.Row` with explicit fields +up front, which would be verbose and not automatic. + +.. [1] https://github.com/python/typing/discussions/1001#discussioncomment-1897813 + +:ticket:`10635` + + +.. _change_8601: + +``filter_by()`` now searches across all FROM clause entities +------------------------------------------------------------- + +The :meth:`_sql.Select.filter_by` method, available for both Core +:class:`_sql.Select` objects and ORM-enabled select statements, has been +enhanced to search for attribute names across **all entities present in the +FROM clause** of the statement, rather than only looking at the last joined +entity or first FROM entity. + +This resolves a long-standing issue where the behavior of +:meth:`_sql.Select.filter_by` was sensitive to the order of operations. For +example, calling :meth:`_sql.Select.with_only_columns` after setting up joins +would reset which entity was searched, causing :meth:`_sql.Select.filter_by` +to fail even though the joined entity was still part of the FROM clause. + +Example - previously failing case now works:: + + from sqlalchemy import select, MetaData, Table, Column, Integer, String, ForeignKey + + metadata = MetaData() + + users = Table( + "users", + metadata, + Column("id", Integer, primary_key=True), + Column("name", String(50)), + ) + + addresses = Table( + "addresses", + metadata, + Column("id", Integer, primary_key=True), + Column("user_id", ForeignKey("users.id")), + Column("email", String(100)), + ) + + # This now works in 2.1 - previously raised an error + stmt = ( + select(users) + .join(addresses) + .with_only_columns(users.c.id) # changes selected columns + .filter_by(email="foo@bar.com") # searches addresses table successfully + ) + +Ambiguous Attribute Names +^^^^^^^^^^^^^^^^^^^^^^^^^^ + +When an attribute name exists in more than one entity in the FROM clause, +:meth:`_sql.Select.filter_by` now raises :class:`_exc.AmbiguousColumnError`, +indicating that :meth:`_sql.Select.filter` should be used instead with +explicit column references:: + + # Both users and addresses have 'id' column + stmt = select(users).join(addresses) + + # Raises AmbiguousColumnError in 2.1 + stmt = stmt.filter_by(id=5) + + # Use filter() with explicit qualification instead + stmt = stmt.filter(addresses.c.id == 5) +The same behavior applies to ORM entities:: -The above ``CreateView`` construct will emit CREATE VIEW when executed directly, -or when a DDL create operation is run. When using :meth:`.MetaData.create_all`, -the view is created after all dependent tables have been created: + from sqlalchemy.orm import Session -.. sourcecode:: pycon+sql + stmt = select(User).join(Address) - >>> from sqlalchemy import create_engine - >>> e = create_engine("sqlite://", echo=True) - >>> metadata_obj.create_all(e) - {opensql}BEGIN (implicit) + # If both User and Address have an 'id' attribute, this raises + # AmbiguousColumnError + stmt = stmt.filter_by(id=5) - CREATE TABLE user_account ( - id INTEGER NOT NULL, - name VARCHAR(30), - fullname VARCHAR, - PRIMARY KEY (id) - ) + # Use filter() with explicit entity qualification + stmt = stmt.filter(Address.id == 5) - CREATE VIEW spongebob_view AS - SELECT user_account.id, user_account.name, user_account.fullname - FROM user_account - WHERE user_account.name LIKE '%spongebob%' +Legacy Query Use is Unchanged +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ - COMMIT +The change to :meth:`.Select.filter_by` has **not** been applied to the +:meth:`.Query.filter_by` method of :class:`.Query`; as :class:`.Query` is +a legacy API, its behavior hasn't changed. -The view is usable in SQL expressions via the :attr:`.CreateView.table` attribute: +Migration Path +^^^^^^^^^^^^^^ -.. sourcecode:: pycon+sql +Code that was previously working should continue to work without modification +in the vast majority of cases. The only breaking changes would be: - >>> with e.connect() as conn: - ... conn.execute(select(view.table)) - {opensql}BEGIN (implicit) - SELECT spongebob_view.id, spongebob_view.name, spongebob_view.fullname - FROM spongebob_view - - ROLLBACK +1. **Ambiguous names that were previously accepted**: If your code had joins + where :meth:`_sql.Select.filter_by` happened to use an ambiguous column + name but it worked because it searched only one entity, this will now + raise :class:`_exc.AmbiguousColumnError`. The fix is to use + :meth:`_sql.Select.filter` with explicit column qualification. -:class:`_schema.CreateTableAs` works in the same way, emitting ``CREATE TABLE AS``:: +2. **Different entity selection**: In rare cases where the old behavior of + selecting the "last joined" or "first FROM" entity was being relied upon, + :meth:`_sql.Select.filter_by` might now find the attribute in a different + entity. Review any :meth:`_sql.Select.filter_by` calls in complex + multi-entity queries. - >>> from sqlalchemy import CreateTableAs - >>> select_stmt = select(users.c.id, users.c.name).where(users.c.name == "squidward") - >>> create_table_as = CreateTableAs(select_stmt, "squidward_users") +In most cases, this change is expected to make +:meth:`_sql.Select.filter_by` more intuitive to use. -In this case, :class:`.CreateTableAs` was not given a :class:`.MetaData` collection. -While a :class:`.MetaData` collection will be created automatically in this case, -the actual ``CREATE TABLE AS`` statement can also be generated by directly -executing the object: +:ticket:`8601` -.. sourcecode:: pycon+sql +.. _change_13381: + +Error handling extended to DBAPI cursor operations in before_cursor_execute/after_cursor_execute event hooks +------------------------------------------------------------------------------------------------------------ + +Applications that use the +:meth:`_events.ConnectionEvents.before_cursor_execute` or +:meth:`_events.ConnectionEvents.after_cursor_execute` event hooks will see +two behavioral changes. Both changes apply specifically to the case where +hook code operates directly on the raw DBAPI cursor or DBAPI connection +object — for example, calling ``cursor.execute()`` or performing other raw +DBAPI operations from inside the hook. When the SQLAlchemy +:class:`_engine.Connection` is used instead, its own error handling already +applies and is unaffected by this change. + +When an exception escapes from one of these hooks (that is, as it propagates +outward toward the calling code such as :meth:`_engine.Connection.execute`), +SQLAlchemy's error handling machinery now intercepts it. This produces two +effects: + +* **Connection invalidation now occurs correctly.** Previously, any exception + that escaped these hooks — including ``BaseException`` subclasses such as + ``asyncio.CancelledError``, ``KeyboardInterrupt``, and ``SystemExit`` — + bypassed SQLAlchemy's error handling path entirely. The connection was not + invalidated and the pool was not notified, potentially leaving a broken + connection checked back into the pool. + +* **Raw DBAPI errors are now wrapped as SQLAlchemy exceptions.** A DBAPI + exception raised by a raw DBAPI operation inside the hook, which previously + propagated as a bare DBAPI exception, is now wrapped in a + :class:`_exc.DBAPIError` subclass, consistent with errors raised during + normal statement execution. + +It is important to note that this new level of exception handling, which +applies **only** to direct operations on the DBAPI cursor passed into the event +hook, is applied **outside** the body of the event hook itself — it takes +effect as the exception propagates outward from the hook back through +SQLAlchemy's execution machinery. This is in contrast to the case where a +:class:`_engine.Connection` is used from within the hook body, where exception +processing occurs locally at the point of that :class:`_engine.Connection` +call, before the exception ever leaves the hook. + +:ticket:`13381` - >>> with e.begin() as conn: - ... conn.execute(create_table_as) - {opensql}BEGIN (implicit) - CREATE TABLE squidward_users AS SELECT user_account.id, user_account.name - FROM user_account - WHERE user_account.name = 'squidward' - COMMIT +.. _change_11234: -Like before, the :class:`.Table` is accessible from :attr:`.CreateTableAs.table`: +URL stringify and parse now supports URL escaping for the "database" portion +---------------------------------------------------------------------------- -.. sourcecode:: pycon+sql +A URL that includes URL-escaped characters in the database portion will +now parse with conversion of those escaped characters:: - >>> with e.connect() as conn: - ... conn.execute(select(create_table_as.table)) - {opensql}BEGIN (implicit) - SELECT squidward_users.id, squidward_users.name - FROM squidward_users - - ROLLBACK + >>> from sqlalchemy import make_url + >>> u = make_url("driver://user:pass@host/database%3Fname") + >>> u.database + 'database?name' -.. seealso:: +Previously, such characters would not be unescaped:: - :ref:`metadata_create_view` - in :ref:`metadata_toplevel` + >>> # pre-2.1 behavior + >>> from sqlalchemy import make_url + >>> u = make_url("driver://user:pass@host/database%3Fname") + >>> u.database + 'database%3Fname' - :ref:`metadata_create_table_as` - in :ref:`metadata_toplevel` +This change also applies to the stringify side; most special characters in +the database name will be URL escaped, omitting a few such as plus signs and +slashes:: - :class:`_schema.CreateView` - DDL construct for CREATE VIEW + >>> from sqlalchemy import URL + >>> u = URL.create("driver", database="a?b=c") + >>> str(u) + 'driver:///a%3Fb%3Dc' - :class:`_schema.CreateTableAs` - DDL construct for CREATE TABLE AS +Where the above URL correctly round-trips to itself:: - :meth:`_sql.SelectBase.into` - convenience method on SELECT and UNION - statements + >>> make_url(str(u)) + driver:///a%3Fb%3Dc + >>> make_url(str(u)).database == u.database + True -:ticket:`4950` + +Whereas previously, special characters applied programmatically would not +be escaped in the result, leading to a URL that does not represent the +original database portion. Below, `b=c` is part of the query string and +not the database portion:: + + >>> # pre-2.1 behavior + >>> from sqlalchemy import URL + >>> u = URL.create("driver", database="a?b=c") + >>> str(u) + 'driver:///a?b=c' + +:ticket:`11234` .. _change_12736: @@ -1373,46 +1520,6 @@ will make use of the operator classes declared by the "impl" type. :ticket:`12736` -.. _change_12596: - -Non-integer RANGE window frame clauses now supported ------------------------------------------------------ - -The :func:`_sql.over` clause now supports non-integer values in the -:paramref:`_sql.over.range_` parameter through the new :class:`_sql.FrameClause` -construct. Previously, only integer values were allowed in RANGE clauses, which -limited their use to integer-based ordering columns. - -With this change, applications can now use RANGE with other data types such -as floating-point numbers, dates, and intervals. The new :class:`_sql.FrameClause` -construct provides explicit control over frame boundaries using the -:class:`_sql.FrameClauseType` enum:: - - from datetime import timedelta - from sqlalchemy import FrameClause, FrameClauseType - - # Example: date-based RANGE with a 7-day window - func.sum(my_table.c.amount).over( - order_by=my_table.c.date, - range_=FrameClause( - start=timedelta(days=7), - end=None, - start_frame_type=FrameClauseType.PRECEDING, - end_frame_type=FrameClauseType.CURRENT, - ), - ) - -For backwards compatibility, the traditional tuple-based syntax continues to -work with integer values:: - - # This continues to work unchanged - func.row_number().over(order_by=table.c.col, range_=(None, 10)) - -However, attempting to use non-integer values in the tuple syntax will now -raise an error, directing users to use :class:`_sql.FrameClause` instead. - - -:ticket:`12596` .. _change_10300: @@ -1470,56 +1577,6 @@ SQLite) all handle ``visit_double()`` by rendering either ``DOUBLE`` or :ticket:`10300` -.. _change_8130: - -Explicit USING support for DELETE (MySQL, PostgreSQL) ------------------------------------------------------- - -The :meth:`_sql.Delete.using` method has been added, allowing explicit -``USING`` expressions to be specified in DELETE statements. This is -useful for backend-specific multiple-table DELETE forms where the secondary -FROM clause needs to be stated explicitly, such as joined DELETE on -MySQL/MariaDB and PostgreSQL. - -Previously, multi-table DELETE was supported by inferring extra FROM entries -from the WHERE clause, which works for simple cases. The new -:meth:`_sql.Delete.using` method allows more complex expressions such as -explicit joins to be stated:: - - from sqlalchemy import delete, table, column - - user_table = table("users", column("id"), column("name")) - address_table = table("addresses", column("id"), column("user_id"), column("email")) - - stmt = ( - delete(user_table) - .using( - user_table.outerjoin( - address_table, - user_table.c.id == address_table.c.user_id, - ) - ) - .where(address_table.c.email == "patrick@aol.com") - ) - -On MySQL/MariaDB, the above renders as: - -.. sourcecode:: sql - - DELETE FROM users USING users LEFT OUTER JOIN addresses - ON users.id = addresses.user_id - WHERE addresses.email = %s - -On PostgreSQL, a similar form is rendered using the PostgreSQL-specific -``DELETE .. USING`` syntax. - -.. seealso:: - - :ref:`tutorial_multi_table_deletes` - updated tutorial section for - multi-table deletes - -:ticket:`8130` - PostgreSQL ========== @@ -1577,7 +1634,7 @@ Changes to Named Type Handling in PostgreSQL --------------------------------------------- Named types such as :class:`_postgresql.ENUM`, :class:`_postgresql.DOMAIN` and -the dialect-agnostic :class:`._types.Enum` have undergone behavioral changes in +the dialect-agnostic :class:`_types.Enum` have undergone behavioral changes in SQLAlchemy 2.1 to better align with how a distinct type object that may be shared among tables works in practice. @@ -1678,8 +1735,8 @@ used by the :class:`.MetaData` is not what's desired. Support for ``VIRTUAL`` computed columns ---------------------------------------- -The behaviour of :paramref:`.Computed.persisted` has change in SQLAlchemy 2.1 -to no longer indicate ``STORED`` computed columns by default in PostgreSQL.. +The behavior of :paramref:`.Computed.persisted` has changed in SQLAlchemy 2.1 +to no longer indicate ``STORED`` computed columns by default in PostgreSQL. This change aligns SQLAlchemy with PostgreSQL 18+, which has introduced support for ``VIRTUAL`` computed columns, and has made them the default @@ -1688,7 +1745,7 @@ type if no qualifier is specified. Migration Path ^^^^^^^^^^^^^^ -To maintain the previous behaviour of ``STORED`` computed columns, +To maintain the previous behavior of ``STORED`` computed columns, :paramref:`.Computed.persisted` should be set to ``True`` explicitly:: from sqlalchemy import Table, Column, MetaData, Computed, Integer @@ -1855,29 +1912,6 @@ server-side function calls). Microsoft SQL Server ==================== -.. _change_11250: - -Potential breaking change to odbc_connect= handling for mssql+pyodbc --------------------------------------------------------------------- - -Fixed a mssql+pyodbc issue where valid plus signs in an already-unquoted -``odbc_connect=`` (raw DBAPI) connection string were replaced with spaces. - -Previously, the pyodbc connector would always pass the odbc_connect value -to unquote_plus(), even if it was not required. So, if the (unquoted) -odbc_connect value contained ``PWD=pass+word`` that would get changed to -``PWD=pass word``, and the login would fail. One workaround was to quote -just the plus sign — ``PWD=pass%2Bword`` — which would then get unquoted -to ``PWD=pass+word``. - -Implementations using the above workaround with :meth:`_engine.URL.create` -to specify a plus sign in the ``PWD=`` argument of an odbc_connect string -will have to remove the workaround and just pass the ``PWD=`` value as it -would appear in a valid ODBC connection string (i.e., the same as would be -required if using the connection string directly with ``pyodbc.connect()``). - -:ticket:`11250` - .. _change_12869: Support for mssql-python driver @@ -1914,6 +1948,73 @@ The ``mssql-python`` driver is available from PyPI: :ticket:`12869` +.. _change_11250: + +Potential breaking change to odbc_connect= handling for mssql+pyodbc +-------------------------------------------------------------------- + +The mssql+pyodbc connector was incorrectly applying ``unquote_plus()`` to +the ``odbc_connect`` value after extracting it. When using +:meth:`_engine.URL.create`, the value is a plain Python string with no URL +encoding, so ``unquote_plus()`` was never appropriate — it silently corrupted +literal ``+`` characters, rewriting ``PWD=pass+word`` as ``PWD=pass word``. +When using a raw URL string, the URL parser already handles decoding of the +query string, making the additional ``unquote_plus()`` call equally wrong. +The fix removes the ``unquote_plus()`` call entirely. + +The **breaking change** affects code that used ``%2B`` as a workaround +when passing ``odbc_connect`` via :meth:`_engine.URL.create`. Previously, +the ``%2B`` would be decoded to a literal ``+`` by the second +``unquote_plus()`` pass. Now it is passed to pyodbc as-is. Remove the +workaround and write the ``+`` directly:: + + # before (workaround — %2B relied on the now-removed unquote_plus pass) + engine = create_engine( + URL.create( + "mssql+pyodbc", + query={"odbc_connect": "DSN=mydsn;PWD=pass%2Bword"}, + ) + ) + + # after + engine = create_engine( + URL.create( + "mssql+pyodbc", + query={"odbc_connect": "DSN=mydsn;PWD=pass+word"}, + ) + ) + +A summary of ``odbc_connect`` patterns is as follows:: + + # raw URL string, password contains a space ("pass word") + # use %20 within the URL-encoded odbc_connect value + engine = create_engine("mssql+pyodbc:///?odbc_connect=DSN%3Dmydsn%3BPWD%3Dpass%20word") + + # raw URL string, password contains a plus sign ("pass+word") + # use %2B within the URL-encoded odbc_connect value + engine = create_engine("mssql+pyodbc:///?odbc_connect=DSN%3Dmydsn%3BPWD%3Dpass%2Bword") + + # URL.create(), password contains a space ("pass word") + # pass the odbc_connect value as a plain ODBC connection string + engine = create_engine( + URL.create( + "mssql+pyodbc", + query={"odbc_connect": "DSN=mydsn;PWD=pass word"}, + ) + ) + + # URL.create(), password contains a plus sign ("pass+word") + # pass the odbc_connect value as a plain ODBC connection string + engine = create_engine( + URL.create( + "mssql+pyodbc", + query={"odbc_connect": "DSN=mydsn;PWD=pass+word"}, + ) + ) + +:ticket:`11250` + + Oracle Database =============== @@ -1991,7 +2092,7 @@ Added :class:`_sqlite.JSONB` json format for SQLite --------------------------------------------------- SQLite version 3.45 added support for serializing json using -a binaly format called ``JSONB``, which provides imporved performance +a binary format called ``JSONB``, which provides improved performance and storage saving. The new :class:`_sqlite.JSONB` type provides support for this format, ensuring that the data is correctly serialized when inserting and deserialized when querying. diff --git a/doc/build/changelog/unreleased_21/13381.rst b/doc/build/changelog/unreleased_21/13381.rst index d1f5fc16cd..2780fa0e75 100644 --- a/doc/build/changelog/unreleased_21/13381.rst +++ b/doc/build/changelog/unreleased_21/13381.rst @@ -11,3 +11,7 @@ invalidation and pool notification when exit-type exceptions are raised in event hooks. As part of this change, DBAPI errors raised from within these event hooks will now be wrapped as SQLAlchemy exceptions. + + .. seealso:: + + :ref:`change_13381`