Mike Bayer [Fri, 5 May 2023 15:50:29 +0000 (11:50 -0400)]
add AsyncAttrs
Added a new helper mixin :class:`_asyncio.AsyncAttrs` that seeks to improve
the use of lazy-loader and other expired or deferred ORM attributes with
asyncio, providing a simple attribute accessor that provides an ``await``
interface to any ORM attribute, whether or not it needs to emit SQL.
Mike Bayer [Fri, 5 May 2023 13:16:10 +0000 (09:16 -0400)]
add explicit step to set populate_existing for bulk insert
Fixed issue in new :ref:`orm_queryguide_upsert_returning` feature where the
``populate_existing`` execution option was not being propagated to the
loading option, preventing existing attributes from being refreshed
in-place.
Mike Bayer [Sun, 30 Apr 2023 17:56:33 +0000 (13:56 -0400)]
do not allow non-cache-key entity objects in annotations
Fixed critical caching issue where combination of :func:`_orm.aliased()`
:func:`_sql.case` and :func:`_hybrid.hybrid_property` expressions would
cause a cache key mismatch, leading to cache keys that held onto the actual
:func:`_orm.aliased` object while also not matching each other, filling up
the cache.
Fixed issues regarding reflection of comments for :class:`_schema.Table`
and :class:`_schema.Column` objects, where the comments contained control
characters such as newlines. Additional testing support for these
characters as well as extended Unicode characters in table and column
comments (the latter of which aren't supported by MySQL/MariaDB) added to
testing overall.
Mike Bayer [Thu, 27 Apr 2023 20:48:25 +0000 (16:48 -0400)]
improve natural_path usage in two places
Fixed loader strategy pathing issues where eager loaders such as
:func:`_orm.joinedload` / :func:`_orm.selectinload` would fail to traverse
fully for many-levels deep following a load that had a
:func:`_orm.with_polymorphic` or similar construct as an interim member.
Here we can take advantage of 2.0's refactoring of strategy_options
to identify the "chop_path" concept can be simplified to work
with "natural" paths alone.
In addition, identified existing
logic in PropRegistry that works fine, but needed the "is_unnatural"
attribute to be more accurate for a given path, so we set that
up front to True if the ancestor is_unnatural.
Mike Bayer [Fri, 28 Apr 2023 03:00:18 +0000 (23:00 -0400)]
adjust fwd_ref logic
Fixed issue where ORM Annotated Declarative would not resolve forward
references correctly in all cases; in particular, when using
``from __future__ import annotations`` in combination with Pydantic
dataclasses.
Add public alias of type _ColumnExpressionArgument
Added type ``ColumnExpressionArgument`` as a public alias of an internal
type. This type is useful since it's what' accepted by the sqlalchemy in
many api calls, such as :meth:`_sql.Select.where`, :meth:`_sql.and` and
many other.
Various performance improvements to Row instanciation
- avoid passing processors if they are all None
- improve processor logic in cython
- improve tuplegetter using slices when contiguous indexes are used
Some timing follow.
In particular [base_]row_new_proc that tests using processors has
a 25% improvement compared to before in cython.
Looking at the [b]row_new_proc_none that test a list of processors
all None, this has 50% improvement in cython when passing the none list,
but in this patch it would usually be disabled by passing None, so the
performance gain is actually 90%, since it would run the case
[base_]row_new.
Tuplegetter is a bit faster in the single item get and when getting
sequential indexes (like indexes 1,2,3,4) at the cost of a bit
longer creation time in python, cython is mostly the same.
Mike Bayer [Sat, 1 Apr 2023 15:56:56 +0000 (11:56 -0400)]
support parameters in all ORM insert modes
Fixed 2.0 regression where use of :func:`_sql.bindparam()` inside of
:meth:`_dml.Insert.values` would fail to be interpreted correctly when
executing the :class:`_dml.Insert` statement using the ORM
:class:`_orm.Session`, due to the new ORM-enabled insert feature not
implementing this use case.
In addition, the bulk INSERT and UPDATE features now add these
capabilities:
* The requirement that extra parameters aren't passed when using ORM
INSERT using the "orm" dml_strategy setting is lifted.
* The requirement that additional WHERE criteria is not passed when using
ORM UPDATE using the "bulk" dml_strategy setting is lifted. Note that
in this case, the check for expected row count is turned off.
Mike Bayer [Wed, 26 Apr 2023 14:34:46 +0000 (10:34 -0400)]
ensure correct cast for floats vs. numeric; other fixes
Fixed regression caused by the fix for :ticket:`9618` where floating point
values would lose precision being inserted in bulk, using either the
psycopg2 or psycopg drivers.
Implemented the :class:`_sqltypes.Double` type for SQL Server, having it
resolve to ``REAL``, or :class:`_mssql.REAL`, at DDL rendering time.
Fixed issue in Oracle dialects where ``Decimal`` returning types such as
:class:`_sqltypes.Numeric` would return floating point values, rather than
``Decimal`` objects, when these columns were used in the
:meth:`_dml.Insert.returning` clause to return INSERTed values.
J. Nick Koston [Wed, 19 Apr 2023 22:39:18 +0000 (18:39 -0400)]
Prebuild the row string to position lookup for Rows
Improved :class:`_engine.Row` implementation to optimize
``__getattr__`` performance.
The serialization of a :class:`_engine.Row` to pickle has changed with
this change. Pickle saved by older SQLAlchemy versions can still be loaded,
but new pickle saved by this version cannot be loaded by older ones.
J. Nick Koston [Tue, 25 Apr 2023 02:32:17 +0000 (22:32 -0400)]
disable "bytes" handler for all drivers other than psycopg2
Improved row processing performance for "binary" datatypes by making the
"bytes" handler conditional on a per driver basis. As a result, the
"bytes" result handler has been disabled for nearly all drivers other than
psycopg2, all of which in modern forms support returning Python "bytes"
directly. Pull request courtesy J. Nick Koston.
Mike Bayer [Sat, 22 Apr 2023 13:41:49 +0000 (09:41 -0400)]
support slice access for .c
Added support for slice access with :class:`.ColumnCollection`, e.g.
``table.c[0:5]``, ``subquery.c[:-1]`` etc. Slice access returns a sub
:class:`.ColumnCollection` in the same way as passing a tuple of keys. This
is a natural continuation of the key-tuple access added for :ticket:`8285`,
which it appears to be an oversight that this usage was omitted.
Mike Bayer [Fri, 21 Apr 2023 19:17:39 +0000 (15:17 -0400)]
restore fallback for uow test
in cf6872d3bdf1a8a9613e85369 I decided this test should
be able to run in all cases, however apparently on windows
/ py37 on GH actions something goes wrong, so restore the skip
to that case.
Pavel Sirotkin [Thu, 20 Apr 2023 17:40:04 +0000 (13:40 -0400)]
Add name_func optional attribute for asyncpg adapter
I faced an issue related to pg bouncer and prepared statement cache flow in asyncpg dialect. Regarding this discussion https://github.com/sqlalchemy/sqlalchemy/issues/6467 I prepared PR to support an optional parameter `name` in prepared statement which is allowed, since 0.25.0 version in `asyncpg` https://github.com/MagicStack/asyncpg/pull/846
**UPD:**
the issue with proposal: https://github.com/sqlalchemy/sqlalchemy/issues/9608
### Description
Added optional parameter `name_func` to `AsyncAdapt_asyncpg_connection` class which will call on the `self._connection.prepare()` function and populate a unique name.
so in general instead this
```python
from uuid import uuid4
from asyncpg import Connection
class CConnection(Connection):
def _get_unique_id(self, prefix: str) -> str:
return f'__asyncpg_{prefix}_{uuid4()}__'
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [x] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Wed, 5 Apr 2023 15:58:52 +0000 (11:58 -0400)]
add deterministic imv returning ordering using sentinel columns
Repaired a major shortcoming which was identified in the
:ref:`engine_insertmanyvalues` performance optimization feature first
introduced in the 2.0 series. This was a continuation of the change in
2.0.9 which disabled the SQL Server version of the feature due to a
reliance in the ORM on apparent row ordering that is not guaranteed to take
place. The fix applies new logic to all "insertmanyvalues" operations,
which takes effect when a new parameter
:paramref:`_dml.Insert.returning.sort_by_parameter_order` on the
:meth:`_dml.Insert.returning` or :meth:`_dml.UpdateBase.return_defaults`
methods, that through a combination of alternate SQL forms, direct
correspondence of client side parameters, and in some cases downgrading to
running row-at-a-time, will apply sorting to each batch of returned rows
using correspondence to primary key or other unique values in each row
which can be correlated to the input data.
Performance impact is expected to be minimal as nearly all common primary
key scenarios are suitable for parameter-ordered batching to be
achieved for all backends other than SQLite, while "row-at-a-time"
mode operates with a bare minimum of Python overhead compared to the very
heavyweight approaches used in the 1.x series. For SQLite, there is no
difference in performance when "row-at-a-time" mode is used.
It's anticipated that with an efficient "row-at-a-time" INSERT with
RETURNING batching capability, the "insertmanyvalues" feature can be later
be more easily generalized to third party backends that include RETURNING
support but not necessarily easy ways to guarantee a correspondence
with parameter order.
Mike Bayer [Fri, 21 Apr 2023 13:48:04 +0000 (09:48 -0400)]
try to omit unnecessary cols for ORM bulk insert + returning
Fixed bug in ORM bulk insert feature where additional unnecessary columns
would be rendered in the INSERT statement if RETURNING of individual
columns were requested.
Andy Freeland [Thu, 20 Apr 2023 17:41:39 +0000 (13:41 -0400)]
Fix `RowMapping`'s `Mapping` type to reflect that it supports `Column`s or strings
### Description
I ran into this originally in sqlalchemy2-stubs: https://github.com/sqlalchemy/sqlalchemy2-stubs/pull/251, where `RowMapping` only supported string keys according to the type hints. I ran into a similar issue here upgrading our application where because `RowMapping` subclassed `Mapping[str, Any]`, `Row._mapping.get()` would fail to typecheck when used with `Column` objects.
This patch adds a test to verify that `Row._mapping.get()` continues to work with both strings and `Column`s, though it doesn't look like mypy checks types in the tests.
Fixes #9644.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Tue, 18 Apr 2023 13:42:50 +0000 (09:42 -0400)]
improve return type for QueryableAttribute.and_()
Fixed typing issue where :meth:`_orm.PropComparator.and_` expressions would
not be correctly typed inside of loader options such as
:func:`_orm.selectinload`.
Mike Bayer [Mon, 17 Apr 2023 17:46:12 +0000 (13:46 -0400)]
apply criteria options from top-level core-only statement
Made an improvement to the :func:`_orm.with_loader_criteria` loader option
to allow it to be indicated in the :meth:`.Executable.options` method of a
top-level statement that is not itself an ORM statement. Examples include
:func:`_sql.select` that's embedded in compound statements such as
:func:`_sql.union`, within an :meth:`_dml.Insert.from_select` construct, as
well as within CTE expressions that are not ORM related at the top level.
Improved propagation of :func:`_orm.with_loader_criteria` within
ORM enabled UPDATE and DELETE statements as well.
Mike Bayer [Mon, 17 Apr 2023 14:16:35 +0000 (10:16 -0400)]
dont assume _compile_options are present
Fixed bug where various ORM-specific getters such as
:attr:`.ORMExecuteState.is_column_load`,
:attr:`.ORMExecuteState.is_relationship_load`,
:attr:`.ORMExecuteState.loader_strategy_path` etc. would throw an
``AttributeError`` if the SQL statement itself were a "compound select"
such as a UNION.
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description Fixes: #9509
<!-- Describe your changes in detail -->
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [x] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
John Clow [Wed, 12 Apr 2023 06:34:58 +0000 (02:34 -0400)]
Adding typing to Postgres dialect file.
Adding typing information for various parameters for Postgres types (in accordance to the docs).
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Added :func:`_sa.create_pool_from_url` and
:func:`_asyncio.create_async_pool_from_url` to create
a :class:`_pool.Pool` instance from an input url passed as string
or :class:`_sa.URL`.
Mike Bayer [Mon, 10 Apr 2023 16:56:47 +0000 (12:56 -0400)]
establish column_property and query_expression as readonly from a dc perspective
Fixed bug in ORM Declarative Dataclasses where the
:func:`_orm.queryable_attribute` and :func:`_orm.column_property`
constructs, which are documented as read-only constructs in the context of
a Declarative mapping, could not be used with a
:class:`_orm.MappedAsDataclass` class without adding ``init=False``, which
in the case of :func:`_orm.queryable_attribute` was not possible as no
``init`` parameter was included. These constructs have been modified from a
dataclass perspective to be assumed to be "read only", setting
``init=False`` by default and no longer including them in the pep-681
constructor. The dataclass parameters for :func:`_orm.column_property`
``init``, ``default``, ``default_factory``, ``kw_only`` are now deprecated;
these fields don't apply to :func:`_orm.column_property` as used in a
Declarative dataclasses configuration where the construct would be
read-only. Also added read-specific parameter
:paramref:`_orm.queryable_attribute.compare` to
:func:`_orm.queryable_attribute`; :paramref:`_orm.queryable_attribute.repr`
was already present.
Added missing :paramref:`_orm.mapped_column.active_history` parameter
to :func:`_orm.mapped_column` construct.
Mike Bayer [Mon, 10 Apr 2023 14:28:44 +0000 (10:28 -0400)]
include declared_directive as a declared_attr
Fixed issue where the :meth:`_orm.declared_attr.directive` modifier was not
correctly honored for subclasses when applied to the ``__mapper_args__``
special method name, as opposed to direct use of
:class:`_orm.declared_attr`. The two constructs should have identical
runtime behaviors.
Mike Bayer [Sat, 8 Apr 2023 22:43:31 +0000 (18:43 -0400)]
fix pg ENUM issues
Restored the :paramref:`_postgresql.ENUM.name` parameter as optional in the
signature for :class:`_postgresql.ENUM`, as this is chosen automatically
from a given pep-435 ``Enum`` type.
Fixed issue where the comparison for :class:`_postgresql.ENUM` against a
plain string would cast that right-hand side type as VARCHAR, which due to
more explicit casting added to dialects such as asyncpg would produce a
PostgreSQL type mismatch error.
Mike Bayer [Sat, 8 Apr 2023 03:31:55 +0000 (23:31 -0400)]
walk back SQL Server language a bit re: insertmanyvalues
In #9618 we both can look to re-enable insertmanyvalues
for SQL Server, and also likely *disable* its use for the
ORM unit of work specifically, since that's really where the
only problem is, and it will likely be for all dialects,
not just SQL Server. An approach using sentinel columns will
be rolled out for the unit of work use case.
I found an extremely and probably insignificant typo in the doc for `mapped_column` while actually trying to use and understand it for my project and have fixed it here.
The typo was for a "for" which was instead mentioned as "or".
Fixed issue that prevented reflection of expression based indexes
with long expressions in PostgreSQL. The expression where erroneously
truncated to the identifier length (that's 63 bytes by default).
Mike Bayer [Wed, 5 Apr 2023 16:59:13 +0000 (12:59 -0400)]
turn off use_insertmanyvalues for SQL Server
we will keep trying to find workarounds, however this
patch is the "turn it off" patch
Due to a critical bug identified in SQL Server, the SQLAlchemy
"insertmanyvalues" feature which allows fast INSERT of many rows while also
supporting RETURNING unfortunately needs to be disabled for SQL Server. SQL
Server is apparently unable to guarantee that the order of rows inserted
matches the order in which they are sent back by OUTPUT inserted when
table-valued rows are used with INSERT in conjunction with OUTPUT inserted.
We are trying to see if Microsoft is able to confirm this undocumented
behavior however there is no known workaround, other than it's not safe to
use table-valued expressions with OUTPUT inserted for now.
Mike Bayer [Sun, 2 Apr 2023 18:24:32 +0000 (14:24 -0400)]
consider aliased mappers in cycles also
Fixed endless loop which could occur when using "relationship to aliased
class" feature and also indicating a recursive eager loader such as
``lazy="selectinload"`` in the loader, in combination with another eager
loader on the opposite side. The check for cycles has been fixed to include
aliased class relationships.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
