Mike Bayer [Wed, 21 Jul 2021 19:44:27 +0000 (15:44 -0400)]
Apply new uniquing rules for future ORM selects
Fixed issue where usage of the :meth:`_result.Result.unique` method with an
ORM result that included column expressions with unhashable types, such as
``JSON`` or ``ARRAY`` using non-tuples would silently fall back to using
the ``id()`` function, rather than raising an error. This now raises an
error when the :meth:`_result.Result.unique` method is used in a 2.0 style
ORM query. Additionally, hashability is assumed to be True for result
values of unknown type, such as often happens when using SQL functions of
unknown return type; if values are truly not hashable then the ``hash()``
itself will raise.
For legacy ORM queries, since the legacy :class:`_orm.Query` object
uniquifies in all cases, the old rules remain in place, which is to use
``id()`` for result values of unknown type as this legacy uniquing is
mostly for the purpose of uniquing ORM entities and not column values.
Mike Bayer [Tue, 20 Jul 2021 15:03:08 +0000 (11:03 -0400)]
dont warn for dictionary passed positionally
Fixed issue where use of the :paramref:`_sql.case.whens` parameter passing
a dictionary positionally and not as a keyword argument would emit a 2.0
deprecation warning, referring to the deprecation of passing a list
positionally. The dictionary format of "whens", passed positionally, is
still supported and was accidentally marked as deprecated.
Mike Bayer [Wed, 21 Jul 2021 15:18:01 +0000 (11:18 -0400)]
implement cache key for return_defaults token
Fixed critical caching issue where the ORM's persistence feature using
INSERT..RETURNING would cache an incorrect query when mixing the "bulk
save" and standard "flush" forms of INSERT.
Mike Bayer [Fri, 16 Jul 2021 15:34:13 +0000 (11:34 -0400)]
guard against unexpected weakref cleanups
Added some guards against ``KeyError`` in the event system to accommodate
the case that the interpreter is shutting down at the same time
:meth:`_engine.Engine.dispose` is being called, which would cause stack
trace warnings.
Mike Bayer [Fri, 16 Jul 2021 14:14:56 +0000 (10:14 -0400)]
reset key/name when TableValuedColumn is adapted
Fixed issue in new :meth:`_schema.Table.table_valued` method where the
resulting :class:`_sql.TableValuedColumn` construct would not respond
correctly to alias adaptation as is used throughout the ORM, such as for
eager loading, polymorphic loading, etc.
Mike Bayer [Thu, 15 Jul 2021 15:25:31 +0000 (11:25 -0400)]
limit None->null coercion to not occur with crud
Fixed issue where type-specific bound parameter handlers would not be
called upon in the case of using the :meth:`_sql.Insert.values` method with
the Python ``None`` value; in particular, this would be noticed when using
the :class:`_types.JSON` datatype as well as related PostgreSQL specific
types such as :class:`_postgresql.JSONB` which would fail to encode the
Python ``None`` value into JSON null, however the issue was generalized to
any bound parameter handler in conjunction with this specific method of
:class:`_sql.Insert`.
The issue with coercions forcing out ``null()`` may still impact
SQL expression usage as well; the change here is limited to crud
as the behavior there is relevant to some use cases, which may
need to be evaluated separately.
Mike Bayer [Thu, 15 Jul 2021 14:36:53 +0000 (10:36 -0400)]
apply list() around weakkeydictionary iteration
Fixed an issue where clearing of mappers during things like test suite
teardowns could cause a "dictionary changed size" warning during garbage
collection, due to iteration of a weak-referencing dictionary. A ``list()``
has been applied to prevent concurrent GC from affecting this operation.
Mike Bayer [Wed, 14 Jul 2021 14:29:50 +0000 (10:29 -0400)]
Ensure alias traversal block works when adapt_from_selectables present
Fixed regression which appeared in version 1.4.3 due to :ticket:`6060`
where rules that limit ORM adaptation of derived selectables interfered
with other ORM-adaptation based cases, in this case when applying
adaptations for a :func:`_orm.with_polymorphic` against a mapping which
uses a :func:`_orm.column_property` which in turn makes use of a scalar
select that includes a :func:`_orm.aliased` object of the mapped table.
Mike Bayer [Sat, 3 Jul 2021 23:48:55 +0000 (19:48 -0400)]
Adjust CTE recrusive col list to accommodate dupe col names
Fixed issue in CTE constructs where a recursive CTE that referred to a
SELECT that has duplicate column names, which are typically deduplicated
using labeling logic in 1.4, would fail to refer to the deduplicated label
name correctly within the WITH clause.
As part of this change we are also attempting to remove the
behavior of SelectStatementGrouping forcing off the "asfrom"
contextual flag, which will have the result of additional labeling
being applied to some UNION and similar statements when they are
interpreted as subqueries. To maintain compatibility with
"grouping", the Grouping/SelectStatementGrouping are now broken
out into two separate compiler cases, as the "asfrom" logic appears
to be tailored towards table valued SELECTS as column expressions.
Mike Bayer [Mon, 12 Jul 2021 22:19:08 +0000 (18:19 -0400)]
Extract format_constraint truncation rules to ON CONFLICT
Fixed issue where a too-long constraint name rendered as part of the "ON
CONFLICT ON CONSTRAINT" element of the :class:`_postgresql.Insert`
construct due to naming convention generation would not correctly truncate
the name in the same way that it normally renders within a CREATE TABLE
statement, thus producing a non-matching and too-long constraint name.
Mike Bayer [Tue, 6 Jul 2021 15:26:53 +0000 (11:26 -0400)]
labeling refactor
To service #6718 and #6710, the system by which columns are
given labels in a SELECT statement as well as the system that
gives them keys in a .c or .selected_columns collection have
been refactored to provide a single source of truth for
both, in constrast to the previous approach that included
similar logic repeated in slightly different ways.
Main ideas:
1. ColumnElement attributes ._label, ._anon_label, ._key_label
are renamed to include the letters "tq", meaning
"table-qualified" - these labels are only used when rendering
a SELECT that has LABEL_STYLE_TABLENAME_PLUS_COL for its
label style; as this label style is primarily legacy, the
"tq" names should be isolated so that in a 2.0 style application
these aren't being used at all
2. The means by which the "labels" and "proxy keys" for the elements
of a SELECT has been centralized to a single source of truth;
previously, the three of _generate_columns_plus_names,
_generate_fromclause_column_proxies, and _column_naming_convention
all had duplicated rules between them, as well as that there
were a little bit of labeling rules in compiler._label_select_column
as well; by this we mean that the various "anon_label" "anon_key"
methods on ColumnElement were called by all four of these methods,
where there were many cases where it was necessary that one method
comes up with the same answer as another of the methods. This
has all been centralized into _generate_columns_plus_names
for all the names except the "proxy key", which is generated
by _column_naming_convention.
3. compiler._label_select_column has been rewritten to both not make
any naming decisions nor any "proxy key" decisions, only whether
to label or not to label; the _generate_columns_plus_names method
gives it the information, where the proxy keys come from
_column_naming_convention; previously, these proxy keys were matched
based on restatement of similar (but not really the same) logic in
two places. The heuristics of "whether to label or not to label"
are also reorganized to be much easier to read and understand.
4. a new method compiler._label_returning_column is added for dialects
to use in their "generate returning columns" methods. A
github search reveals a small number of third party dialects also
doing this using the prior _label_select_column method so we
try to make sure _label_select_column continues to work the
exact same way for that specific use case; for the "SELECT" use
case it now needs
5. After some attempts to do it different ways, for the case where
_proxy_key is giving us some kind of anon label, we are hard
changing it to "_no_label" right now, as there's not currently
a way to fully match anonymized labels from stmt.c or
stmt.selected_columns to what will be in the result map. The
idea of "_no_label" is to encourage the user to use label('name')
for columns they want to be able to target by string name that
don't have a natural name.
Mike Bayer [Mon, 12 Jul 2021 18:28:19 +0000 (14:28 -0400)]
implement independent CTEs
Added new method :meth:`_sql.HasCTE.add_cte` to each of the
:func:`_sql.select`, :func:`_sql.insert`, :func:`_sql.update` and
:func:`_sql.delete` constructs. This method will add the given
:class:`_sql.CTE` as an "independent" CTE of the statement, meaning it
renders in the WITH clause above the statement unconditionally even if it
is not otherwise referenced in the primary statement. This is a popular use
case on the PostgreSQL database where a CTE is used for a DML statement
that runs against database rows independently of the primary statement.
Mike Bayer [Sun, 11 Jul 2021 23:40:59 +0000 (19:40 -0400)]
represent tablesample.sampling as FunctionElement in all cases
Fixed regression where the :func:`_sql.tablesample` construct would fail to
be executable when constructed given a floating-point sampling value not
embedded within a SQL function.
Mike Bayer [Sun, 11 Jul 2021 23:23:40 +0000 (19:23 -0400)]
repair schema_translate_map for schema type use cases
Fixed issue where the PostgreSQL ``ENUM`` datatype as embedded in the
``ARRAY`` datatype would fail to emit correctly in create/drop when the
``schema_translate_map`` feature were also in use. Additionally repairs a
related issue where the same ``schema_translate_map`` feature would not
work for the ``ENUM`` datatype in combination with a ``CAST``, that's also
intrinsic to how the ``ARRAY(ENUM)`` combination works on the PostgreSQL
dialect.
Mike Bayer [Fri, 2 Jul 2021 15:23:20 +0000 (11:23 -0400)]
implement deferred scalarobject history load
Modified the approach used for history tracking of scalar object
relationships that are not many-to-one, i.e. one-to-one relationships that
would otherwise be one-to-many. When replacing a one-to-one value, the
"old" value that would be replaced is no longer loaded immediately, and is
instead handled during the flush process. This eliminates an historically
troublesome lazy load that otherwise often occurs when assigning to a
one-to-one attribute, and is particularly troublesome when using
"lazy='raise'" as well as asyncio use cases.
This change does cause a behavioral change within the
:meth:`_orm.AttributeEvents.set` event, which is nonetheless currently
documented, which is that the event applied to such a one-to-one attribute
will no longer receive the "old" parameter if it is unloaded and the
:paramref:`_orm.relationship.active_history` flag is not set. As is
documented in :meth:`_orm.AttributeEvents.set`, if the event handler needs
to receive the "old" value when the event fires off, the active_history
flag must be established either with the event listener or with the
relationship. This is already the behavior with other kinds of attributes
such as many-to-one and column value references.
The change additionally will defer updating a backref on the "old" value
in the less common case that the "old" value is locally present in the
session, but isn't loaded on the relationship in question, until the
next flush occurs. If this causes an issue, again the normal
:paramref:`_orm.relationship.active_history` flag can be set to ``True``
on the relationship.
A private flag which restores the old value is retained for now,
as support within relevant test suites to exercise the old and
new behaviors together. This is so that if the behavioral change
produces problems we have test harnesses set up to further examine these
behaviors. The "legacy" style can go away in 2.0 or in a much later
1.4 release.
Mike Bayer [Mon, 5 Jul 2021 18:37:58 +0000 (14:37 -0400)]
Add additional support to honor _proxy_key in Core selects
Fixed ORM regression where ad-hoc label names generated for hybrid
properties and potentially other similar types of ORM-enabled expressions
would usually be propagated outwards through subqueries, allowing the name
to be retained in the final keys of the result set even when selecting from
subqueries. Additional state is now tracked in this case that isn't lost
when a hybrid is selected out of a Core select / subquery.
as we have removed things like column.label() from
ORM, since we now have to export the cols with the same names
as what we will render, experiment with giving a greater role
to the _proxy_key annotation so that a desired name can be
carried through more transarently.
Mike Bayer [Wed, 30 Jun 2021 20:04:07 +0000 (16:04 -0400)]
clear new Query._memoized_select_entities in _from_selectable
Fixed regression caused in 1.4.19 due to #6503 and related involving
:meth:`_orm.Query.with_entities` where the new structure used would be
inappropriately transferred to an enclosing :class:`_orm.Query` when making
use of set operations such as :meth:`_orm.Query.union`, causing the JOIN
instructions within to be applied to the outside query as well.
Mike Bayer [Wed, 30 Jun 2021 14:52:09 +0000 (10:52 -0400)]
Ensure compiler uses quote_schema hook for translates renders
Fixed regression where the special dotted-schema name handling for the SQL
Server dialect would not function correctly if the dotted schema name were
used within the ``schema_translate_map`` feature.
Mike Bayer [Wed, 30 Jun 2021 13:22:00 +0000 (09:22 -0400)]
apply quoting to "ON CONSTRAINT" symbol
Fixed issue in :meth:`_postgresql.Insert.on_conflict_do_nothing` and
:meth:`_postgresql.Insert.on_conflict_do_update` where the name of a unique
constraint passed as the ``constraint`` parameter would not be properly
quoted if it contained characters which required quoting.
Miłosz Stypiński [Thu, 24 Jun 2021 16:21:30 +0000 (12:21 -0400)]
Be less fiscal regarding validators functions
Adjusted the check in the mapper for a callable object that is used as a
``@validates`` validator function or a ``@reconstructor`` reconstruction
function, to check for "callable" more liberally such as to accommodate
objects based on fundamental attributes like ``__func__`` and
``__call___``, rather than testing for ``MethodType`` / ``FunctionType``,
allowing things like cython functions to work properly. Pull request
courtesy Miłosz Stypiński.
Kai Mueller [Thu, 24 Jun 2021 15:57:20 +0000 (11:57 -0400)]
Fix missing None handling of Table.prefixes
Fixed issue where passing ``None`` for the value of
:paramref:`_schema.Table.prefixes` would not store an empty list, but
rather the constant ``None``, which may be unexpected by third party
dialects. The issue is revealed by a usage in recent versions of Alembic
that are passing ``None`` for this value. Pull request courtesy Kai
Mueller.
Daniël van Eeden [Wed, 23 Jun 2021 14:00:43 +0000 (10:00 -0400)]
Don't require space after MySQL pk comment
Made a small adjustment in the table reflection feature of the MySQL
dialect to accommodate for alternate MySQL-oriented databases such as TiDB
which include their own "comment" directives at the end of a constraint
directive within "CREATE TABLE" where the format doesn't have the
additional space character after the comment, in this case the TiDB
"clustered index" feature. Pull request courtesy Daniël van Eeden.
Mike Bayer [Sat, 26 Jun 2021 20:40:09 +0000 (16:40 -0400)]
ensure with poly entities are also reconstituted for GC'ed AC
Fixed regression in ORM regarding an internal reconstitution step for the
func:`_orm.with_polymorphic` construct, when the user-facing object is
garbage collected as the query is processed. The reconstitution was not
ensuring the sub-entities for the "polymorphic" case were handled, leading
to an ``AttributeError``.
Mike Bayer [Fri, 25 Jun 2021 20:10:01 +0000 (16:10 -0400)]
set _render_for_subquery for legacy set ops
Adjusted :meth:`_orm.Query.union` and similar set operations to be
correctly compatible with the new capabilities just added in
:ticket:`6661`, with SQLAlchemy 1.4.19, such that the SELECT statements
rendered as elements of the UNION or other set operation will include
directly mapped columns that are mapped as deferred; this both fixes a
regression involving unions with multiple levels of nesting that would
produce a column mismatch, and also allows the :func:`_orm.undefer` option
to be used at the top level of such a :class:`_orm.Query` without having to
apply the option to each of the elements within the UNION.
Mike Bayer [Sat, 26 Jun 2021 13:55:00 +0000 (09:55 -0400)]
turn pg provision error into a warning
We haven't had any real cases of the PG "cant drop tables"
condition since this error was first introduced; instead we
seem to get it for a non-critical query during pool reconnect
tests, and I have not been able to isolate what is causing it.
Therefore turn the error into a new class of warning that can
emit within the test suite without failing the test, so that
if we do get a real PG drop timeout, the warning will be there
to show us what the query was in which it was stuck.
Mike Bayer [Fri, 25 Jun 2021 20:51:50 +0000 (16:51 -0400)]
have automap suppress overlaps warning for mapped secondary
Fixed regression in :mod:`sqlalchemy.ext.automap` extension such that the
use case of creating an explicit mapped class to a table that is also the
:paramref:`_orm.relationship.secondary` element of a
:func:`_orm.relationship` that automap will be generating would emit the
"overlaps" warnings introduced in 1.4 and discussed at :ref:`error_qzyx`.
While generating this case from automap is still subject to the same
caveats that the "overlaps" warning refers towards, as automap is intended
for more ad-hoc use cases, the condition which produces the warning is
disabled when a many-to-many relationship with this particular pattern is
generated.
jason3gb [Thu, 24 Jun 2021 16:11:04 +0000 (12:11 -0400)]
Add "impl" parameter to PickleType
Add a impl parameter to :class:`_types.PickleType` constructor, allowing
any arbitary type to be used in place of the default implementation of
:class:`_types.LargeBinary`. Pull request courtesy jason3gb.
Mike Bayer [Thu, 24 Jun 2021 13:12:31 +0000 (09:12 -0400)]
Use Py_TPFLAGS_HAVE_GC for Row
Fixed an issue in the C extension for the :class:`_result.Row` class which
could lead to a memory leak in the unlikely case of a :class:`_result.Row`
object which referred to an ORM object that then was mutated to refer back
to the ``Row`` itself, creating a cycle. The Python C APIs for tracking GC
cycles has been added to the native :class:`_result.Row` implementation to
accommodate for this case.
Mike Bayer [Wed, 23 Jun 2021 20:34:05 +0000 (16:34 -0400)]
Add Executable to DefaultGenerator
Fixed the class hierarchy for the :class:`_schema.Sequence` and the more
general :class:`_schema.DefaultGenerator` base, as these are "executable"
as statements they need to include :class:`_sql.Executable` in their
hierarchy, not just :class:`_roles.StatementRole` as was applied
arbitrarily to :class:`_schema.Sequence` previously. The fix allows
:class:`_schema.Sequence` to work in all ``.execute()`` methods including
with :meth:`_orm.Session.execute` which was not working in the case that a
``do_orm_execute()`` handler was also established.
Mike Bayer [Wed, 23 Jun 2021 12:50:48 +0000 (08:50 -0400)]
consider "*" col as textual ordered
Fixed old issue where a :func:`_sql.select()` made against the token "*",
which then yielded exactly one column, would fail to correctly organize the
``cursor.description`` column name into the keys of the result object.
MajorDallas [Tue, 22 Jun 2021 19:34:09 +0000 (15:34 -0400)]
Add impl property to PostgreSQL / Oracle INTERVAL class
Fixed issue where the ``INTERVAL`` datatype on PostgreSQL and Oracle would
produce an ``AttributeError`` when used in the context of a comparison
operation against a ``timedelta()`` object. Pull request courtesy
MajorDallas.
Mike Bayer [Tue, 22 Jun 2021 17:27:18 +0000 (13:27 -0400)]
Export deferred columns but not col props; fix CTE labeling
Refined the behavior of ORM subquery rendering with regards to deferred
columns and column properties to be more compatible with that of 1.3 while
also providing for 1.4's newer features. As a subquery in 1.4 does not make
use of loader options, including :func:`_orm.deferred`, a subquery that is
against an ORM entity with deferred attributes will now render those
deferred attributes that refer directly to mapped table columns, as these
are needed in the outer SELECT if that outer SELECT makes use of these
columns; however a deferred attribute that refers to a composed SQL
expression as we normally do with :func:`_orm.column_property` will not be
part of the subquery, as these can be selected explicitly if needed in the
subquery. If the entity is being SELECTed from this subquery, the column
expression can still render on "the outside" in terms of the derived
subquery columns. This produces essentially the same behavior as when
working with 1.3. However in this case the fix has to also make sure that
the ``.selected_columns`` collection of an ORM-enabled :func:`_sql.select`
also follows these rules, which in particular allows recursive CTEs to
render correctly in this scenario, which were previously failing to render
correctly due to this issue.
As part of this change the _exported_columns_iterator() method has been
removed and logic simplified to use ._all_selected_columns from any
SelectBase object where _exported_columns_iterator() was used before.
Additionally sets up UpdateBase to include ReturnsRows in its hierarchy;
the literal point of ReturnsRows was to be a common base for UpdateBase
and SelectBase so it was kind of weird it wasn't there.
Fixes: #6661
Fixed issue in CTE constructs mostly relevant to ORM use cases where a
recursive CTE against "anonymous" labels such as those seen in ORM
``column_property()`` mappings would render in the
``WITH RECURSIVE xyz(...)`` section as their raw internal label and not a
cleanly anonymized name.
Mike Bayer [Mon, 21 Jun 2021 22:13:55 +0000 (18:13 -0400)]
apply render_schema_translates to identity insert directives
Fixed bug where the "schema_translate_map" feature would fail to function
correctly in conjunction with an INSERT into a table that has an IDENTITY
column, where the value of the IDENTITY column were specified in the values
of the INSERT thus triggering SQLAlchemy's feature of setting IDENTITY
INSERT to "on"; it's in this directive where the schema translate map would
fail to be honored.
Anton Kovalevich [Fri, 18 Jun 2021 14:33:48 +0000 (10:33 -0400)]
Implement MySQL-specific MATCH
Added new construct :class:`_mysql.match`, which provides for the full
range of MySQL's MATCH operator including multiple column support and
modifiers. Pull request courtesy Anton Kovalevich.
Mike Bayer [Mon, 21 Jun 2021 17:08:59 +0000 (13:08 -0400)]
ensure test has deterministic FROM rendering
test_options_entities_replaced_with_equivs_three did not
have deterministic FROM ordering, so adding an inner
join from user->address should ensure there's a single
FROM element that is a series of joins.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
