Mike Bayer [Tue, 6 Jul 2021 15:26:53 +0000 (11:26 -0400)]
labeling refactor
To service #6718 and #6710, the system by which columns are
given labels in a SELECT statement as well as the system that
gives them keys in a .c or .selected_columns collection have
been refactored to provide a single source of truth for
both, in constrast to the previous approach that included
similar logic repeated in slightly different ways.
Main ideas:
1. ColumnElement attributes ._label, ._anon_label, ._key_label
are renamed to include the letters "tq", meaning
"table-qualified" - these labels are only used when rendering
a SELECT that has LABEL_STYLE_TABLENAME_PLUS_COL for its
label style; as this label style is primarily legacy, the
"tq" names should be isolated so that in a 2.0 style application
these aren't being used at all
2. The means by which the "labels" and "proxy keys" for the elements
of a SELECT has been centralized to a single source of truth;
previously, the three of _generate_columns_plus_names,
_generate_fromclause_column_proxies, and _column_naming_convention
all had duplicated rules between them, as well as that there
were a little bit of labeling rules in compiler._label_select_column
as well; by this we mean that the various "anon_label" "anon_key"
methods on ColumnElement were called by all four of these methods,
where there were many cases where it was necessary that one method
comes up with the same answer as another of the methods. This
has all been centralized into _generate_columns_plus_names
for all the names except the "proxy key", which is generated
by _column_naming_convention.
3. compiler._label_select_column has been rewritten to both not make
any naming decisions nor any "proxy key" decisions, only whether
to label or not to label; the _generate_columns_plus_names method
gives it the information, where the proxy keys come from
_column_naming_convention; previously, these proxy keys were matched
based on restatement of similar (but not really the same) logic in
two places. The heuristics of "whether to label or not to label"
are also reorganized to be much easier to read and understand.
4. a new method compiler._label_returning_column is added for dialects
to use in their "generate returning columns" methods. A
github search reveals a small number of third party dialects also
doing this using the prior _label_select_column method so we
try to make sure _label_select_column continues to work the
exact same way for that specific use case; for the "SELECT" use
case it now needs
5. After some attempts to do it different ways, for the case where
_proxy_key is giving us some kind of anon label, we are hard
changing it to "_no_label" right now, as there's not currently
a way to fully match anonymized labels from stmt.c or
stmt.selected_columns to what will be in the result map. The
idea of "_no_label" is to encourage the user to use label('name')
for columns they want to be able to target by string name that
don't have a natural name.
Mike Bayer [Mon, 12 Jul 2021 18:28:19 +0000 (14:28 -0400)]
implement independent CTEs
Added new method :meth:`_sql.HasCTE.add_cte` to each of the
:func:`_sql.select`, :func:`_sql.insert`, :func:`_sql.update` and
:func:`_sql.delete` constructs. This method will add the given
:class:`_sql.CTE` as an "independent" CTE of the statement, meaning it
renders in the WITH clause above the statement unconditionally even if it
is not otherwise referenced in the primary statement. This is a popular use
case on the PostgreSQL database where a CTE is used for a DML statement
that runs against database rows independently of the primary statement.
Mike Bayer [Sun, 11 Jul 2021 23:40:59 +0000 (19:40 -0400)]
represent tablesample.sampling as FunctionElement in all cases
Fixed regression where the :func:`_sql.tablesample` construct would fail to
be executable when constructed given a floating-point sampling value not
embedded within a SQL function.
Mike Bayer [Sun, 11 Jul 2021 23:23:40 +0000 (19:23 -0400)]
repair schema_translate_map for schema type use cases
Fixed issue where the PostgreSQL ``ENUM`` datatype as embedded in the
``ARRAY`` datatype would fail to emit correctly in create/drop when the
``schema_translate_map`` feature were also in use. Additionally repairs a
related issue where the same ``schema_translate_map`` feature would not
work for the ``ENUM`` datatype in combination with a ``CAST``, that's also
intrinsic to how the ``ARRAY(ENUM)`` combination works on the PostgreSQL
dialect.
Mike Bayer [Fri, 2 Jul 2021 15:23:20 +0000 (11:23 -0400)]
implement deferred scalarobject history load
Modified the approach used for history tracking of scalar object
relationships that are not many-to-one, i.e. one-to-one relationships that
would otherwise be one-to-many. When replacing a one-to-one value, the
"old" value that would be replaced is no longer loaded immediately, and is
instead handled during the flush process. This eliminates an historically
troublesome lazy load that otherwise often occurs when assigning to a
one-to-one attribute, and is particularly troublesome when using
"lazy='raise'" as well as asyncio use cases.
This change does cause a behavioral change within the
:meth:`_orm.AttributeEvents.set` event, which is nonetheless currently
documented, which is that the event applied to such a one-to-one attribute
will no longer receive the "old" parameter if it is unloaded and the
:paramref:`_orm.relationship.active_history` flag is not set. As is
documented in :meth:`_orm.AttributeEvents.set`, if the event handler needs
to receive the "old" value when the event fires off, the active_history
flag must be established either with the event listener or with the
relationship. This is already the behavior with other kinds of attributes
such as many-to-one and column value references.
The change additionally will defer updating a backref on the "old" value
in the less common case that the "old" value is locally present in the
session, but isn't loaded on the relationship in question, until the
next flush occurs. If this causes an issue, again the normal
:paramref:`_orm.relationship.active_history` flag can be set to ``True``
on the relationship.
A private flag which restores the old value is retained for now,
as support within relevant test suites to exercise the old and
new behaviors together. This is so that if the behavioral change
produces problems we have test harnesses set up to further examine these
behaviors. The "legacy" style can go away in 2.0 or in a much later
1.4 release.
Mike Bayer [Mon, 5 Jul 2021 18:37:58 +0000 (14:37 -0400)]
Add additional support to honor _proxy_key in Core selects
Fixed ORM regression where ad-hoc label names generated for hybrid
properties and potentially other similar types of ORM-enabled expressions
would usually be propagated outwards through subqueries, allowing the name
to be retained in the final keys of the result set even when selecting from
subqueries. Additional state is now tracked in this case that isn't lost
when a hybrid is selected out of a Core select / subquery.
as we have removed things like column.label() from
ORM, since we now have to export the cols with the same names
as what we will render, experiment with giving a greater role
to the _proxy_key annotation so that a desired name can be
carried through more transarently.
Mike Bayer [Wed, 30 Jun 2021 20:04:07 +0000 (16:04 -0400)]
clear new Query._memoized_select_entities in _from_selectable
Fixed regression caused in 1.4.19 due to #6503 and related involving
:meth:`_orm.Query.with_entities` where the new structure used would be
inappropriately transferred to an enclosing :class:`_orm.Query` when making
use of set operations such as :meth:`_orm.Query.union`, causing the JOIN
instructions within to be applied to the outside query as well.
Mike Bayer [Wed, 30 Jun 2021 14:52:09 +0000 (10:52 -0400)]
Ensure compiler uses quote_schema hook for translates renders
Fixed regression where the special dotted-schema name handling for the SQL
Server dialect would not function correctly if the dotted schema name were
used within the ``schema_translate_map`` feature.
Mike Bayer [Wed, 30 Jun 2021 13:22:00 +0000 (09:22 -0400)]
apply quoting to "ON CONSTRAINT" symbol
Fixed issue in :meth:`_postgresql.Insert.on_conflict_do_nothing` and
:meth:`_postgresql.Insert.on_conflict_do_update` where the name of a unique
constraint passed as the ``constraint`` parameter would not be properly
quoted if it contained characters which required quoting.
Miłosz Stypiński [Thu, 24 Jun 2021 16:21:30 +0000 (12:21 -0400)]
Be less fiscal regarding validators functions
Adjusted the check in the mapper for a callable object that is used as a
``@validates`` validator function or a ``@reconstructor`` reconstruction
function, to check for "callable" more liberally such as to accommodate
objects based on fundamental attributes like ``__func__`` and
``__call___``, rather than testing for ``MethodType`` / ``FunctionType``,
allowing things like cython functions to work properly. Pull request
courtesy Miłosz Stypiński.
Kai Mueller [Thu, 24 Jun 2021 15:57:20 +0000 (11:57 -0400)]
Fix missing None handling of Table.prefixes
Fixed issue where passing ``None`` for the value of
:paramref:`_schema.Table.prefixes` would not store an empty list, but
rather the constant ``None``, which may be unexpected by third party
dialects. The issue is revealed by a usage in recent versions of Alembic
that are passing ``None`` for this value. Pull request courtesy Kai
Mueller.
Daniël van Eeden [Wed, 23 Jun 2021 14:00:43 +0000 (10:00 -0400)]
Don't require space after MySQL pk comment
Made a small adjustment in the table reflection feature of the MySQL
dialect to accommodate for alternate MySQL-oriented databases such as TiDB
which include their own "comment" directives at the end of a constraint
directive within "CREATE TABLE" where the format doesn't have the
additional space character after the comment, in this case the TiDB
"clustered index" feature. Pull request courtesy Daniël van Eeden.
Mike Bayer [Sat, 26 Jun 2021 20:40:09 +0000 (16:40 -0400)]
ensure with poly entities are also reconstituted for GC'ed AC
Fixed regression in ORM regarding an internal reconstitution step for the
func:`_orm.with_polymorphic` construct, when the user-facing object is
garbage collected as the query is processed. The reconstitution was not
ensuring the sub-entities for the "polymorphic" case were handled, leading
to an ``AttributeError``.
Mike Bayer [Fri, 25 Jun 2021 20:10:01 +0000 (16:10 -0400)]
set _render_for_subquery for legacy set ops
Adjusted :meth:`_orm.Query.union` and similar set operations to be
correctly compatible with the new capabilities just added in
:ticket:`6661`, with SQLAlchemy 1.4.19, such that the SELECT statements
rendered as elements of the UNION or other set operation will include
directly mapped columns that are mapped as deferred; this both fixes a
regression involving unions with multiple levels of nesting that would
produce a column mismatch, and also allows the :func:`_orm.undefer` option
to be used at the top level of such a :class:`_orm.Query` without having to
apply the option to each of the elements within the UNION.
Mike Bayer [Sat, 26 Jun 2021 13:55:00 +0000 (09:55 -0400)]
turn pg provision error into a warning
We haven't had any real cases of the PG "cant drop tables"
condition since this error was first introduced; instead we
seem to get it for a non-critical query during pool reconnect
tests, and I have not been able to isolate what is causing it.
Therefore turn the error into a new class of warning that can
emit within the test suite without failing the test, so that
if we do get a real PG drop timeout, the warning will be there
to show us what the query was in which it was stuck.
Mike Bayer [Fri, 25 Jun 2021 20:51:50 +0000 (16:51 -0400)]
have automap suppress overlaps warning for mapped secondary
Fixed regression in :mod:`sqlalchemy.ext.automap` extension such that the
use case of creating an explicit mapped class to a table that is also the
:paramref:`_orm.relationship.secondary` element of a
:func:`_orm.relationship` that automap will be generating would emit the
"overlaps" warnings introduced in 1.4 and discussed at :ref:`error_qzyx`.
While generating this case from automap is still subject to the same
caveats that the "overlaps" warning refers towards, as automap is intended
for more ad-hoc use cases, the condition which produces the warning is
disabled when a many-to-many relationship with this particular pattern is
generated.
jason3gb [Thu, 24 Jun 2021 16:11:04 +0000 (12:11 -0400)]
Add "impl" parameter to PickleType
Add a impl parameter to :class:`_types.PickleType` constructor, allowing
any arbitary type to be used in place of the default implementation of
:class:`_types.LargeBinary`. Pull request courtesy jason3gb.
Mike Bayer [Thu, 24 Jun 2021 13:12:31 +0000 (09:12 -0400)]
Use Py_TPFLAGS_HAVE_GC for Row
Fixed an issue in the C extension for the :class:`_result.Row` class which
could lead to a memory leak in the unlikely case of a :class:`_result.Row`
object which referred to an ORM object that then was mutated to refer back
to the ``Row`` itself, creating a cycle. The Python C APIs for tracking GC
cycles has been added to the native :class:`_result.Row` implementation to
accommodate for this case.
Mike Bayer [Wed, 23 Jun 2021 20:34:05 +0000 (16:34 -0400)]
Add Executable to DefaultGenerator
Fixed the class hierarchy for the :class:`_schema.Sequence` and the more
general :class:`_schema.DefaultGenerator` base, as these are "executable"
as statements they need to include :class:`_sql.Executable` in their
hierarchy, not just :class:`_roles.StatementRole` as was applied
arbitrarily to :class:`_schema.Sequence` previously. The fix allows
:class:`_schema.Sequence` to work in all ``.execute()`` methods including
with :meth:`_orm.Session.execute` which was not working in the case that a
``do_orm_execute()`` handler was also established.
Mike Bayer [Wed, 23 Jun 2021 12:50:48 +0000 (08:50 -0400)]
consider "*" col as textual ordered
Fixed old issue where a :func:`_sql.select()` made against the token "*",
which then yielded exactly one column, would fail to correctly organize the
``cursor.description`` column name into the keys of the result object.
MajorDallas [Tue, 22 Jun 2021 19:34:09 +0000 (15:34 -0400)]
Add impl property to PostgreSQL / Oracle INTERVAL class
Fixed issue where the ``INTERVAL`` datatype on PostgreSQL and Oracle would
produce an ``AttributeError`` when used in the context of a comparison
operation against a ``timedelta()`` object. Pull request courtesy
MajorDallas.
Mike Bayer [Tue, 22 Jun 2021 17:27:18 +0000 (13:27 -0400)]
Export deferred columns but not col props; fix CTE labeling
Refined the behavior of ORM subquery rendering with regards to deferred
columns and column properties to be more compatible with that of 1.3 while
also providing for 1.4's newer features. As a subquery in 1.4 does not make
use of loader options, including :func:`_orm.deferred`, a subquery that is
against an ORM entity with deferred attributes will now render those
deferred attributes that refer directly to mapped table columns, as these
are needed in the outer SELECT if that outer SELECT makes use of these
columns; however a deferred attribute that refers to a composed SQL
expression as we normally do with :func:`_orm.column_property` will not be
part of the subquery, as these can be selected explicitly if needed in the
subquery. If the entity is being SELECTed from this subquery, the column
expression can still render on "the outside" in terms of the derived
subquery columns. This produces essentially the same behavior as when
working with 1.3. However in this case the fix has to also make sure that
the ``.selected_columns`` collection of an ORM-enabled :func:`_sql.select`
also follows these rules, which in particular allows recursive CTEs to
render correctly in this scenario, which were previously failing to render
correctly due to this issue.
As part of this change the _exported_columns_iterator() method has been
removed and logic simplified to use ._all_selected_columns from any
SelectBase object where _exported_columns_iterator() was used before.
Additionally sets up UpdateBase to include ReturnsRows in its hierarchy;
the literal point of ReturnsRows was to be a common base for UpdateBase
and SelectBase so it was kind of weird it wasn't there.
Fixes: #6661
Fixed issue in CTE constructs mostly relevant to ORM use cases where a
recursive CTE against "anonymous" labels such as those seen in ORM
``column_property()`` mappings would render in the
``WITH RECURSIVE xyz(...)`` section as their raw internal label and not a
cleanly anonymized name.
Mike Bayer [Mon, 21 Jun 2021 22:13:55 +0000 (18:13 -0400)]
apply render_schema_translates to identity insert directives
Fixed bug where the "schema_translate_map" feature would fail to function
correctly in conjunction with an INSERT into a table that has an IDENTITY
column, where the value of the IDENTITY column were specified in the values
of the INSERT thus triggering SQLAlchemy's feature of setting IDENTITY
INSERT to "on"; it's in this directive where the schema translate map would
fail to be honored.
Anton Kovalevich [Fri, 18 Jun 2021 14:33:48 +0000 (10:33 -0400)]
Implement MySQL-specific MATCH
Added new construct :class:`_mysql.match`, which provides for the full
range of MySQL's MATCH operator including multiple column support and
modifiers. Pull request courtesy Anton Kovalevich.
Mike Bayer [Mon, 21 Jun 2021 17:08:59 +0000 (13:08 -0400)]
ensure test has deterministic FROM rendering
test_options_entities_replaced_with_equivs_three did not
have deterministic FROM ordering, so adding an inner
join from user->address should ensure there's a single
FROM element that is a series of joins.
Mike Bayer [Mon, 21 Jun 2021 16:52:42 +0000 (12:52 -0400)]
accommodate no cls info found in _scan_declarative
Fixed issue in mypy plugin where class info for a custom declarative base
would not be handled correctly on a cached mypy pass, leading to an
AssertionError being raised.
Mike Bayer [Fri, 18 Jun 2021 21:42:53 +0000 (17:42 -0400)]
ensure greenlet_spawn propagates BaseException
Fixed bug in asyncio implementation where the greenlet adaptation system
failed to propagate ``BaseException`` subclasses, most notably including
``asyncio.CancelledError``, to the exception handling logic used by the
engine to invalidate and clean up the connection, thus preventing
connections from being correctly disposed when a task was cancelled.
Abhishek Jog [Wed, 16 Jun 2021 19:15:24 +0000 (15:15 -0400)]
Added netezza in external dialects in documentation
### Description
IBM Netezza Performance Server (aka Netezza, NPS) now has fully function SQLAlchemy dialect. Need to add it in SQLAlchemy 1.3 documentation.
This PR fixes https://github.com/sqlalchemy/sqlalchemy/issues/6609
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Tue, 15 Jun 2021 19:13:34 +0000 (15:13 -0400)]
memoize current options and joins w with_entities/with_only_cols
Fixed further regressions in the same area as that of :ticket:`6052` where
loader options as well as invocations of methods like
:meth:`_orm.Query.join` would fail if the left side of the statement for
which the option/join depends upon were replaced by using the
:meth:`_orm.Query.with_entities` method, or when using 2.0 style queries
when using the :meth:`_sql.Select.with_only_columns` method. A new set of
state has been added to the objects which tracks the "left" entities that
the options / join were made against which is memoized when the lead
entities are changed.
jason3gb [Wed, 16 Jun 2021 14:18:08 +0000 (10:18 -0400)]
Implement async_scoped_session
Implemented :class:`_asyncio.async_scoped_session` to address some
asyncio-related incompatibilities between :class:`_orm.scoped_session` and
:class:`_asyncio.AsyncSession`, in which some methods (notably the
:meth:`_asyncio.async_scoped_session.remove` method) should be used with
the ``await`` keyword.
Mike Bayer [Fri, 11 Jun 2021 14:23:51 +0000 (10:23 -0400)]
set autocommit for psycopg2 pre-ping
Fixed issue where the pool "pre ping" feature would implicitly start a
transaction, which would then interfere with custom transactional flags
such as PostgreSQL's "read only" mode when used with the psycopg2 driver.
Mike Bayer [Tue, 8 Jun 2021 19:43:13 +0000 (15:43 -0400)]
simplify relationship caching options
Clarified the current purpose of the
:paramref:`_orm.relationship.bake_queries` flag, which in 1.4 is to enable
or disable "lambda caching" of statements within the "lazyload" and
"selectinload" loader strategies; this is separate from the more
foundational SQL query cache that is used for most statements.
Additionally, the lazy loader no longer uses its own cache for many-to-one
SQL queries, which was an implementation quirk that doesn't exist for any
other loader scenario. Finally, the "lru cache" warning that the lazyloader
and selectinloader strategies could emit when handling a wide array of
class/relationship combinations has been removed; based on analysis of some
end-user cases, this warning doesn't suggest any significant issue. While
setting ``bake_queries=False`` for such a relationship will remove this
cache from being used, there's no particular performance gain in this case
as using no caching vs. using a cache that needs to refresh often likely
still wins out on the caching being used side.
Propagate asyncio flag from the dialect to selected pool classes
Fixed an issue that presented itself when using the :class:`_pool.NullPool`
or the :class:`_pool.StaticPool` with an async engine. This mostly affected
the aiosqlite dialect.
Mike Bayer [Mon, 7 Jun 2021 22:49:04 +0000 (18:49 -0400)]
restore adapter logic in ORM loading
Fixed regression involving how the ORM would resolve a given mapped column
to a result row, where under cases such as joined eager loading, a slightly
more expensive "fallback" could take place to set up this resolution due to
some logic that was removed since 1.3. The issue could also cause
deprecation warnings involving column resolution to be emitted when using a
1.4 style query with joined eager loading.
In order to ensure we don't look up columns by string name in the ORM,
we've turned on future_result=True in all cases, which I thought was
already the assumption here, but apparently not. That in turn
led to the issue that Session autocommit relies on close_with_result=True,
which is legacy result only. This was also hard to figure out.
So a new exception is raised if one tries to use future_result=True
along with close_with_result, and the Session now has an explicit path
for "autocommit" that sets these flags to their legacy values.
This does leave the possibility of some of these fallback cases
emitting warnings for users using session in autocommit along with
joined inheritance and column properties, as this patch identifies
that joined inheritance + column properties produce the fallback
logic when looking up in the result via the adapted column, which
in those tests is actually a Label object that doesn't adapt
nicely.
Mike Bayer [Mon, 7 Jun 2021 21:47:07 +0000 (17:47 -0400)]
repair Join.is_derived_from() to not rely on simple identity
Fixed issue where query production for joinedload against a complex left
hand side involving joined-table inheritance could fail to produce a
correct query, due to a clause adaption issue.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
