Mike Bayer [Fri, 2 Jul 2021 15:23:20 +0000 (11:23 -0400)]
implement deferred scalarobject history load
Modified the approach used for history tracking of scalar object
relationships that are not many-to-one, i.e. one-to-one relationships that
would otherwise be one-to-many. When replacing a one-to-one value, the
"old" value that would be replaced is no longer loaded immediately, and is
instead handled during the flush process. This eliminates an historically
troublesome lazy load that otherwise often occurs when assigning to a
one-to-one attribute, and is particularly troublesome when using
"lazy='raise'" as well as asyncio use cases.
This change does cause a behavioral change within the
:meth:`_orm.AttributeEvents.set` event, which is nonetheless currently
documented, which is that the event applied to such a one-to-one attribute
will no longer receive the "old" parameter if it is unloaded and the
:paramref:`_orm.relationship.active_history` flag is not set. As is
documented in :meth:`_orm.AttributeEvents.set`, if the event handler needs
to receive the "old" value when the event fires off, the active_history
flag must be established either with the event listener or with the
relationship. This is already the behavior with other kinds of attributes
such as many-to-one and column value references.
The change additionally will defer updating a backref on the "old" value
in the less common case that the "old" value is locally present in the
session, but isn't loaded on the relationship in question, until the
next flush occurs. If this causes an issue, again the normal
:paramref:`_orm.relationship.active_history` flag can be set to ``True``
on the relationship.
A private flag which restores the old value is retained for now,
as support within relevant test suites to exercise the old and
new behaviors together. This is so that if the behavioral change
produces problems we have test harnesses set up to further examine these
behaviors. The "legacy" style can go away in 2.0 or in a much later
1.4 release.
Mike Bayer [Wed, 30 Jun 2021 20:04:07 +0000 (16:04 -0400)]
clear new Query._memoized_select_entities in _from_selectable
Fixed regression caused in 1.4.19 due to #6503 and related involving
:meth:`_orm.Query.with_entities` where the new structure used would be
inappropriately transferred to an enclosing :class:`_orm.Query` when making
use of set operations such as :meth:`_orm.Query.union`, causing the JOIN
instructions within to be applied to the outside query as well.
Mike Bayer [Wed, 30 Jun 2021 14:52:09 +0000 (10:52 -0400)]
Ensure compiler uses quote_schema hook for translates renders
Fixed regression where the special dotted-schema name handling for the SQL
Server dialect would not function correctly if the dotted schema name were
used within the ``schema_translate_map`` feature.
Mike Bayer [Wed, 30 Jun 2021 13:22:00 +0000 (09:22 -0400)]
apply quoting to "ON CONSTRAINT" symbol
Fixed issue in :meth:`_postgresql.Insert.on_conflict_do_nothing` and
:meth:`_postgresql.Insert.on_conflict_do_update` where the name of a unique
constraint passed as the ``constraint`` parameter would not be properly
quoted if it contained characters which required quoting.
Miłosz Stypiński [Thu, 24 Jun 2021 16:21:30 +0000 (12:21 -0400)]
Be less fiscal regarding validators functions
Adjusted the check in the mapper for a callable object that is used as a
``@validates`` validator function or a ``@reconstructor`` reconstruction
function, to check for "callable" more liberally such as to accommodate
objects based on fundamental attributes like ``__func__`` and
``__call___``, rather than testing for ``MethodType`` / ``FunctionType``,
allowing things like cython functions to work properly. Pull request
courtesy Miłosz Stypiński.
Kai Mueller [Thu, 24 Jun 2021 15:57:20 +0000 (11:57 -0400)]
Fix missing None handling of Table.prefixes
Fixed issue where passing ``None`` for the value of
:paramref:`_schema.Table.prefixes` would not store an empty list, but
rather the constant ``None``, which may be unexpected by third party
dialects. The issue is revealed by a usage in recent versions of Alembic
that are passing ``None`` for this value. Pull request courtesy Kai
Mueller.
Daniël van Eeden [Wed, 23 Jun 2021 14:00:43 +0000 (10:00 -0400)]
Don't require space after MySQL pk comment
Made a small adjustment in the table reflection feature of the MySQL
dialect to accommodate for alternate MySQL-oriented databases such as TiDB
which include their own "comment" directives at the end of a constraint
directive within "CREATE TABLE" where the format doesn't have the
additional space character after the comment, in this case the TiDB
"clustered index" feature. Pull request courtesy Daniël van Eeden.
Mike Bayer [Sat, 26 Jun 2021 20:40:09 +0000 (16:40 -0400)]
ensure with poly entities are also reconstituted for GC'ed AC
Fixed regression in ORM regarding an internal reconstitution step for the
func:`_orm.with_polymorphic` construct, when the user-facing object is
garbage collected as the query is processed. The reconstitution was not
ensuring the sub-entities for the "polymorphic" case were handled, leading
to an ``AttributeError``.
Mike Bayer [Fri, 25 Jun 2021 20:10:01 +0000 (16:10 -0400)]
set _render_for_subquery for legacy set ops
Adjusted :meth:`_orm.Query.union` and similar set operations to be
correctly compatible with the new capabilities just added in
:ticket:`6661`, with SQLAlchemy 1.4.19, such that the SELECT statements
rendered as elements of the UNION or other set operation will include
directly mapped columns that are mapped as deferred; this both fixes a
regression involving unions with multiple levels of nesting that would
produce a column mismatch, and also allows the :func:`_orm.undefer` option
to be used at the top level of such a :class:`_orm.Query` without having to
apply the option to each of the elements within the UNION.
Mike Bayer [Sat, 26 Jun 2021 13:55:00 +0000 (09:55 -0400)]
turn pg provision error into a warning
We haven't had any real cases of the PG "cant drop tables"
condition since this error was first introduced; instead we
seem to get it for a non-critical query during pool reconnect
tests, and I have not been able to isolate what is causing it.
Therefore turn the error into a new class of warning that can
emit within the test suite without failing the test, so that
if we do get a real PG drop timeout, the warning will be there
to show us what the query was in which it was stuck.
Mike Bayer [Fri, 25 Jun 2021 20:51:50 +0000 (16:51 -0400)]
have automap suppress overlaps warning for mapped secondary
Fixed regression in :mod:`sqlalchemy.ext.automap` extension such that the
use case of creating an explicit mapped class to a table that is also the
:paramref:`_orm.relationship.secondary` element of a
:func:`_orm.relationship` that automap will be generating would emit the
"overlaps" warnings introduced in 1.4 and discussed at :ref:`error_qzyx`.
While generating this case from automap is still subject to the same
caveats that the "overlaps" warning refers towards, as automap is intended
for more ad-hoc use cases, the condition which produces the warning is
disabled when a many-to-many relationship with this particular pattern is
generated.
jason3gb [Thu, 24 Jun 2021 16:11:04 +0000 (12:11 -0400)]
Add "impl" parameter to PickleType
Add a impl parameter to :class:`_types.PickleType` constructor, allowing
any arbitary type to be used in place of the default implementation of
:class:`_types.LargeBinary`. Pull request courtesy jason3gb.
Mike Bayer [Thu, 24 Jun 2021 13:12:31 +0000 (09:12 -0400)]
Use Py_TPFLAGS_HAVE_GC for Row
Fixed an issue in the C extension for the :class:`_result.Row` class which
could lead to a memory leak in the unlikely case of a :class:`_result.Row`
object which referred to an ORM object that then was mutated to refer back
to the ``Row`` itself, creating a cycle. The Python C APIs for tracking GC
cycles has been added to the native :class:`_result.Row` implementation to
accommodate for this case.
Mike Bayer [Wed, 23 Jun 2021 20:34:05 +0000 (16:34 -0400)]
Add Executable to DefaultGenerator
Fixed the class hierarchy for the :class:`_schema.Sequence` and the more
general :class:`_schema.DefaultGenerator` base, as these are "executable"
as statements they need to include :class:`_sql.Executable` in their
hierarchy, not just :class:`_roles.StatementRole` as was applied
arbitrarily to :class:`_schema.Sequence` previously. The fix allows
:class:`_schema.Sequence` to work in all ``.execute()`` methods including
with :meth:`_orm.Session.execute` which was not working in the case that a
``do_orm_execute()`` handler was also established.
Mike Bayer [Wed, 23 Jun 2021 12:50:48 +0000 (08:50 -0400)]
consider "*" col as textual ordered
Fixed old issue where a :func:`_sql.select()` made against the token "*",
which then yielded exactly one column, would fail to correctly organize the
``cursor.description`` column name into the keys of the result object.
MajorDallas [Tue, 22 Jun 2021 19:34:09 +0000 (15:34 -0400)]
Add impl property to PostgreSQL / Oracle INTERVAL class
Fixed issue where the ``INTERVAL`` datatype on PostgreSQL and Oracle would
produce an ``AttributeError`` when used in the context of a comparison
operation against a ``timedelta()`` object. Pull request courtesy
MajorDallas.
Mike Bayer [Tue, 22 Jun 2021 17:27:18 +0000 (13:27 -0400)]
Export deferred columns but not col props; fix CTE labeling
Refined the behavior of ORM subquery rendering with regards to deferred
columns and column properties to be more compatible with that of 1.3 while
also providing for 1.4's newer features. As a subquery in 1.4 does not make
use of loader options, including :func:`_orm.deferred`, a subquery that is
against an ORM entity with deferred attributes will now render those
deferred attributes that refer directly to mapped table columns, as these
are needed in the outer SELECT if that outer SELECT makes use of these
columns; however a deferred attribute that refers to a composed SQL
expression as we normally do with :func:`_orm.column_property` will not be
part of the subquery, as these can be selected explicitly if needed in the
subquery. If the entity is being SELECTed from this subquery, the column
expression can still render on "the outside" in terms of the derived
subquery columns. This produces essentially the same behavior as when
working with 1.3. However in this case the fix has to also make sure that
the ``.selected_columns`` collection of an ORM-enabled :func:`_sql.select`
also follows these rules, which in particular allows recursive CTEs to
render correctly in this scenario, which were previously failing to render
correctly due to this issue.
As part of this change the _exported_columns_iterator() method has been
removed and logic simplified to use ._all_selected_columns from any
SelectBase object where _exported_columns_iterator() was used before.
Additionally sets up UpdateBase to include ReturnsRows in its hierarchy;
the literal point of ReturnsRows was to be a common base for UpdateBase
and SelectBase so it was kind of weird it wasn't there.
Fixes: #6661
Fixed issue in CTE constructs mostly relevant to ORM use cases where a
recursive CTE against "anonymous" labels such as those seen in ORM
``column_property()`` mappings would render in the
``WITH RECURSIVE xyz(...)`` section as their raw internal label and not a
cleanly anonymized name.
Mike Bayer [Mon, 21 Jun 2021 22:13:55 +0000 (18:13 -0400)]
apply render_schema_translates to identity insert directives
Fixed bug where the "schema_translate_map" feature would fail to function
correctly in conjunction with an INSERT into a table that has an IDENTITY
column, where the value of the IDENTITY column were specified in the values
of the INSERT thus triggering SQLAlchemy's feature of setting IDENTITY
INSERT to "on"; it's in this directive where the schema translate map would
fail to be honored.
Anton Kovalevich [Fri, 18 Jun 2021 14:33:48 +0000 (10:33 -0400)]
Implement MySQL-specific MATCH
Added new construct :class:`_mysql.match`, which provides for the full
range of MySQL's MATCH operator including multiple column support and
modifiers. Pull request courtesy Anton Kovalevich.
Mike Bayer [Mon, 21 Jun 2021 17:08:59 +0000 (13:08 -0400)]
ensure test has deterministic FROM rendering
test_options_entities_replaced_with_equivs_three did not
have deterministic FROM ordering, so adding an inner
join from user->address should ensure there's a single
FROM element that is a series of joins.
Mike Bayer [Mon, 21 Jun 2021 16:52:42 +0000 (12:52 -0400)]
accommodate no cls info found in _scan_declarative
Fixed issue in mypy plugin where class info for a custom declarative base
would not be handled correctly on a cached mypy pass, leading to an
AssertionError being raised.
Mike Bayer [Fri, 18 Jun 2021 21:42:53 +0000 (17:42 -0400)]
ensure greenlet_spawn propagates BaseException
Fixed bug in asyncio implementation where the greenlet adaptation system
failed to propagate ``BaseException`` subclasses, most notably including
``asyncio.CancelledError``, to the exception handling logic used by the
engine to invalidate and clean up the connection, thus preventing
connections from being correctly disposed when a task was cancelled.
Abhishek Jog [Wed, 16 Jun 2021 19:15:24 +0000 (15:15 -0400)]
Added netezza in external dialects in documentation
### Description
IBM Netezza Performance Server (aka Netezza, NPS) now has fully function SQLAlchemy dialect. Need to add it in SQLAlchemy 1.3 documentation.
This PR fixes https://github.com/sqlalchemy/sqlalchemy/issues/6609
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Tue, 15 Jun 2021 19:13:34 +0000 (15:13 -0400)]
memoize current options and joins w with_entities/with_only_cols
Fixed further regressions in the same area as that of :ticket:`6052` where
loader options as well as invocations of methods like
:meth:`_orm.Query.join` would fail if the left side of the statement for
which the option/join depends upon were replaced by using the
:meth:`_orm.Query.with_entities` method, or when using 2.0 style queries
when using the :meth:`_sql.Select.with_only_columns` method. A new set of
state has been added to the objects which tracks the "left" entities that
the options / join were made against which is memoized when the lead
entities are changed.
jason3gb [Wed, 16 Jun 2021 14:18:08 +0000 (10:18 -0400)]
Implement async_scoped_session
Implemented :class:`_asyncio.async_scoped_session` to address some
asyncio-related incompatibilities between :class:`_orm.scoped_session` and
:class:`_asyncio.AsyncSession`, in which some methods (notably the
:meth:`_asyncio.async_scoped_session.remove` method) should be used with
the ``await`` keyword.
Mike Bayer [Fri, 11 Jun 2021 14:23:51 +0000 (10:23 -0400)]
set autocommit for psycopg2 pre-ping
Fixed issue where the pool "pre ping" feature would implicitly start a
transaction, which would then interfere with custom transactional flags
such as PostgreSQL's "read only" mode when used with the psycopg2 driver.
Mike Bayer [Tue, 8 Jun 2021 19:43:13 +0000 (15:43 -0400)]
simplify relationship caching options
Clarified the current purpose of the
:paramref:`_orm.relationship.bake_queries` flag, which in 1.4 is to enable
or disable "lambda caching" of statements within the "lazyload" and
"selectinload" loader strategies; this is separate from the more
foundational SQL query cache that is used for most statements.
Additionally, the lazy loader no longer uses its own cache for many-to-one
SQL queries, which was an implementation quirk that doesn't exist for any
other loader scenario. Finally, the "lru cache" warning that the lazyloader
and selectinloader strategies could emit when handling a wide array of
class/relationship combinations has been removed; based on analysis of some
end-user cases, this warning doesn't suggest any significant issue. While
setting ``bake_queries=False`` for such a relationship will remove this
cache from being used, there's no particular performance gain in this case
as using no caching vs. using a cache that needs to refresh often likely
still wins out on the caching being used side.
Propagate asyncio flag from the dialect to selected pool classes
Fixed an issue that presented itself when using the :class:`_pool.NullPool`
or the :class:`_pool.StaticPool` with an async engine. This mostly affected
the aiosqlite dialect.
Mike Bayer [Mon, 7 Jun 2021 22:49:04 +0000 (18:49 -0400)]
restore adapter logic in ORM loading
Fixed regression involving how the ORM would resolve a given mapped column
to a result row, where under cases such as joined eager loading, a slightly
more expensive "fallback" could take place to set up this resolution due to
some logic that was removed since 1.3. The issue could also cause
deprecation warnings involving column resolution to be emitted when using a
1.4 style query with joined eager loading.
In order to ensure we don't look up columns by string name in the ORM,
we've turned on future_result=True in all cases, which I thought was
already the assumption here, but apparently not. That in turn
led to the issue that Session autocommit relies on close_with_result=True,
which is legacy result only. This was also hard to figure out.
So a new exception is raised if one tries to use future_result=True
along with close_with_result, and the Session now has an explicit path
for "autocommit" that sets these flags to their legacy values.
This does leave the possibility of some of these fallback cases
emitting warnings for users using session in autocommit along with
joined inheritance and column properties, as this patch identifies
that joined inheritance + column properties produce the fallback
logic when looking up in the result via the adapted column, which
in those tests is actually a Label object that doesn't adapt
nicely.
Mike Bayer [Mon, 7 Jun 2021 21:47:07 +0000 (17:47 -0400)]
repair Join.is_derived_from() to not rely on simple identity
Fixed issue where query production for joinedload against a complex left
hand side involving joined-table inheritance could fail to produce a
correct query, due to a clause adaption issue.
Mike Bayer [Mon, 7 Jun 2021 15:28:49 +0000 (11:28 -0400)]
Add asyncio.TimeoutError as an exit exception
Added ``asyncio.exceptions.TimeoutError``,
``asyncio.exceptions.CancelledError`` as so-called "exit exceptions", a
class of exceptions that include things like ``GreenletExit`` and
``KeyboardInterrupt``, which are considered to be events that warrant
considering a DBAPI connection to be in an unusable state where it should
be recycled.
Mike Bayer [Mon, 7 Jun 2021 13:21:25 +0000 (09:21 -0400)]
init extra_criteria_entities in fromstatement w/ DML
Fixed issue in experimental "select ORM objects from INSERT/UPDATE" use
case where an error was raised if the statement were against a
single-table-inheritance subclass.
Additionally makes some adjustments in the SQL assertion
fixture to test a FromStatement w/ DML.
Mike Bayer [Sun, 6 Jun 2021 14:12:05 +0000 (10:12 -0400)]
Pass URL object, not the string, to on_connect_url
The fix for pysqlcipher released in version 1.4.3 :ticket:`5848` was
unfortunately non-working, in that the new ``on_connect_url`` hook was
erroneously not receiving a ``URL`` object under normal usage of
:func:`_sa.create_engine` and instead received a string that was unhandled;
the test suite failed to fully set up the actual conditions under which
this hook is called. This has been fixed.
Mike Bayer [Fri, 4 Jun 2021 23:45:45 +0000 (19:45 -0400)]
don't enable "fast insert executemany" for ON CONFLICT etc
Fixed issue where using the PostgreSQL "INSERT..ON CONFLICT" structure
would fail to work with the psycopg2 driver if it were used in an
"executemany" context along with bound parameters in the "SET" clause, due
to the implicit use of the psycopg2 fast execution helpers which are not
appropriate for this style of INSERT statement. Additional checks to
exclude this kind of statement from that particular extension have been
added.
Mike Bayer [Thu, 3 Jun 2021 13:37:27 +0000 (09:37 -0400)]
Adjust create_proxy_methods() to use kw arguments
Adjusted the means by which classes such as :class:`_orm.scoped_session`
and :class:`_asyncio.AsyncSession` are generated from the base
:class:`_orm.Session` class, such that custom :class:`_orm.Session`
subclasses such as that used by Flask-SQLAlchemy don't need to implement
positional arguments when they call into the superclass method, and can
continue using the same argument styles as in previous releases.
Mike Bayer [Thu, 3 Jun 2021 03:07:28 +0000 (23:07 -0400)]
delete exception traceback in _expect_raises
a new segfault is observed in python 3.10.0b2 in conjunction
with asyncio and possibly greenlet. Ensuring the
traceback object is deleted from scope here, which is a
good idea anyway, apparently seems to resolve the issue.
Mike Bayer [Wed, 2 Jun 2021 22:24:43 +0000 (18:24 -0400)]
temporarily disable test_no_attach_to_event_loop
this test currently causes the test suite to hang; it previously
was not actually running the worker thread
as the testing_engine() fixture
was rejecting the "transfer_staticpool" keyword argument.
as we seem to have a greenlet-related segfault in 3.10.0b2 I am
going to have to get the greenlet devs to run the test suite
so i want to get anything not totally smooth out of it for the
moment.
Mike Bayer [Wed, 2 Jun 2021 16:23:31 +0000 (12:23 -0400)]
Implement proxy back reference system for asyncio
Implemented a new registry architecture that allows the ``Async`` version
of an object, like ``AsyncSession``, ``AsyncConnection``, etc., to be
locatable given the proxied "sync" object, i.e. ``Session``,
``Connection``. Previously, to the degree such lookup functions were used,
an ``Async`` object would be re-created each time, which was less than
ideal as the identity and state of the "async" object would not be
preserved across calls.
From there, new helper functions :func:`_asyncio.async_object_session`,
:func:`_asyncio.async_session` as well as a new :class:`_orm.InstanceState`
attribute :attr:`_orm.InstanceState.asyncio_session` have been added, which
are used to retrieve the original :class:`_asyncio.AsyncSession` associated
with an ORM mapped object, a :class:`_orm.Session` associated with an
:class:`_asyncio.AsyncSession`, and an :class:`_asyncio.AsyncSession`
associated with an :class:`_orm.InstanceState`, respectively.
This patch also implements new methods
:meth:`_asyncio.AsyncSession.in_nested_transaction`,
:meth:`_asyncio.AsyncSession.get_transaction`,
:meth:`_asyncio.AsyncSession.get_nested_transaction`.
Mike Bayer [Tue, 1 Jun 2021 20:03:52 +0000 (16:03 -0400)]
improve "overlaps" warning; test for m2m
The warning that's emitted for :func:`_orm.relationship` when multiple
relationships would overlap with each other as far as foreign key
attributes written towards, now includes the specific "overlaps" argument
to use for each warning in order to silence the warning without changing
the mapping.
Mike Bayer [Sat, 29 May 2021 15:05:25 +0000 (11:05 -0400)]
skip dupe order by test on mssql
SQL Server wants to complain if ORDER BY specifies the same
expression more than once, so just skip this test added in aaba0650d7410f579b2c14f8f1b0680a1d1852c4 as this only needs
to test ORM functionality.
Mike Bayer [Sat, 29 May 2021 14:05:20 +0000 (10:05 -0400)]
Ensure propagate_attrs available on PropComparator
Fixed regression caused by just-released performance fix mentioned in #6550
where a query.join() to a relationship could produce an AttributeError if
the query were made against non-ORM structures only, a fairly unusual
calling pattern.
In this fix, since we are no longer going through the production
of ``__clause_element__()`` for Cls.some_relationship, I assumed we
just throw this object away completely but I missed the one little
bit where we might be getting ``_propagate_attrs`` from it.
So we implement ``_propagate_attrs`` on ``PropComparator`` as well,
since this is easy to define.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
