Mike Bayer [Tue, 8 Jun 2021 19:43:13 +0000 (15:43 -0400)]
simplify relationship caching options
Clarified the current purpose of the
:paramref:`_orm.relationship.bake_queries` flag, which in 1.4 is to enable
or disable "lambda caching" of statements within the "lazyload" and
"selectinload" loader strategies; this is separate from the more
foundational SQL query cache that is used for most statements.
Additionally, the lazy loader no longer uses its own cache for many-to-one
SQL queries, which was an implementation quirk that doesn't exist for any
other loader scenario. Finally, the "lru cache" warning that the lazyloader
and selectinloader strategies could emit when handling a wide array of
class/relationship combinations has been removed; based on analysis of some
end-user cases, this warning doesn't suggest any significant issue. While
setting ``bake_queries=False`` for such a relationship will remove this
cache from being used, there's no particular performance gain in this case
as using no caching vs. using a cache that needs to refresh often likely
still wins out on the caching being used side.
Mike Bayer [Mon, 7 Jun 2021 22:49:04 +0000 (18:49 -0400)]
restore adapter logic in ORM loading
Fixed regression involving how the ORM would resolve a given mapped column
to a result row, where under cases such as joined eager loading, a slightly
more expensive "fallback" could take place to set up this resolution due to
some logic that was removed since 1.3. The issue could also cause
deprecation warnings involving column resolution to be emitted when using a
1.4 style query with joined eager loading.
In order to ensure we don't look up columns by string name in the ORM,
we've turned on future_result=True in all cases, which I thought was
already the assumption here, but apparently not. That in turn
led to the issue that Session autocommit relies on close_with_result=True,
which is legacy result only. This was also hard to figure out.
So a new exception is raised if one tries to use future_result=True
along with close_with_result, and the Session now has an explicit path
for "autocommit" that sets these flags to their legacy values.
This does leave the possibility of some of these fallback cases
emitting warnings for users using session in autocommit along with
joined inheritance and column properties, as this patch identifies
that joined inheritance + column properties produce the fallback
logic when looking up in the result via the adapted column, which
in those tests is actually a Label object that doesn't adapt
nicely.
Mike Bayer [Mon, 7 Jun 2021 21:47:07 +0000 (17:47 -0400)]
repair Join.is_derived_from() to not rely on simple identity
Fixed issue where query production for joinedload against a complex left
hand side involving joined-table inheritance could fail to produce a
correct query, due to a clause adaption issue.
Mike Bayer [Mon, 7 Jun 2021 15:28:49 +0000 (11:28 -0400)]
Add asyncio.TimeoutError as an exit exception
Added ``asyncio.exceptions.TimeoutError``,
``asyncio.exceptions.CancelledError`` as so-called "exit exceptions", a
class of exceptions that include things like ``GreenletExit`` and
``KeyboardInterrupt``, which are considered to be events that warrant
considering a DBAPI connection to be in an unusable state where it should
be recycled.
Mike Bayer [Mon, 7 Jun 2021 13:21:25 +0000 (09:21 -0400)]
init extra_criteria_entities in fromstatement w/ DML
Fixed issue in experimental "select ORM objects from INSERT/UPDATE" use
case where an error was raised if the statement were against a
single-table-inheritance subclass.
Additionally makes some adjustments in the SQL assertion
fixture to test a FromStatement w/ DML.
Mike Bayer [Sun, 6 Jun 2021 14:12:05 +0000 (10:12 -0400)]
Pass URL object, not the string, to on_connect_url
The fix for pysqlcipher released in version 1.4.3 :ticket:`5848` was
unfortunately non-working, in that the new ``on_connect_url`` hook was
erroneously not receiving a ``URL`` object under normal usage of
:func:`_sa.create_engine` and instead received a string that was unhandled;
the test suite failed to fully set up the actual conditions under which
this hook is called. This has been fixed.
Mike Bayer [Fri, 4 Jun 2021 23:45:45 +0000 (19:45 -0400)]
don't enable "fast insert executemany" for ON CONFLICT etc
Fixed issue where using the PostgreSQL "INSERT..ON CONFLICT" structure
would fail to work with the psycopg2 driver if it were used in an
"executemany" context along with bound parameters in the "SET" clause, due
to the implicit use of the psycopg2 fast execution helpers which are not
appropriate for this style of INSERT statement. Additional checks to
exclude this kind of statement from that particular extension have been
added.
Mike Bayer [Thu, 3 Jun 2021 13:37:27 +0000 (09:37 -0400)]
Adjust create_proxy_methods() to use kw arguments
Adjusted the means by which classes such as :class:`_orm.scoped_session`
and :class:`_asyncio.AsyncSession` are generated from the base
:class:`_orm.Session` class, such that custom :class:`_orm.Session`
subclasses such as that used by Flask-SQLAlchemy don't need to implement
positional arguments when they call into the superclass method, and can
continue using the same argument styles as in previous releases.
Mike Bayer [Thu, 3 Jun 2021 03:07:28 +0000 (23:07 -0400)]
delete exception traceback in _expect_raises
a new segfault is observed in python 3.10.0b2 in conjunction
with asyncio and possibly greenlet. Ensuring the
traceback object is deleted from scope here, which is a
good idea anyway, apparently seems to resolve the issue.
Mike Bayer [Wed, 2 Jun 2021 22:24:43 +0000 (18:24 -0400)]
temporarily disable test_no_attach_to_event_loop
this test currently causes the test suite to hang; it previously
was not actually running the worker thread
as the testing_engine() fixture
was rejecting the "transfer_staticpool" keyword argument.
as we seem to have a greenlet-related segfault in 3.10.0b2 I am
going to have to get the greenlet devs to run the test suite
so i want to get anything not totally smooth out of it for the
moment.
Mike Bayer [Wed, 2 Jun 2021 16:23:31 +0000 (12:23 -0400)]
Implement proxy back reference system for asyncio
Implemented a new registry architecture that allows the ``Async`` version
of an object, like ``AsyncSession``, ``AsyncConnection``, etc., to be
locatable given the proxied "sync" object, i.e. ``Session``,
``Connection``. Previously, to the degree such lookup functions were used,
an ``Async`` object would be re-created each time, which was less than
ideal as the identity and state of the "async" object would not be
preserved across calls.
From there, new helper functions :func:`_asyncio.async_object_session`,
:func:`_asyncio.async_session` as well as a new :class:`_orm.InstanceState`
attribute :attr:`_orm.InstanceState.asyncio_session` have been added, which
are used to retrieve the original :class:`_asyncio.AsyncSession` associated
with an ORM mapped object, a :class:`_orm.Session` associated with an
:class:`_asyncio.AsyncSession`, and an :class:`_asyncio.AsyncSession`
associated with an :class:`_orm.InstanceState`, respectively.
This patch also implements new methods
:meth:`_asyncio.AsyncSession.in_nested_transaction`,
:meth:`_asyncio.AsyncSession.get_transaction`,
:meth:`_asyncio.AsyncSession.get_nested_transaction`.
Mike Bayer [Tue, 1 Jun 2021 20:03:52 +0000 (16:03 -0400)]
improve "overlaps" warning; test for m2m
The warning that's emitted for :func:`_orm.relationship` when multiple
relationships would overlap with each other as far as foreign key
attributes written towards, now includes the specific "overlaps" argument
to use for each warning in order to silence the warning without changing
the mapping.
Mike Bayer [Sat, 29 May 2021 15:05:25 +0000 (11:05 -0400)]
skip dupe order by test on mssql
SQL Server wants to complain if ORDER BY specifies the same
expression more than once, so just skip this test added in aaba0650d7410f579b2c14f8f1b0680a1d1852c4 as this only needs
to test ORM functionality.
Mike Bayer [Sat, 29 May 2021 14:05:20 +0000 (10:05 -0400)]
Ensure propagate_attrs available on PropComparator
Fixed regression caused by just-released performance fix mentioned in #6550
where a query.join() to a relationship could produce an AttributeError if
the query were made against non-ORM structures only, a fairly unusual
calling pattern.
In this fix, since we are no longer going through the production
of ``__clause_element__()`` for Cls.some_relationship, I assumed we
just throw this object away completely but I missed the one little
bit where we might be getting ``_propagate_attrs`` from it.
So we implement ``_propagate_attrs`` on ``PropComparator`` as well,
since this is easy to define.
Mike Bayer [Fri, 28 May 2021 12:29:24 +0000 (08:29 -0400)]
Fix adaption in AnnotatedLabel; repair needless expense in coercion
Fixed regression involving clause adaption of labeled ORM compound
elements, such as single-table inheritance discriminator expressions with
conditionals or CASE expressions, which could cause aliased expressions
such as those used in ORM join / joinedload operations to not be adapted
correctly, such as referring to the wrong table in the ON clause in a join.
This change also improves a performance bump that was located within the
process of invoking :meth:`_sql.Select.join` given an ORM attribute
as a target.
Mike Bayer [Fri, 28 May 2021 01:15:01 +0000 (21:15 -0400)]
ensure relationship.order_by stored as a tuple; check in dynamic also
Fixed regression in dynamic loader strategy and :func:`_orm.relationship`
overall where the :paramref:`_orm.relationship.order_by` parameter were
stored as a mutable list, which could then be mutated when combined with
additional "order_by" methods used against the dynamic query object,
causing the ORDER BY criteria to continue to grow repetitively.
Federico Caselli [Thu, 27 May 2021 20:51:40 +0000 (22:51 +0200)]
Make other loop test compatible with py3.10
Python 3.10 fixes the issue where the loop would bind to a queue
on instantiation. Now an object binds to the loop only when first
needs it. I've looked at queues, but I'm assuming locks behave
the same,
jason3gb [Tue, 25 May 2021 19:40:48 +0000 (15:40 -0400)]
Removed automap deprecation warning
Fixed a deprecation warning that was emitted when using
:func:`_automap.automap_base` without passing an existing
``Base``.
Fixes: #6529
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
Fixed the automap's declarative import from sqlalchemy.orm instead from the deprecated ext.declarative package
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Lele Gaifax [Tue, 25 May 2021 19:51:44 +0000 (21:51 +0200)]
Fix minor glitches in the visitors.py docstrings (#6522)
* Fix docstring typo
* Capitalize first sentence word
* Properly markup class name
This is just for consistency with nearby references: the HasCacheKey class has no docstring so
it does not appear in the documentation, moreover it's in the traversals module, so Sphinx
cannot render it as a link anyway.
Jim Fulton [Tue, 18 May 2021 18:09:07 +0000 (14:09 -0400)]
Provide primary key values for data in tests that aren't about primary keys.
(message written by Mike) some backends such as BigQuery have no
autoincrement mechanism at all.
while we would like to pursue a strategy where provisioning.py could
provide for an in-Python sequence generator, at least remove
the need for autoincrement in suite tests that don't need it.
Mike Bayer [Mon, 24 May 2021 15:12:11 +0000 (11:12 -0400)]
URL parsing fixes
Fixed a long-standing issue with :class:`.URL` where query parameters
following the question mark would not be parsed correctly if the URL did
not contain a database portion with a backslash.
Fixed issue where an ``@`` sign in the database portion of a URL would not
be interpreted correctly if the URL also had a username:password section.
Mike Bayer [Thu, 20 May 2021 15:28:03 +0000 (11:28 -0400)]
Ensure extended instrumentation is fully disposed
Fixed regression in the ``sqlalchemy.ext.instrumentation`` extension that
prevented instrumentation disposal from working completely. This fix
includes both a 1.4 regression fix as well as a fix for a related issue
that existed in 1.3 also. As part of this change, the
:class:`sqlalchemy.ext.instrumentation.InstrumentationManager` class now
has a new method ``unregister()``, which replaces the previous method
``dispose()``, which was not called as of version 1.4.
Mike Bayer [Mon, 17 May 2021 15:20:10 +0000 (11:20 -0400)]
Run SelectState from obj normalize ahead of calcing ORM joins
Fixed regression where the full combination of joined inheritance, global
with_polymorphic, self-referential relationship and joined loading would
fail to be able to produce a query with the scope of lazy loads and object
refresh operations that also attempted to render the joined loader.
Jim Fulton [Mon, 17 May 2021 15:29:54 +0000 (11:29 -0400)]
Use type_ when creating literals in _LiteralRoundTripFixture
Fixes #6496 by passing the `type_` argument passed to `run` to `literal`.
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [X] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Federico Caselli [Thu, 13 May 2021 19:20:51 +0000 (21:20 +0200)]
Remove pep484 type comments from the code
Current effort is around the stub package, and having typing in
two places makes thing worse, since the types here are usually
outdated compared to the version in the stubs.
Once v2 gets under way we can start consolidating the types
here.
Mike Bayer [Fri, 14 May 2021 14:29:55 +0000 (10:29 -0400)]
Consult plugin_subject for non-ORM enabled stmts in get_bind()
Enhanced the bind resolution rules for :meth:`_orm.Session.execute` so that
when a non-ORM statement such as an :func:`_sql.insert` construct
nonetheless is built against ORM objects, to the greatest degree possible
the ORM entity will be used to resolve the bind, such as for a
:class:`_orm.Session` that has a bind map set up on a common superclass
without specific mappers or tables named in the map.
Mike Bayer [Wed, 12 May 2021 22:13:06 +0000 (18:13 -0400)]
reformat URL attributes
these were using :var: which seems to not work now,
not sure if this broke due to sphinx 1.4 or if this was
broken anyway, but these are not even refs that can be picked
up by zzzeeksphinx now.
Mike Bayer [Wed, 12 May 2021 13:26:03 +0000 (09:26 -0400)]
Create new event for collection add w/o mutation
Fixed issue when using :paramref:`_orm.relationship.cascade_backrefs`
parameter set to ``False``, which per :ref:`change_5150` is set to become
the standard behavior in SQLAlchemy 2.0, where adding the item to a
collection that uniquifies, such as ``set`` or ``dict`` would fail to fire
a cascade event if the object were already associated in that collection
via the backref. This fix represents a fundamental change in the collection
mechanics by introducing a new event state which can fire off for a
collection mutation even if there is no net change on the collection; the
action is now suited using a new event hook
:meth:`_orm.AttributeEvents.append_wo_mutation`.
Mike Bayer [Tue, 11 May 2021 12:39:24 +0000 (08:39 -0400)]
allow CTE to be direct DML target
Implemented support for a :class:`_sql.CTE` construct to be used directly
as the target of a :func:`_sql.delete` construct, i.e. "WITH ... AS cte
DELETE FROM cte". This appears to be a useful feature of SQL Server.
The CTE is now generally usable as a DML target table however
it's not clear if this syntax is valid beyond the use case of
DELETE itself.
Mike Bayer [Tue, 11 May 2021 02:52:49 +0000 (22:52 -0400)]
set bindparam.expanding in coercion again
Adjusted the logic added as part of :ticket:`6397` in 1.4.12 so that
internal mutation of the :class:`.BindParameter` object occurs within the
clause construction phase as it did before, rather than in the compilation
phase. In the latter case, the mutation still produced side effects against
the incoming construct and additionally could potentially interfere with
other internal mutation routines.
In order to solve the issue of the correct operator being present
on the BindParameter.expand_op, we necessarily have to expand the
BinaryExpression._negate() routine to flip the operator on the
BindParameter also.
Mike Bayer [Fri, 7 May 2021 17:01:40 +0000 (13:01 -0400)]
calculate warnings stacklevels dynamically
A new approach has been applied to the warnings system in SQLAlchemy to
accurately predict the appropriate stack level for each warning
dynamically. This allows evaluating the source of SQLAlchemy-generated
warnings and deprecation warnings to be more straightforward as the warning
will indicate the source line within end-user code, rather than from an
arbitrary level within SQLAlchemy's own source code.
Mike Bayer [Mon, 10 May 2021 17:19:14 +0000 (13:19 -0400)]
Correct cache key for proxy_owner, with_context_options
Fixed issue in subquery loader strategy which prevented caching from
working correctly. This would have been seen in the logs as a "generated"
message instead of "cached" for all subqueryload SQL emitted, which by
saturating the cache with new keys would degrade overall performance; it
also would produce "LRU size alert" warnings.
In this issue we also observe that the local LRU cache for lazyloader
and selectinloader will get used for all subsequent loads as well,
which makes it more likely to hit the limit of 30. However rather than
trying to work this out, it would be better if we removed the
loader-local LRU caches altogether once we are confident these
are working well.
Mike Bayer [Sun, 9 May 2021 15:54:15 +0000 (11:54 -0400)]
remove extra verbiage
having all this verbiage in the template itself means
all the issues will have all this verbiage in them, so
pull it back again. there is unfortunately no way to
cleanly direct reporters through the guidelines with github.
Mike Bayer [Sun, 9 May 2021 01:21:35 +0000 (21:21 -0400)]
Ensure propagate_to_loaders honored at the sub-loader level
Fixed additional regression caused by "eager loaders run on unexpire"
feature :ticket:`1763` where the feature would run for a
``contains_eager()`` eagerload option in the case that the
``contains_eager()`` were chained to an additional eager loader option,
which would then produce an incorrect query as the original query-bound
join criteria were no longer present.
The contains_eager() option correctly included
propagate_to_loaders=False however this would not be considered
if the contains_eager() were chained and therefore bundled inside
of an enclosing loader. We don't want to turn off propagation
completely in that case because we still want the other
loaders inside to be handled individually, so add a check
as each option is moved into the query context.
Jerry Zhao [Fri, 23 Apr 2021 15:54:04 +0000 (11:54 -0400)]
add ssl_check_hostname option in mysqldb
Added support for the ``ssl_check_hostname=`` parameter in mysql connection
URIs and updated the mysql dialect documentation regarding secure
connections. Original pull request courtesy of Jerry Zhao.
Mike Bayer [Thu, 6 May 2021 16:24:00 +0000 (12:24 -0400)]
don't cache TypeDecorator by default
The :class:`.TypeDecorator` class will now emit a warning when used in SQL
compilation with caching unless the ``.cache_ok`` flag is set to ``True``
or ``False``. ``.cache_ok`` indicates that all the parameters passed to the
object are safe to be used as a cache key, ``False`` means they are not.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
