Mike Bayer [Sun, 12 Dec 2021 16:23:07 +0000 (11:23 -0500)]
check for string_types, not str, for py2 support
Fixed regression in the :func:`_engine.make_url` function used to parse URL
strings where the query string parsing would go into a recursion overflow
if a Python 2 ``u''`` string were used.
Mike Bayer [Thu, 9 Dec 2021 17:51:43 +0000 (12:51 -0500)]
implement attributes.Proxy._clone()
Fixed issue where the internal cloning used by the
:meth:`_orm.PropComparator.any` method on a :func:`_orm.relationship` in
the case where the related class also makes use of ORM polymorphic loading,
would fail if a hybrid property on the related, polymorphic class were used
within the criteria for the ``any()`` operation.
Mike Bayer [Fri, 3 Dec 2021 19:04:05 +0000 (14:04 -0500)]
Warn when caching is disabled / document
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.
As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``. These have been amended to use inherit
cache where appropriate. "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.
The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.
Mike Bayer [Thu, 2 Dec 2021 00:27:25 +0000 (19:27 -0500)]
contextmanager skips rollback if trans says to skip it
Fixed issue where if an exception occurred when the :class:`_orm.Session`
were to close the connection within the :meth:`_orm.Session.commit` method,
when using a context manager for :meth:`_orm.Session.begin` , it would
attempt a rollback which would not be possible as the :class:`_orm.Session`
was in between where the transaction is committed and the connection is
then to be returned to the pool, raising the exception "this
sessiontransaction is in the committed state". This exception can occur
mostly in an asyncio context where CancelledError can be raised.
The where method of exists now accepts multiple cluase.
Support multiple clause elements in the :meth:`_sql.Exists.where` method,
unifying the api with the on presented by a normal :func:`_sql.select`
construct.
Mike Bayer [Thu, 2 Dec 2021 02:39:59 +0000 (21:39 -0500)]
copy list for __iadd__
Fixed issue where a list mapped with :func:`_orm.relationship` would go
into an endless loop if in-place added to itself, i.e. the ``+=`` operator
were used, as well as if ``.extend()`` were given the same list.
Kai Mueller [Wed, 1 Dec 2021 15:58:40 +0000 (10:58 -0500)]
Add __class_getitem__ to the declarative Base class
Fixed issue where the :func:`_orm.as_declarative` decorator and similar
functions used to generate the declarative base class would not copy the
``__class_getitem__()`` method from a given superclass, which prevented the
use of pep-484 generics in conjunction with the ``Base`` class. Pull
request courtesy Kai Mueller.
Federico Caselli [Sat, 27 Nov 2021 08:53:29 +0000 (09:53 +0100)]
adapt pytest plugin to support pytest v7
Implemented support for the test suite to run correctly under Pytest 7.
Previously, only Pytest 6.x was supported for Python 3, however the version
was not pinned on the upper bound in tox.ini. Pytest is not pinned in
tox.ini to be lower than version 8 so that SQLAlchemy versions released
with the current codebase will be able to be tested under tox without
changes to the environment. Much thanks to the Pytest developers for
their help with this issue.
Mike Bayer [Thu, 18 Nov 2021 18:39:54 +0000 (13:39 -0500)]
use typing.Dict
newer Pythons seem to accept ``dict[Any, Any]`` which is why
this wasn't noticed. Revise fix for #7321 made in
I55656e867876677c5c55143449db371344be8600.
Mike Bayer [Thu, 18 Nov 2021 17:46:25 +0000 (12:46 -0500)]
disable raiseerr for refresh state loader options
Fixed ORM regression where the new behavior of "eager loaders run on
unexpire" added in :ticket:`1763` would lead to loader option errors being
raised inappropriately for the case where a single :class:`_orm.Query` or
:class:`_sql.Select` were used to load multiple kinds of entities, along
with loader options that apply to just one of those kinds of entity like a
:func:`_orm.joinedload`, and later the objects would be refreshed from
expiration, where the loader options would attempt to be applied to the
mismatched object type and then raise an exception. The check for this
mismatch now bypasses raising an error for this case.
Mike Bayer [Mon, 15 Nov 2021 20:06:06 +0000 (15:06 -0500)]
favor setuptools imports over distutils
Python 3.10 has deprecated "distutils" in favor of explicit use of
"setuptools" in :pep:`632`; SQLAlchemy's setup.py has replaced imports
accordingly. However, since setuptools itself only recently added the
replacement symbols mentioned in pep-632 as of November of 2022 in version
59.0.1, ``setup.py`` still has fallback imports to distutils, as SQLAlchemy
1.4 does not have a hard setuptools versioning requirement at this time.
SQLAlchemy 2.0 is expected to use a full :pep:`517` installation layout
which will indicate appropriate setuptools versioning up front.
Mike Bayer [Mon, 15 Nov 2021 01:02:10 +0000 (20:02 -0500)]
handle dunder names in @declared_attr separately
Fixed Mypy crash which would occur when using Mypy plugin against code
which made use of :class:`_orm.declared_attr` methods for non-mapped names
like ``__mapper_args__``, ``__table_args__``, or other dunder names, as the
plugin would try to interpret these as mapped attributes which would then
be later mis-handled. As part of this change, the decorated function is
still converted by the plugin into a generic assignment statement (e.g.
``__mapper_args__: Any``) so that the argument signature can continue to be
annotated in the same way one would for any other ``@classmethod`` without
Mypy complaining about the wrong argument type for a method that isn't
explicitly ``@classmethod``.
jonathan vanasco [Mon, 27 Sep 2021 16:41:24 +0000 (12:41 -0400)]
Add new sections regarding schemas and reflection
* add a new section to reflection.rst `Schemas and Reflection`.
* this contains some text from the ticket
* migrate some text from `Specifying the Schema Name` to new section
* migrate some text from PostgreSQL dialect to new section
* target text is made more generic
* cross-reference the postgres and new sections to one another, to avoid duplication of docs
* update some docs 'meta' to 'metadata_obj'
Fixes: #4387 Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: I2b08672753fb2575d30ada07ead77587468fdade
(cherry picked from commit 0fa0beacb465c61e792c97d530a0e8fdd7139256)
Mike Bayer [Wed, 17 Nov 2021 20:08:29 +0000 (15:08 -0500)]
generalize cache_ok to UserDefinedType
Extended the ``cache_ok`` flag and corresponding warning message if this
flag is not defined, a behavior first established for
:class:`.TypeDecorator` as part of :ticket:`6436`, to also take place for
:class:`.UserDefinedType`, by generalizing the flag and associated caching
logic to a new common base for these two types, :class:`.ExternalType`.
The change means any current :class:`.UserDefinedType` will now cause SQL
statement caching to no longer take place for statements which make use of
the datatype, along with a warning being emitted, unless the class defines
the :attr:`.UserDefinedType.cache_ok` flag as True. If the datatype cannot
form a deterministic, hashable cache key derived from its arguments, it may return
False which will continue to keep caching disabled but will suppress the
warning. In particular, custom datatypes currently used in packages such as
SQLAlchemy-utils will need to implement this flag. The issue was observed
as a result of a SQLAlchemy-utils datatype that is not currently cacheable.
Eric Masseran [Tue, 2 Nov 2021 20:40:04 +0000 (16:40 -0400)]
Add Non linear CTE support
"Compound select" methods like :meth:`_sql.Select.union`,
:meth:`_sql.Select.intersect_all` etc. now accept ``*other`` as an argument
rather than ``other`` to allow for multiple additional SELECTs to be
compounded with the parent statement at once. In particular, the change as
applied to :meth:`_sql.CTE.union` and :meth:`_sql.CTE.union_all` now allow
for a so-called "non-linear CTE" to be created with the :class:`_sql.CTE`
construct, whereas previously there was no way to have more than two CTE
sub-elements in a UNION together while still correctly calling upon the CTE
in recursive fashion. Pull request courtesy Eric Masseran.
Allow:
```sql
WITH RECURSIVE nodes(x) AS (
SELECT 59
UNION
SELECT aa FROM edge JOIN nodes ON bb=x
UNION
SELECT bb FROM edge JOIN nodes ON aa=x
)
SELECT x FROM nodes;
```
Based on @zzzeek suggestion: https://github.com/sqlalchemy/sqlalchemy/pull/7133#issuecomment-933882348
Mike Bayer [Wed, 10 Nov 2021 14:52:18 +0000 (09:52 -0500)]
qualify asyncpg API tests for python 3.8
Getting
TypeError: object MagicMock can't be used in 'await' expression
for Python 3.7 and earlier. this test is not needed
on all platforms it's confirming that two methods
are present.
Mike Bayer [Mon, 8 Nov 2021 23:29:16 +0000 (18:29 -0500)]
change the POSTCOMPILE/ SCHEMA symbols to not conflict w mssql quoting
Adjusted the compiler's generation of "post compile" symbols including
those used for "expanding IN" as well as for the "schema translate map" to
not be based directly on plain bracketed strings with underscores, as this
conflicts directly with SQL Server's quoting format of also using brackets,
which produces false matches when the compiler replaces "post compile" and
"schema translate" symbols. The issue created easy to reproduce examples
both with the :meth:`.Inspector.get_schema_names` method when used in
conjunction with the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature, as well in the unlikely case that a symbol overlapping with the
internal name "POSTCOMPILE" would be used with a feature like "expanding
in".
Mike Bayer [Tue, 9 Nov 2021 20:02:44 +0000 (15:02 -0500)]
set within_columns_clause=False for all sub-elements of select()
Fixed issue where using the feature of using a string label for ordering or
grouping described at :ref:`tutorial_order_by_label` would fail to function
correctly if used on a :class:`.CTE` construct, when the CTE were embedded
inside of an enclosing :class:`_sql.Select` statement that itself was set
up as a scalar subquery.
Mike Bayer [Tue, 9 Nov 2021 16:31:23 +0000 (11:31 -0500)]
upgrade deferred loader to regular loader if refresh_state
Fixed issue where deferred polymorphic loading of attributes from a
joined-table inheritance subclass would fail to populate the attribute
correctly if the :func:`_orm.load_only` option were used to originally
exclude that attribute, in the case where the load_only were descending
from a relationship loader option. The fix allows that other valid options
such as ``defer(..., raiseload=True)`` etc. still function as expected.
jonathan vanasco [Mon, 27 Sep 2021 16:51:32 +0000 (12:51 -0400)]
Fixes: #4390
Deprecated an undocumented loader option syntax ``".*"``, which appears to
be no different than passing a single asterisk, and will emit a deprecation
warning if used. This syntax may have been intended for something but there
is currently no need for it.
The original ticket was to document the `.{WILDCARD}` (e.g. `.*`) format,
however this format does not appear to be used or needed by SQLAlchemy
and is likely not used by any projects or developers.
This PR invokes `util.warn_deprecated` to notify users this functionality
is deprecated, and directs them to the #4390 issue if they actually use it.
Assuming there are no complaints over this warning in the coming months,
this code can be removed in a future major release.
Mike Bayer [Fri, 5 Nov 2021 14:18:42 +0000 (10:18 -0400)]
use tuple expansion if type._is_tuple, test for Sequence if no type
Fixed regression where the row objects returned for ORM queries, which are
now the normal :class:`_sql.Row` objects, would not be interpreted by the
:meth:`_sql.ColumnOperators.in_` operator as tuple values to be broken out
into individual bound parameters, and would instead pass them as single
values to the driver leading to failures. The change to the "expanding IN"
system now accommodates for the expression already being of type
:class:`.TupleType` and treats values accordingly if so. In the uncommon
case of using "tuple-in" with an untyped statement such as a textual
statement with no typing information, a tuple value is detected for values
that implement ``collections.abc.Sequence``, but that are not ``str`` or
``bytes``, as always when testing for ``Sequence``.
Added :class:`.TupleType` to the top level ``sqlalchemy`` import namespace.
Mike Bayer [Thu, 4 Nov 2021 21:02:24 +0000 (17:02 -0400)]
Check for Mapping explicitly in 2.0 params
Fixed issue in future :class:`_future.Connection` object where the
:meth:`_future.Connection.execute` method would not accept a non-dict
mapping object, such as SQLAlchemy's own :class:`.RowMapping` or other
``abc.collections.Mapping`` object as a parameter dictionary.
Mike Bayer [Thu, 4 Nov 2021 17:36:43 +0000 (13:36 -0400)]
Update "transaction has already begun" language
As future connections will now be autobeginning, there
will be more cases where begin() can't be called as well as where isolation level
can't be set, which will be surprising as this is a behavioral
change for 2.0; additionally, when DBAPI autocommit is set,
there isn't actually a DBAPI level transaction in effect even though
Connection has a Transaction object. Clarify the language in these
two error messages to make it clear that begin() and autobegin
are tracking a SQLAlchemy-level Transaction() object, whether or not
the DBAPI has actually started a transaction, and that this is the
reason rollback() or commit() is required before performing
the requsted operation. Additionally make sure the error message
mentions "autobegin" as a likely reason this error is being
encountered along with what Connection needs the user to do in
order to resolve.
Mike Bayer [Wed, 3 Nov 2021 15:32:51 +0000 (11:32 -0400)]
simplify and publicize the asyncpg JSON(B) codec registrsation
Added overridable methods ``PGDialect_asyncpg.setup_asyncpg_json_codec``
and ``PGDialect_asyncpg.setup_asyncpg_jsonb_codec`` codec, which handle the
required task of registering JSON/JSONB codecs for these datatypes when
using asyncpg. The change is that methods are broken out as individual,
overridable methods to support third party dialects that need to alter or
disable how these particular codecs are set up.
Mike Bayer [Thu, 4 Nov 2021 01:26:44 +0000 (21:26 -0400)]
use ExpressionElementRole for case targets in case()
Fixed regression where the :func:`_sql.text` construct would no longer be
accepted as a target case in the "whens" list within a :func:`_sql.case`
construct. The regression appears related to an attempt to guard against
some forms of literal values that were considered to be ambiguous when
passed here; however, there's no reason the target cases shouldn't be
interpreted as open-ended SQL expressions just like anywhere else, and a
literal string or tuple will be converted to a bound parameter as would be
the case elsewhere.
Gord Thompson [Tue, 2 Nov 2021 22:16:50 +0000 (16:16 -0600)]
Gracefully degrade unsupported types with asyncpg
Fixes: #7284
Modify the on_connect() method of PGDialect_asyncpg to
gracefully degrade unsupported types instead of throwing a
ValueError. Useful for third-party dialects that derive
from PGDialect_asyncpg but whose databases do not support
all types (e.g., CockroachDB supports JSONB but not JSON).
Fixed issue in visit_on_duplicate_key_update within a composed expression
Fixed issue in MySQL :meth:`_mysql.Insert.on_duplicate_key_update` which
would render the wrong column name when an expression were used in a VALUES
expression. Pull request courtesy Cristian Sabaila.
Mike Bayer [Tue, 2 Nov 2021 14:58:01 +0000 (10:58 -0400)]
ensure soft_close occurs for fetchmany with server side cursor
Fixed regression where the :meth:`_engine.CursorResult.fetchmany` method
would fail to autoclose a server-side cursor (i.e. when ``stream_results``
or ``yield_per`` is in use, either Core or ORM oriented results) when the
results were fully exhausted.
All :class:`_result.Result` objects will now consistently raise
:class:`_exc.ResourceClosedError` if they are used after a hard close,
which includes the "hard close" that occurs after calling "single row or
value" methods like :meth:`_result.Result.first` and
:meth:`_result.Result.scalar`. This was already the behavior of the most
common class of result objects returned for Core statement executions, i.e.
those based on :class:`_engine.CursorResult`, so this behavior is not new.
However, the change has been extended to properly accommodate for the ORM
"filtering" result objects returned when using 2.0 style ORM queries,
which would previously behave in "soft closed" style of returning empty
results, or wouldn't actually "soft close" at all and would continue
yielding from the underlying cursor.
As part of this change, also added :meth:`_result.Result.close` to the base
:class:`_result.Result` class and implemented it for the filtered result
implementations that are used by the ORM, so that it is possible to call
the :meth:`_engine.CursorResult.close` method on the underlying
:class:`_engine.CursorResult` when the the ``yield_per`` execution option
is in use to close a server side cursor before remaining ORM results have
been fetched. This was again already available for Core result sets but the
change makes it available for 2.0 style ORM results as well.
Mike Bayer [Mon, 1 Nov 2021 20:36:51 +0000 (16:36 -0400)]
use full context manager flow for future.Engine.begin()
Fixed issue in future :class:`_future.Engine` where calling upon
:meth:`_future.Engine.begin` and entering the context manager would not
close the connection if the actual BEGIN operation failed for some reason,
such as an event handler raising an exception; this use case failed to be
tested for the future version of the engine. Note that the "future" context
managers which handle ``begin()`` blocks in Core and ORM don't actually run
the "BEGIN" operation until the context managers are actually entered. This
is different from the legacy version which runs the "BEGIN" operation up
front.
Mike Bayer [Mon, 1 Nov 2021 16:06:32 +0000 (12:06 -0400)]
Revise "literal parameters" FAQ section
based on feedback in #7271, the emphasis on TypeDecorator
as a solution to this problem is not very practical. illustrate
a series of quick recipes that are useful for debugging purposes
to print out a repr() or simple stringify of a parameter
without the need to construct custom dialects or types.
Mike Bayer [Fri, 29 Oct 2021 21:36:26 +0000 (17:36 -0400)]
warnings removal, merge_result
this is the last warning to remove.
Also fixes some mistakes I made with the new
Base20DeprecationWarning and LegacyAPIWarning classes created,
where functions in deprecations.py were still hardcoded to
RemovedIn20Warning.
most of the work for aliased / from_joinpoint has been done
already as I added all new tests for these and moved
most aliased/from_joinpoint to test/orm/test_deprecations.py
already
Mike Bayer [Wed, 27 Oct 2021 15:14:50 +0000 (11:14 -0400)]
consider "inspect(_of_type)" to be the entity of a comparator
Fixed 1.4 regression where :meth:`_orm.Query.filter_by` would not function
correctly when :meth:`_orm.Query.join` were joined to an entity which made
use of :meth:`_orm.PropComparator.of_type` to specify an aliased version of
the target entity. The issue also applies to future style ORM queries
constructed with :func:`_sql.select`.
Mike Bayer [Tue, 26 Oct 2021 19:52:25 +0000 (15:52 -0400)]
add additional "oracle mode" reserved words
despite mariadb's docs, the word "system" must be
quoted in plain mariadb 10.5, not sure if that's
"oracle mode" but it is > 10.3. observed keystone
tests failing on a column of this name.
Mike Bayer [Tue, 26 Oct 2021 15:28:42 +0000 (11:28 -0400)]
dont pull filter_by from from_obj, entity is hopefully sufficient
Fixed 1.4 regression where :meth:`_orm.Query.filter_by` would not function
correctly on a :class:`_orm.Query` that was produced from
:meth:`_orm.Query.union`, :meth:`_orm.Query.from_self` or similar.
jonathan vanasco [Fri, 24 Sep 2021 18:55:59 +0000 (14:55 -0400)]
Fixes: #4100
Add a warning, in two places, stating `with_for_update` will lock joinedload
tables, because at least one person did not expect the obvious to happen.
Also warn that eager loading techniques may not work with `with_for_update`
and combining the two is not officially supported or recommended.
Mike Bayer [Wed, 20 Oct 2021 16:50:53 +0000 (12:50 -0400)]
deprecation warnings: strings in loader options, join, with_parent
Repairs one in-library deprecation warning regarding
mapper propagation of options
raises maxfail to 250, as 25 is too low when we are trying
to address many errors at once. the 25 was originally
due to the fact that our fixtures would be broken after
that many failures in most cases, which today should not
be the case nearly as often.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
