Kai Mueller [Tue, 21 Dec 2021 20:00:30 +0000 (15:00 -0500)]
Fix missing class attributes when using __class_getitem__
Fixed issue where the ``__class_getitem__()`` method of the generated
declarative base class by :func:`_orm.as_declarative` would lead to
inaccessible class attributes such as ``__table__``, for cases where a
``Generic[T]`` style typing declaration were used in the class hierarchy.
This is in continuation from the basic addition of ``__class_getitem__()``
in :ticket:`7368`. Pull request courtesy Kai Mueller.
Mike Bayer [Wed, 22 Dec 2021 20:33:11 +0000 (15:33 -0500)]
add recursion check for with_loader_criteria() option
Fixed recursion overflow which could occur within ORM statement compilation
when using either the :func:`_orm.with_loader_criteria` feature or the the
:meth:`_orm.PropComparator.and_` method within a loader strategy in
conjunction with a subquery which referred to the same entity being altered
by the criteria option, or loaded by the loader strategy. A check for
coming across the same loader criteria option in a recursive fashion has
been added to accommodate for this scenario.
Mike Bayer [Wed, 22 Dec 2021 21:21:33 +0000 (16:21 -0500)]
use fully qualified, locatable names for all use of api.named_type()
Fixed mypy regression where the release of mypy 0.930 added additional
internal checks to the format of "named types", requiring that they be
fully qualified and locatable. This broke the mypy plugin for SQLAlchemy,
raising an assertion error, as there was use of symbols such as
``__builtins__`` and other un-locatable or unqualified names that
previously had not raised any assertions.
Mike Bayer [Tue, 21 Dec 2021 23:08:33 +0000 (18:08 -0500)]
accommodate for "clone" of ColumnClause
for use with the ClauseElement.params() method,
altered ColumnClause._clone() so that while the element
stays immutable, if the column is associated with a subquery,
it returns a new version of itself as corresponding to a
clone of the subquery. this allows processing functions
to access the parameters in the subquery and produce a
copy of it. The use case here is the expanded use of
.params() within loader strategies that use
HasCacheKey._apply_params_to_element().
Fixed issue in new "loader criteria" method
:meth:`_orm.PropComparator.and_` where usage with a loader strategy like
:func:`_orm.selectinload` against a column that was a member of the ``.c.``
collection of a subquery object, where the subquery would be dynamically
added to the FROM clause of the statement, would be subject to stale
parameter values within the subquery in the SQL statement cache, as the
process used by the loader strategy to replace the parameters at execution
time would fail to accommodate the subquery when received in this form.
Mike Bayer [Tue, 14 Dec 2021 21:46:50 +0000 (16:46 -0500)]
include InterfaceError for mariadb disconnect check
Corrected the error classes inspected for the "is_disconnect" check for the
``mariadbconnector`` dialect, which was failing for disconnects that
occurred due to common MySQL/MariaDB error codes such as 2006; the DBAPI
appears to currently use the ``mariadb.InterfaceError`` exception class for
disconnect errors such as error code 2006, which has been added to the list
of classes checked.
For the current "real reconnect test", shutting down the mariadb
connection from the client side produces
ProgrammingError("Connection isn't valid anymore") which we also
continue to intercept.
Nils Philippsen [Sun, 12 Dec 2021 23:35:03 +0000 (18:35 -0500)]
Ignore ephemeral classes in test_all_present()
Fixed a regression in the test suite where the test called
``CompareAndCopyTest::test_all_present`` would fail on some platforms due
to additional testing artifacts being detected. Pull request courtesy Nils
Philippsen.
In some circumstances, ephemeral class objects that are created within
the scope of a test method don't seem to be garbage collected directly
on exit. Filter out classes created in test modules.
Mike Bayer [Sun, 12 Dec 2021 18:37:21 +0000 (13:37 -0500)]
use the options from the cached statement for propagate_options
Fixed caching-related issue where the use of a loader option of the form
``lazyload(aliased(A).bs).joinedload(B.cs)`` would fail to result in the
joinedload being invoked for runs subsequent to the query being cached, due
to a mismatch for the options / object path applied to the objects loaded
for a query with a lead entity that used ``aliased()``.
Mike Bayer [Sun, 12 Dec 2021 16:23:07 +0000 (11:23 -0500)]
check for string_types, not str, for py2 support
Fixed regression in the :func:`_engine.make_url` function used to parse URL
strings where the query string parsing would go into a recursion overflow
if a Python 2 ``u''`` string were used.
Nils Philippsen [Sat, 13 Nov 2021 16:11:32 +0000 (11:11 -0500)]
Add async_engine_from_config()
Added :func:`_asyncio.async_engine_config` function to create
an async engine from a configuration dict. This otherwise
behaves the same as :func:`_sa.engine_from_config`.
Mike Bayer [Thu, 9 Dec 2021 19:23:42 +0000 (14:23 -0500)]
implement correct errors for Row immutability
Corrected the error message for the ``AttributeError`` that's raised when
attempting to write to an attribute on the :class:`_result.Row` class,
which is immutable. The previous message claimed the column didn't exist
which is misleading.
Mike Bayer [Thu, 9 Dec 2021 17:51:43 +0000 (12:51 -0500)]
implement attributes.Proxy._clone()
Fixed issue where the internal cloning used by the
:meth:`_orm.PropComparator.any` method on a :func:`_orm.relationship` in
the case where the related class also makes use of ORM polymorphic loading,
would fail if a hybrid property on the related, polymorphic class were used
within the criteria for the ``any()`` operation.
Mike Bayer [Fri, 3 Dec 2021 19:04:05 +0000 (14:04 -0500)]
Warn when caching is disabled / document
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.
As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``. These have been amended to use inherit
cache where appropriate. "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.
The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.
Mike Bayer [Thu, 2 Dec 2021 00:27:25 +0000 (19:27 -0500)]
contextmanager skips rollback if trans says to skip it
Fixed issue where if an exception occurred when the :class:`_orm.Session`
were to close the connection within the :meth:`_orm.Session.commit` method,
when using a context manager for :meth:`_orm.Session.begin` , it would
attempt a rollback which would not be possible as the :class:`_orm.Session`
was in between where the transaction is committed and the connection is
then to be returned to the pool, raising the exception "this
sessiontransaction is in the committed state". This exception can occur
mostly in an asyncio context where CancelledError can be raised.
The where method of exists now accepts multiple cluase.
Support multiple clause elements in the :meth:`_sql.Exists.where` method,
unifying the api with the on presented by a normal :func:`_sql.select`
construct.
Mike Bayer [Thu, 2 Dec 2021 02:39:59 +0000 (21:39 -0500)]
copy list for __iadd__
Fixed issue where a list mapped with :func:`_orm.relationship` would go
into an endless loop if in-place added to itself, i.e. the ``+=`` operator
were used, as well as if ``.extend()`` were given the same list.
Kai Mueller [Wed, 1 Dec 2021 15:58:40 +0000 (10:58 -0500)]
Add __class_getitem__ to the declarative Base class
Fixed issue where the :func:`_orm.as_declarative` decorator and similar
functions used to generate the declarative base class would not copy the
``__class_getitem__()`` method from a given superclass, which prevented the
use of pep-484 generics in conjunction with the ``Base`` class. Pull
request courtesy Kai Mueller.
Federico Caselli [Sat, 27 Nov 2021 08:53:29 +0000 (09:53 +0100)]
adapt pytest plugin to support pytest v7
Implemented support for the test suite to run correctly under Pytest 7.
Previously, only Pytest 6.x was supported for Python 3, however the version
was not pinned on the upper bound in tox.ini. Pytest is not pinned in
tox.ini to be lower than version 8 so that SQLAlchemy versions released
with the current codebase will be able to be tested under tox without
changes to the environment. Much thanks to the Pytest developers for
their help with this issue.
Mike Bayer [Thu, 18 Nov 2021 18:39:54 +0000 (13:39 -0500)]
use typing.Dict
newer Pythons seem to accept ``dict[Any, Any]`` which is why
this wasn't noticed. Revise fix for #7321 made in
I55656e867876677c5c55143449db371344be8600.
Mike Bayer [Thu, 18 Nov 2021 17:46:25 +0000 (12:46 -0500)]
disable raiseerr for refresh state loader options
Fixed ORM regression where the new behavior of "eager loaders run on
unexpire" added in :ticket:`1763` would lead to loader option errors being
raised inappropriately for the case where a single :class:`_orm.Query` or
:class:`_sql.Select` were used to load multiple kinds of entities, along
with loader options that apply to just one of those kinds of entity like a
:func:`_orm.joinedload`, and later the objects would be refreshed from
expiration, where the loader options would attempt to be applied to the
mismatched object type and then raise an exception. The check for this
mismatch now bypasses raising an error for this case.
Mike Bayer [Mon, 15 Nov 2021 20:06:06 +0000 (15:06 -0500)]
favor setuptools imports over distutils
Python 3.10 has deprecated "distutils" in favor of explicit use of
"setuptools" in :pep:`632`; SQLAlchemy's setup.py has replaced imports
accordingly. However, since setuptools itself only recently added the
replacement symbols mentioned in pep-632 as of November of 2022 in version
59.0.1, ``setup.py`` still has fallback imports to distutils, as SQLAlchemy
1.4 does not have a hard setuptools versioning requirement at this time.
SQLAlchemy 2.0 is expected to use a full :pep:`517` installation layout
which will indicate appropriate setuptools versioning up front.
Mike Bayer [Mon, 15 Nov 2021 01:02:10 +0000 (20:02 -0500)]
handle dunder names in @declared_attr separately
Fixed Mypy crash which would occur when using Mypy plugin against code
which made use of :class:`_orm.declared_attr` methods for non-mapped names
like ``__mapper_args__``, ``__table_args__``, or other dunder names, as the
plugin would try to interpret these as mapped attributes which would then
be later mis-handled. As part of this change, the decorated function is
still converted by the plugin into a generic assignment statement (e.g.
``__mapper_args__: Any``) so that the argument signature can continue to be
annotated in the same way one would for any other ``@classmethod`` without
Mypy complaining about the wrong argument type for a method that isn't
explicitly ``@classmethod``.
jonathan vanasco [Mon, 27 Sep 2021 16:41:24 +0000 (12:41 -0400)]
Add new sections regarding schemas and reflection
* add a new section to reflection.rst `Schemas and Reflection`.
* this contains some text from the ticket
* migrate some text from `Specifying the Schema Name` to new section
* migrate some text from PostgreSQL dialect to new section
* target text is made more generic
* cross-reference the postgres and new sections to one another, to avoid duplication of docs
* update some docs 'meta' to 'metadata_obj'
Fixes: #4387 Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: I2b08672753fb2575d30ada07ead77587468fdade
(cherry picked from commit 0fa0beacb465c61e792c97d530a0e8fdd7139256)
Mike Bayer [Wed, 17 Nov 2021 20:08:29 +0000 (15:08 -0500)]
generalize cache_ok to UserDefinedType
Extended the ``cache_ok`` flag and corresponding warning message if this
flag is not defined, a behavior first established for
:class:`.TypeDecorator` as part of :ticket:`6436`, to also take place for
:class:`.UserDefinedType`, by generalizing the flag and associated caching
logic to a new common base for these two types, :class:`.ExternalType`.
The change means any current :class:`.UserDefinedType` will now cause SQL
statement caching to no longer take place for statements which make use of
the datatype, along with a warning being emitted, unless the class defines
the :attr:`.UserDefinedType.cache_ok` flag as True. If the datatype cannot
form a deterministic, hashable cache key derived from its arguments, it may return
False which will continue to keep caching disabled but will suppress the
warning. In particular, custom datatypes currently used in packages such as
SQLAlchemy-utils will need to implement this flag. The issue was observed
as a result of a SQLAlchemy-utils datatype that is not currently cacheable.
Eric Masseran [Tue, 2 Nov 2021 20:40:04 +0000 (16:40 -0400)]
Add Non linear CTE support
"Compound select" methods like :meth:`_sql.Select.union`,
:meth:`_sql.Select.intersect_all` etc. now accept ``*other`` as an argument
rather than ``other`` to allow for multiple additional SELECTs to be
compounded with the parent statement at once. In particular, the change as
applied to :meth:`_sql.CTE.union` and :meth:`_sql.CTE.union_all` now allow
for a so-called "non-linear CTE" to be created with the :class:`_sql.CTE`
construct, whereas previously there was no way to have more than two CTE
sub-elements in a UNION together while still correctly calling upon the CTE
in recursive fashion. Pull request courtesy Eric Masseran.
Allow:
```sql
WITH RECURSIVE nodes(x) AS (
SELECT 59
UNION
SELECT aa FROM edge JOIN nodes ON bb=x
UNION
SELECT bb FROM edge JOIN nodes ON aa=x
)
SELECT x FROM nodes;
```
Based on @zzzeek suggestion: https://github.com/sqlalchemy/sqlalchemy/pull/7133#issuecomment-933882348
Mike Bayer [Wed, 10 Nov 2021 14:52:18 +0000 (09:52 -0500)]
qualify asyncpg API tests for python 3.8
Getting
TypeError: object MagicMock can't be used in 'await' expression
for Python 3.7 and earlier. this test is not needed
on all platforms it's confirming that two methods
are present.
Mike Bayer [Mon, 8 Nov 2021 23:29:16 +0000 (18:29 -0500)]
change the POSTCOMPILE/ SCHEMA symbols to not conflict w mssql quoting
Adjusted the compiler's generation of "post compile" symbols including
those used for "expanding IN" as well as for the "schema translate map" to
not be based directly on plain bracketed strings with underscores, as this
conflicts directly with SQL Server's quoting format of also using brackets,
which produces false matches when the compiler replaces "post compile" and
"schema translate" symbols. The issue created easy to reproduce examples
both with the :meth:`.Inspector.get_schema_names` method when used in
conjunction with the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature, as well in the unlikely case that a symbol overlapping with the
internal name "POSTCOMPILE" would be used with a feature like "expanding
in".
Mike Bayer [Tue, 9 Nov 2021 20:02:44 +0000 (15:02 -0500)]
set within_columns_clause=False for all sub-elements of select()
Fixed issue where using the feature of using a string label for ordering or
grouping described at :ref:`tutorial_order_by_label` would fail to function
correctly if used on a :class:`.CTE` construct, when the CTE were embedded
inside of an enclosing :class:`_sql.Select` statement that itself was set
up as a scalar subquery.
Mike Bayer [Tue, 9 Nov 2021 16:31:23 +0000 (11:31 -0500)]
upgrade deferred loader to regular loader if refresh_state
Fixed issue where deferred polymorphic loading of attributes from a
joined-table inheritance subclass would fail to populate the attribute
correctly if the :func:`_orm.load_only` option were used to originally
exclude that attribute, in the case where the load_only were descending
from a relationship loader option. The fix allows that other valid options
such as ``defer(..., raiseload=True)`` etc. still function as expected.
jonathan vanasco [Mon, 27 Sep 2021 16:51:32 +0000 (12:51 -0400)]
Fixes: #4390
Deprecated an undocumented loader option syntax ``".*"``, which appears to
be no different than passing a single asterisk, and will emit a deprecation
warning if used. This syntax may have been intended for something but there
is currently no need for it.
The original ticket was to document the `.{WILDCARD}` (e.g. `.*`) format,
however this format does not appear to be used or needed by SQLAlchemy
and is likely not used by any projects or developers.
This PR invokes `util.warn_deprecated` to notify users this functionality
is deprecated, and directs them to the #4390 issue if they actually use it.
Assuming there are no complaints over this warning in the coming months,
this code can be removed in a future major release.
Mike Bayer [Fri, 5 Nov 2021 14:18:42 +0000 (10:18 -0400)]
use tuple expansion if type._is_tuple, test for Sequence if no type
Fixed regression where the row objects returned for ORM queries, which are
now the normal :class:`_sql.Row` objects, would not be interpreted by the
:meth:`_sql.ColumnOperators.in_` operator as tuple values to be broken out
into individual bound parameters, and would instead pass them as single
values to the driver leading to failures. The change to the "expanding IN"
system now accommodates for the expression already being of type
:class:`.TupleType` and treats values accordingly if so. In the uncommon
case of using "tuple-in" with an untyped statement such as a textual
statement with no typing information, a tuple value is detected for values
that implement ``collections.abc.Sequence``, but that are not ``str`` or
``bytes``, as always when testing for ``Sequence``.
Added :class:`.TupleType` to the top level ``sqlalchemy`` import namespace.
Mike Bayer [Thu, 4 Nov 2021 21:02:24 +0000 (17:02 -0400)]
Check for Mapping explicitly in 2.0 params
Fixed issue in future :class:`_future.Connection` object where the
:meth:`_future.Connection.execute` method would not accept a non-dict
mapping object, such as SQLAlchemy's own :class:`.RowMapping` or other
``abc.collections.Mapping`` object as a parameter dictionary.
Mike Bayer [Thu, 4 Nov 2021 17:36:43 +0000 (13:36 -0400)]
Update "transaction has already begun" language
As future connections will now be autobeginning, there
will be more cases where begin() can't be called as well as where isolation level
can't be set, which will be surprising as this is a behavioral
change for 2.0; additionally, when DBAPI autocommit is set,
there isn't actually a DBAPI level transaction in effect even though
Connection has a Transaction object. Clarify the language in these
two error messages to make it clear that begin() and autobegin
are tracking a SQLAlchemy-level Transaction() object, whether or not
the DBAPI has actually started a transaction, and that this is the
reason rollback() or commit() is required before performing
the requsted operation. Additionally make sure the error message
mentions "autobegin" as a likely reason this error is being
encountered along with what Connection needs the user to do in
order to resolve.
Mike Bayer [Wed, 3 Nov 2021 15:32:51 +0000 (11:32 -0400)]
simplify and publicize the asyncpg JSON(B) codec registrsation
Added overridable methods ``PGDialect_asyncpg.setup_asyncpg_json_codec``
and ``PGDialect_asyncpg.setup_asyncpg_jsonb_codec`` codec, which handle the
required task of registering JSON/JSONB codecs for these datatypes when
using asyncpg. The change is that methods are broken out as individual,
overridable methods to support third party dialects that need to alter or
disable how these particular codecs are set up.
Mike Bayer [Thu, 4 Nov 2021 01:26:44 +0000 (21:26 -0400)]
use ExpressionElementRole for case targets in case()
Fixed regression where the :func:`_sql.text` construct would no longer be
accepted as a target case in the "whens" list within a :func:`_sql.case`
construct. The regression appears related to an attempt to guard against
some forms of literal values that were considered to be ambiguous when
passed here; however, there's no reason the target cases shouldn't be
interpreted as open-ended SQL expressions just like anywhere else, and a
literal string or tuple will be converted to a bound parameter as would be
the case elsewhere.
Gord Thompson [Tue, 2 Nov 2021 22:16:50 +0000 (16:16 -0600)]
Gracefully degrade unsupported types with asyncpg
Fixes: #7284
Modify the on_connect() method of PGDialect_asyncpg to
gracefully degrade unsupported types instead of throwing a
ValueError. Useful for third-party dialects that derive
from PGDialect_asyncpg but whose databases do not support
all types (e.g., CockroachDB supports JSONB but not JSON).
Fixed issue in visit_on_duplicate_key_update within a composed expression
Fixed issue in MySQL :meth:`_mysql.Insert.on_duplicate_key_update` which
would render the wrong column name when an expression were used in a VALUES
expression. Pull request courtesy Cristian Sabaila.
Mike Bayer [Tue, 2 Nov 2021 14:58:01 +0000 (10:58 -0400)]
ensure soft_close occurs for fetchmany with server side cursor
Fixed regression where the :meth:`_engine.CursorResult.fetchmany` method
would fail to autoclose a server-side cursor (i.e. when ``stream_results``
or ``yield_per`` is in use, either Core or ORM oriented results) when the
results were fully exhausted.
All :class:`_result.Result` objects will now consistently raise
:class:`_exc.ResourceClosedError` if they are used after a hard close,
which includes the "hard close" that occurs after calling "single row or
value" methods like :meth:`_result.Result.first` and
:meth:`_result.Result.scalar`. This was already the behavior of the most
common class of result objects returned for Core statement executions, i.e.
those based on :class:`_engine.CursorResult`, so this behavior is not new.
However, the change has been extended to properly accommodate for the ORM
"filtering" result objects returned when using 2.0 style ORM queries,
which would previously behave in "soft closed" style of returning empty
results, or wouldn't actually "soft close" at all and would continue
yielding from the underlying cursor.
As part of this change, also added :meth:`_result.Result.close` to the base
:class:`_result.Result` class and implemented it for the filtered result
implementations that are used by the ORM, so that it is possible to call
the :meth:`_engine.CursorResult.close` method on the underlying
:class:`_engine.CursorResult` when the the ``yield_per`` execution option
is in use to close a server side cursor before remaining ORM results have
been fetched. This was again already available for Core result sets but the
change makes it available for 2.0 style ORM results as well.
Mike Bayer [Mon, 1 Nov 2021 20:36:51 +0000 (16:36 -0400)]
use full context manager flow for future.Engine.begin()
Fixed issue in future :class:`_future.Engine` where calling upon
:meth:`_future.Engine.begin` and entering the context manager would not
close the connection if the actual BEGIN operation failed for some reason,
such as an event handler raising an exception; this use case failed to be
tested for the future version of the engine. Note that the "future" context
managers which handle ``begin()`` blocks in Core and ORM don't actually run
the "BEGIN" operation until the context managers are actually entered. This
is different from the legacy version which runs the "BEGIN" operation up
front.
Mike Bayer [Mon, 1 Nov 2021 16:06:32 +0000 (12:06 -0400)]
Revise "literal parameters" FAQ section
based on feedback in #7271, the emphasis on TypeDecorator
as a solution to this problem is not very practical. illustrate
a series of quick recipes that are useful for debugging purposes
to print out a repr() or simple stringify of a parameter
without the need to construct custom dialects or types.
Mike Bayer [Fri, 29 Oct 2021 21:36:26 +0000 (17:36 -0400)]
warnings removal, merge_result
this is the last warning to remove.
Also fixes some mistakes I made with the new
Base20DeprecationWarning and LegacyAPIWarning classes created,
where functions in deprecations.py were still hardcoded to
RemovedIn20Warning.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
