Mike Bayer [Thu, 18 Nov 2021 18:39:54 +0000 (13:39 -0500)]
use typing.Dict
newer Pythons seem to accept ``dict[Any, Any]`` which is why
this wasn't noticed. Revise fix for #7321 made in
I55656e867876677c5c55143449db371344be8600.
Mike Bayer [Thu, 18 Nov 2021 17:46:25 +0000 (12:46 -0500)]
disable raiseerr for refresh state loader options
Fixed ORM regression where the new behavior of "eager loaders run on
unexpire" added in :ticket:`1763` would lead to loader option errors being
raised inappropriately for the case where a single :class:`_orm.Query` or
:class:`_sql.Select` were used to load multiple kinds of entities, along
with loader options that apply to just one of those kinds of entity like a
:func:`_orm.joinedload`, and later the objects would be refreshed from
expiration, where the loader options would attempt to be applied to the
mismatched object type and then raise an exception. The check for this
mismatch now bypasses raising an error for this case.
Mike Bayer [Mon, 15 Nov 2021 01:02:10 +0000 (20:02 -0500)]
handle dunder names in @declared_attr separately
Fixed Mypy crash which would occur when using Mypy plugin against code
which made use of :class:`_orm.declared_attr` methods for non-mapped names
like ``__mapper_args__``, ``__table_args__``, or other dunder names, as the
plugin would try to interpret these as mapped attributes which would then
be later mis-handled. As part of this change, the decorated function is
still converted by the plugin into a generic assignment statement (e.g.
``__mapper_args__: Any``) so that the argument signature can continue to be
annotated in the same way one would for any other ``@classmethod`` without
Mypy complaining about the wrong argument type for a method that isn't
explicitly ``@classmethod``.
jonathan vanasco [Mon, 27 Sep 2021 16:41:24 +0000 (12:41 -0400)]
Add new sections regarding schemas and reflection
* add a new section to reflection.rst `Schemas and Reflection`.
* this contains some text from the ticket
* migrate some text from `Specifying the Schema Name` to new section
* migrate some text from PostgreSQL dialect to new section
* target text is made more generic
* cross-reference the postgres and new sections to one another, to avoid duplication of docs
* update some docs 'meta' to 'metadata_obj'
Fixes: #4387 Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: I2b08672753fb2575d30ada07ead77587468fdade
(cherry picked from commit 0fa0beacb465c61e792c97d530a0e8fdd7139256)
Eric Masseran [Tue, 2 Nov 2021 20:40:04 +0000 (16:40 -0400)]
Add Non linear CTE support
"Compound select" methods like :meth:`_sql.Select.union`,
:meth:`_sql.Select.intersect_all` etc. now accept ``*other`` as an argument
rather than ``other`` to allow for multiple additional SELECTs to be
compounded with the parent statement at once. In particular, the change as
applied to :meth:`_sql.CTE.union` and :meth:`_sql.CTE.union_all` now allow
for a so-called "non-linear CTE" to be created with the :class:`_sql.CTE`
construct, whereas previously there was no way to have more than two CTE
sub-elements in a UNION together while still correctly calling upon the CTE
in recursive fashion. Pull request courtesy Eric Masseran.
Allow:
```sql
WITH RECURSIVE nodes(x) AS (
SELECT 59
UNION
SELECT aa FROM edge JOIN nodes ON bb=x
UNION
SELECT bb FROM edge JOIN nodes ON aa=x
)
SELECT x FROM nodes;
```
Based on @zzzeek suggestion: https://github.com/sqlalchemy/sqlalchemy/pull/7133#issuecomment-933882348
Mike Bayer [Wed, 10 Nov 2021 14:52:18 +0000 (09:52 -0500)]
qualify asyncpg API tests for python 3.8
Getting
TypeError: object MagicMock can't be used in 'await' expression
for Python 3.7 and earlier. this test is not needed
on all platforms it's confirming that two methods
are present.
Mike Bayer [Mon, 8 Nov 2021 23:29:16 +0000 (18:29 -0500)]
change the POSTCOMPILE/ SCHEMA symbols to not conflict w mssql quoting
Adjusted the compiler's generation of "post compile" symbols including
those used for "expanding IN" as well as for the "schema translate map" to
not be based directly on plain bracketed strings with underscores, as this
conflicts directly with SQL Server's quoting format of also using brackets,
which produces false matches when the compiler replaces "post compile" and
"schema translate" symbols. The issue created easy to reproduce examples
both with the :meth:`.Inspector.get_schema_names` method when used in
conjunction with the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature, as well in the unlikely case that a symbol overlapping with the
internal name "POSTCOMPILE" would be used with a feature like "expanding
in".
Mike Bayer [Tue, 9 Nov 2021 20:02:44 +0000 (15:02 -0500)]
set within_columns_clause=False for all sub-elements of select()
Fixed issue where using the feature of using a string label for ordering or
grouping described at :ref:`tutorial_order_by_label` would fail to function
correctly if used on a :class:`.CTE` construct, when the CTE were embedded
inside of an enclosing :class:`_sql.Select` statement that itself was set
up as a scalar subquery.
Mike Bayer [Tue, 9 Nov 2021 16:31:23 +0000 (11:31 -0500)]
upgrade deferred loader to regular loader if refresh_state
Fixed issue where deferred polymorphic loading of attributes from a
joined-table inheritance subclass would fail to populate the attribute
correctly if the :func:`_orm.load_only` option were used to originally
exclude that attribute, in the case where the load_only were descending
from a relationship loader option. The fix allows that other valid options
such as ``defer(..., raiseload=True)`` etc. still function as expected.
Mike Bayer [Fri, 5 Nov 2021 14:18:42 +0000 (10:18 -0400)]
use tuple expansion if type._is_tuple, test for Sequence if no type
Fixed regression where the row objects returned for ORM queries, which are
now the normal :class:`_sql.Row` objects, would not be interpreted by the
:meth:`_sql.ColumnOperators.in_` operator as tuple values to be broken out
into individual bound parameters, and would instead pass them as single
values to the driver leading to failures. The change to the "expanding IN"
system now accommodates for the expression already being of type
:class:`.TupleType` and treats values accordingly if so. In the uncommon
case of using "tuple-in" with an untyped statement such as a textual
statement with no typing information, a tuple value is detected for values
that implement ``collections.abc.Sequence``, but that are not ``str`` or
``bytes``, as always when testing for ``Sequence``.
Added :class:`.TupleType` to the top level ``sqlalchemy`` import namespace.
Mike Bayer [Thu, 4 Nov 2021 21:02:24 +0000 (17:02 -0400)]
Check for Mapping explicitly in 2.0 params
Fixed issue in future :class:`_future.Connection` object where the
:meth:`_future.Connection.execute` method would not accept a non-dict
mapping object, such as SQLAlchemy's own :class:`.RowMapping` or other
``abc.collections.Mapping`` object as a parameter dictionary.
Mike Bayer [Thu, 4 Nov 2021 17:36:43 +0000 (13:36 -0400)]
Update "transaction has already begun" language
As future connections will now be autobeginning, there
will be more cases where begin() can't be called as well as where isolation level
can't be set, which will be surprising as this is a behavioral
change for 2.0; additionally, when DBAPI autocommit is set,
there isn't actually a DBAPI level transaction in effect even though
Connection has a Transaction object. Clarify the language in these
two error messages to make it clear that begin() and autobegin
are tracking a SQLAlchemy-level Transaction() object, whether or not
the DBAPI has actually started a transaction, and that this is the
reason rollback() or commit() is required before performing
the requsted operation. Additionally make sure the error message
mentions "autobegin" as a likely reason this error is being
encountered along with what Connection needs the user to do in
order to resolve.
Mike Bayer [Wed, 3 Nov 2021 15:32:51 +0000 (11:32 -0400)]
simplify and publicize the asyncpg JSON(B) codec registrsation
Added overridable methods ``PGDialect_asyncpg.setup_asyncpg_json_codec``
and ``PGDialect_asyncpg.setup_asyncpg_jsonb_codec`` codec, which handle the
required task of registering JSON/JSONB codecs for these datatypes when
using asyncpg. The change is that methods are broken out as individual,
overridable methods to support third party dialects that need to alter or
disable how these particular codecs are set up.
Mike Bayer [Thu, 4 Nov 2021 01:26:44 +0000 (21:26 -0400)]
use ExpressionElementRole for case targets in case()
Fixed regression where the :func:`_sql.text` construct would no longer be
accepted as a target case in the "whens" list within a :func:`_sql.case`
construct. The regression appears related to an attempt to guard against
some forms of literal values that were considered to be ambiguous when
passed here; however, there's no reason the target cases shouldn't be
interpreted as open-ended SQL expressions just like anywhere else, and a
literal string or tuple will be converted to a bound parameter as would be
the case elsewhere.
Gord Thompson [Tue, 2 Nov 2021 22:16:50 +0000 (16:16 -0600)]
Gracefully degrade unsupported types with asyncpg
Fixes: #7284
Modify the on_connect() method of PGDialect_asyncpg to
gracefully degrade unsupported types instead of throwing a
ValueError. Useful for third-party dialects that derive
from PGDialect_asyncpg but whose databases do not support
all types (e.g., CockroachDB supports JSONB but not JSON).
Fixed issue in visit_on_duplicate_key_update within a composed expression
Fixed issue in MySQL :meth:`_mysql.Insert.on_duplicate_key_update` which
would render the wrong column name when an expression were used in a VALUES
expression. Pull request courtesy Cristian Sabaila.
Mike Bayer [Tue, 2 Nov 2021 14:58:01 +0000 (10:58 -0400)]
ensure soft_close occurs for fetchmany with server side cursor
Fixed regression where the :meth:`_engine.CursorResult.fetchmany` method
would fail to autoclose a server-side cursor (i.e. when ``stream_results``
or ``yield_per`` is in use, either Core or ORM oriented results) when the
results were fully exhausted.
All :class:`_result.Result` objects will now consistently raise
:class:`_exc.ResourceClosedError` if they are used after a hard close,
which includes the "hard close" that occurs after calling "single row or
value" methods like :meth:`_result.Result.first` and
:meth:`_result.Result.scalar`. This was already the behavior of the most
common class of result objects returned for Core statement executions, i.e.
those based on :class:`_engine.CursorResult`, so this behavior is not new.
However, the change has been extended to properly accommodate for the ORM
"filtering" result objects returned when using 2.0 style ORM queries,
which would previously behave in "soft closed" style of returning empty
results, or wouldn't actually "soft close" at all and would continue
yielding from the underlying cursor.
As part of this change, also added :meth:`_result.Result.close` to the base
:class:`_result.Result` class and implemented it for the filtered result
implementations that are used by the ORM, so that it is possible to call
the :meth:`_engine.CursorResult.close` method on the underlying
:class:`_engine.CursorResult` when the the ``yield_per`` execution option
is in use to close a server side cursor before remaining ORM results have
been fetched. This was again already available for Core result sets but the
change makes it available for 2.0 style ORM results as well.
Mike Bayer [Mon, 1 Nov 2021 20:36:51 +0000 (16:36 -0400)]
use full context manager flow for future.Engine.begin()
Fixed issue in future :class:`_future.Engine` where calling upon
:meth:`_future.Engine.begin` and entering the context manager would not
close the connection if the actual BEGIN operation failed for some reason,
such as an event handler raising an exception; this use case failed to be
tested for the future version of the engine. Note that the "future" context
managers which handle ``begin()`` blocks in Core and ORM don't actually run
the "BEGIN" operation until the context managers are actually entered. This
is different from the legacy version which runs the "BEGIN" operation up
front.
Mike Bayer [Mon, 1 Nov 2021 16:06:32 +0000 (12:06 -0400)]
Revise "literal parameters" FAQ section
based on feedback in #7271, the emphasis on TypeDecorator
as a solution to this problem is not very practical. illustrate
a series of quick recipes that are useful for debugging purposes
to print out a repr() or simple stringify of a parameter
without the need to construct custom dialects or types.
Mike Bayer [Fri, 29 Oct 2021 21:36:26 +0000 (17:36 -0400)]
warnings removal, merge_result
this is the last warning to remove.
Also fixes some mistakes I made with the new
Base20DeprecationWarning and LegacyAPIWarning classes created,
where functions in deprecations.py were still hardcoded to
RemovedIn20Warning.
most of the work for aliased / from_joinpoint has been done
already as I added all new tests for these and moved
most aliased/from_joinpoint to test/orm/test_deprecations.py
already
Mike Bayer [Wed, 27 Oct 2021 15:14:50 +0000 (11:14 -0400)]
consider "inspect(_of_type)" to be the entity of a comparator
Fixed 1.4 regression where :meth:`_orm.Query.filter_by` would not function
correctly when :meth:`_orm.Query.join` were joined to an entity which made
use of :meth:`_orm.PropComparator.of_type` to specify an aliased version of
the target entity. The issue also applies to future style ORM queries
constructed with :func:`_sql.select`.
Mike Bayer [Tue, 26 Oct 2021 19:52:25 +0000 (15:52 -0400)]
add additional "oracle mode" reserved words
despite mariadb's docs, the word "system" must be
quoted in plain mariadb 10.5, not sure if that's
"oracle mode" but it is > 10.3. observed keystone
tests failing on a column of this name.
Mike Bayer [Tue, 26 Oct 2021 15:28:42 +0000 (11:28 -0400)]
dont pull filter_by from from_obj, entity is hopefully sufficient
Fixed 1.4 regression where :meth:`_orm.Query.filter_by` would not function
correctly on a :class:`_orm.Query` that was produced from
:meth:`_orm.Query.union`, :meth:`_orm.Query.from_self` or similar.
jonathan vanasco [Fri, 24 Sep 2021 18:55:59 +0000 (14:55 -0400)]
Fixes: #4100
Add a warning, in two places, stating `with_for_update` will lock joinedload
tables, because at least one person did not expect the obvious to happen.
Also warn that eager loading techniques may not work with `with_for_update`
and combining the two is not officially supported or recommended.
Mike Bayer [Wed, 20 Oct 2021 16:50:53 +0000 (12:50 -0400)]
deprecation warnings: strings in loader options, join, with_parent
Repairs one in-library deprecation warning regarding
mapper propagation of options
raises maxfail to 250, as 25 is too low when we are trying
to address many errors at once. the 25 was originally
due to the fact that our fixtures would be broken after
that many failures in most cases, which today should not
be the case nearly as often.
Mike Bayer [Sun, 24 Oct 2021 00:36:41 +0000 (20:36 -0400)]
add full docs/notes for all TypeEngine/TypeDecorator interactions
Added notes to all TypeEngine methods as to which should be
overridden by TypeDecorator implementations. Made sure
docstrings are present for all TypeDecorator methods as some
were missing, and clarified which should be overridden, which
should not, as well as what phase each method is called within.
Mike Bayer [Fri, 22 Oct 2021 16:31:06 +0000 (12:31 -0400)]
use correct entity in path for aliased class relationship
Fixed bug in "relationship to aliased class" feature introduced at
:ref:`relationship_aliased_class` where it was not possible to create a
loader strategy option targeting an attribute on the target using the
:func:`_orm.aliased` construct directly in a second loader option, such as
``selectinload(A.aliased_bs).joinedload(aliased_b.cs)``, without explicitly
qualifying using :meth:`_orm.PropComparator.of_type` on the preceding
element of the path. Additionally, targeting the non-aliased class directly
would be accepted (inappropriately), but would silently fail, such as
``selectinload(A.aliased_bs).joinedload(B.cs)``; this now raises an error
referring to the typing mismatch.
Gord Thompson [Wed, 20 Oct 2021 21:15:37 +0000 (15:15 -0600)]
Apply minor update to async_orm example
- Fixed import to avoid MovedIn20Warning.
- Separated drop_all() and create_all() into separate
context managers to avoid dropping and creating the
same table within the same transaction. Apparently
some databases (e.g., CockroachDB) don't allow that.
Kevin Kirsche [Tue, 19 Oct 2021 21:22:03 +0000 (17:22 -0400)]
fix: Update reserved words list of MySQL / MariaDB dialect
Reorganized the list of reserved words into two separate lists, one for
MySQL and one for MariaDB, so that these diverging sets of words can be
managed more accurately; adjusted the MySQL/MariaDB dialect to switch among
these lists based on either explicitly configured or
server-version-detected "MySQL" or "MariaDB" backend. Added all current
reserved words through MySQL 8 and current MariaDB versions including
recently added keywords like "lead" . Pull request courtesy Kevin Kirsche.
1. Move reserved words to it's own file.
2. Add missing reserved words from https://mariadb.com/kb/en/reserved-words/
* Note: this only adds MariaDB though links to MySQL, it also does not
include the reserved words for Oracle mode, as listed in the link.
Mike Bayer [Mon, 18 Oct 2021 20:35:21 +0000 (16:35 -0400)]
remove _resolve_label and related attributes
these seem to date back to 0.9 and revs like #3148 but none
of it seems to be affecting things now, try removing it all
and seeing what fails.
we've noted that _resolve_label does not appear to be
needed, however allow_label_resolve remains relevant both
for non-column elements like text() as well as that it is
used explicitly by the joined eager loader.
Mike Bayer [Tue, 19 Oct 2021 18:07:32 +0000 (14:07 -0400)]
process bulk_update_tuples before cache key or compilation
Fixed regression where the use of a :class:`_orm.hybrid_property` attribute
or a mapped :func:`_orm.composite` attribute as a key passed to the
:meth:`_dml.Update.values` method for an ORM-enabled :class:`_dml.Update`
statement, as well as when using it via the legacy
:meth:`_orm.Query.update` method, would be processed for incoming
ORM/hybrid/composite values within the compilation stage of the UPDATE
statement, which meant that in those cases where caching occurred,
subsequent invocations of the same statement would no longer receive the
correct values. This would include not only hybrids that use the
:meth:`_orm.hybrid_property.update_expression` method, but any use of a
plain hybrid attribute as well. For composites, the issue instead caused a
non-repeatable cache key to be generated, which would break caching and
could fill up the statement cache with repeated statements.
The :class:`_dml.Update` construct now handles the processing of key/value
pairs passed to :meth:`_dml.Update.values` and
:meth:`_dml.Update.ordered_values` up front when the construct is first
generated, before the cache key has been generated so that the key/value
pairs are processed each time, and so that the cache key is generated
against the individual column/value pairs that will ultimately be
used in the statement.
Mike Bayer [Mon, 18 Oct 2021 19:43:27 +0000 (15:43 -0400)]
use coercions for label element, ensure propagate_attrs
Fixed bug where the ORM "plugin", necessary for features such as
:func:`_orm.with_loader_criteria` to work correctly, would not be applied
to a :func:`_sql.select` which queried from an ORM column expression if it
made use of the :meth:`_sql.ColumnElement.label` modifier.
Mike Bayer [Fri, 15 Oct 2021 15:03:46 +0000 (11:03 -0400)]
fix with_loader_criteria for select(A).join(B)
Also revise the contains_eager() note that was just
added in fec2b6560c14bb28ee7f, as I mistakenly forgot that
WLC also affects elements of the query that are there
as part of the normal entites/criteria.
Mike Bayer [Wed, 13 Oct 2021 19:52:12 +0000 (15:52 -0400)]
support bind expressions w/ expanding IN; apply to psycopg2
Fixed issue where "expanding IN" would fail to function correctly with
datatypes that use the :meth:`_types.TypeEngine.bind_expression` method,
where the method would need to be applied to each element of the
IN expression rather than the overall IN expression itself.
Fixed issue where IN expressions against a series of array elements, as can
be done with PostgreSQL, would fail to function correctly due to multiple
issues within the "expanding IN" feature of SQLAlchemy Core that was
standardized in version 1.4. The psycopg2 dialect now makes use of the
:meth:`_types.TypeEngine.bind_expression` method with :class:`_types.ARRAY`
to portably apply the correct casts to elements. The asyncpg dialect was
not affected by this issue as it applies bind-level casts at the driver
level rather than at the compiler level.
as part of this commit the "bind translate" feature has been
simplified and also applies to the names in the POSTCOMPILE tag to
accommodate for brackets.
Mike Bayer [Thu, 14 Oct 2021 22:24:26 +0000 (18:24 -0400)]
clarify that with_loader_criteria does not affect contains_eager()
The ORM contains_eager() option does not affect the rendering
of a SQL query in any way except that it adds additional columns
to the columns clause of the SELECT. Therefore the
with_loader_criteria() function has no interaction with this
loader option.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
