Mike Bayer [Fri, 24 Jun 2022 14:31:46 +0000 (10:31 -0400)]
add fallback for old mutable format
Fixed regression caused by :ticket:`8133` where the pickle format for
mutable attributes was changed, without a fallback to recognize the old
format, causing in-place upgrades of SQLAlchemy to no longer be able to
read pickled data from previous versions. A check plus a fallback for the
old format is now in place.
Mike Bayer [Thu, 23 Jun 2022 15:15:19 +0000 (11:15 -0400)]
refine _include_fn to not include sibling mappers
Fixed regression caused by :ticket:`8064` where a particular check for
column correspondence was made too liberal, resulting in incorrect
rendering for some ORM subqueries such as those using
:meth:`.PropComparator.has` or :meth:`.PropComparator.any` in conjunction
with joined-inheritance queries that also use legacy aliasing features.
Mike Bayer [Wed, 22 Jun 2022 22:50:35 +0000 (18:50 -0400)]
sub-categorize special function forms
this is the tutorial, which should have some semblence of
not getting too far into the weeds. however, as we dont
really have other places to explain SQL concepts, and SQL
functions have a lot of them, we dont have another home right
now. so at least further sub-categorize window functions,
table/column valued functions, and WITHIN GROUP into an
"advanced function techniques" section with a disclaimer that
these are less common use cases.
Mike Bayer [Mon, 13 Jun 2022 15:46:28 +0000 (11:46 -0400)]
rework ORM mapping docs
prepare docs for newly incoming mapper styles, including
new dataclass mapping. move the existing dataclass/attrs
docs all into their own section and try to improve organization
and wording into the relatively recent "mapping styles"
document.
David Baumgold [Fri, 11 Feb 2022 17:30:24 +0000 (12:30 -0500)]
Domain type
Added a new Postgresql :class:`_postgresql.DOMAIN` datatype, which follows
the same CREATE TYPE / DROP TYPE behaviors as that of PostgreSQL
:class:`_postgresql.ENUM`. Much thanks to David Baumgold for the efforts on
this.
Mike Bayer [Mon, 20 Jun 2022 15:06:34 +0000 (11:06 -0400)]
remove warnings for index/unique skipped due to exclude_cols
The warnings that are emitted regarding reflection of indexes or unique
constraints, when the :paramref:`.Table.include_columns` parameter is used
to exclude columns that are then found to be part of those constraints,
have been removed. When the :paramref:`.Table.include_columns` parameter is
used it should be expected that the resulting :class:`.Table` construct
will not include constraints that rely upon omitted columns. This change
was made in response to :ticket:`8100` which repaired
:paramref:`.Table.include_columns` in conjunction with foreign key
constraints that rely upon omitted columns, where the use case became
clear that omitting such constraints should be expected.
Mike Bayer [Thu, 16 Jun 2022 17:35:16 +0000 (13:35 -0400)]
create new approach for deeply nested post loader options
Added very experimental feature to the :func:`_orm.selectinload` and
:func:`_orm.immediateload` loader options called
:paramref:`_orm.selectinload.recursion_depth` /
:paramref:`_orm.immediateload.recursion_depth` , which allows a single
loader option to automatically recurse into self-referential relationships.
Is set to an integer indicating depth, and may also be set to -1 to
indicate to continue loading until no more levels deep are found.
Major internal changes to :func:`_orm.selectinload` and
:func:`_orm.immediateload` allow this feature to work while continuing
to make correct use of the compilation cache, as well as not using
arbitrary recursion, so any level of depth is supported (though would
emit that many queries). This may be useful for
self-referential structures that must be loaded fully eagerly, such as when
using asyncio.
A warning is also emitted when loader options are connected together with
arbitrary lengths (that is, without using the new ``recursion_depth``
option) when excessive recursion depth is detected in related object
loading. This operation continues to use huge amounts of memory and
performs extremely poorly; the cache is disabled when this condition is
detected to protect the cache from being flooded with arbitrary statements.
Federico Caselli [Thu, 14 Oct 2021 19:45:57 +0000 (21:45 +0200)]
rearchitect reflection for batched performance
Rearchitected the schema reflection API to allow some dialects to make use
of high performing batch queries to reflect the schemas of many tables at
once using much fewer queries. The new performance features are targeted
first at the PostgreSQL and Oracle backends, and may be applied to any
dialect that makes use of SELECT queries against system catalog tables to
reflect tables (currently this omits the MySQL and SQLite dialects which
instead make use of parsing the "CREATE TABLE" statement, however these
dialects do not have a pre-existing performance issue with reflection. MS
SQL Server is still a TODO).
The new API is backwards compatible with the previous system, and should
require no changes to third party dialects to retain compatibility;
third party dialects can also opt into the new system by implementing
batched queries for schema reflection.
Along with this change is an updated reflection API that is fully
:pep:`484` typed, features many new methods and some changes.
Gord Thompson [Tue, 14 Jun 2022 16:09:04 +0000 (10:09 -0600)]
Allow NUMERIC()/DECIMAL() IDENTITY columns
Fixed issue where :class:`.Table` objects that made use of IDENTITY columns
with a :class:`.Numeric` datatype would produce errors when attempting to
reconcile the "autoincrement" column, preventing construction of the
:class:`.Column` from using the :paramref:`.Column.autoincrement` parameter
as well as emitting errors when attempting to invoke an :class:`.Insert`
construct.
Mike Bayer [Thu, 16 Jun 2022 18:46:11 +0000 (14:46 -0400)]
Revert "add auto_recurse option to selectinload, immediateload"
this option works very badly with caching and the API
is likely not what we want either. Work continues for
#8126 including that the additional work in
I9f162e0a09c1ed327dd19498aac193f649333a01
tries to add new recursive features.
Mike Bayer [Wed, 15 Jun 2022 16:42:44 +0000 (12:42 -0400)]
implement literal stringification for arrays
as we already implement stringification for the contents,
provide a bracketed syntax for default and ARRAY literal
for PG specifically. ARRAY literal seems much simpler to
render than their quoted syntax which requires double quotes
for strings.
also open up testing for pg8000 which has likely been
fine with arrays for awhile now, bump the version pin
also.
Mike Bayer [Tue, 14 Jun 2022 21:05:44 +0000 (17:05 -0400)]
new features for pep 593 Annotated
* extract the inner type from Annotated when the outer type
isn't present in the type map, to allow for arbitrary Annotated
* allow _IntrospectsAnnotations objects to be directly present
in an Annotated and resolve the mapper property from that.
Currently implemented for mapped_column(), with message for
others. Can work for composite() and likely some
relationship() as well at some point
References: https://twitter.com/zzzeek/status/1536693554621341697 and
replies
Mike Bayer [Tue, 14 Jun 2022 19:41:31 +0000 (15:41 -0400)]
pickle mutable parents according to key
Fixed bug in :class:`.Mutable` where pickling and unpickling of an ORM
mapped instance would not correctly restore state for mappings that
contained multiple :class:`.Mutable`-enabled attributes.
Mike Bayer [Tue, 14 Jun 2022 13:31:09 +0000 (09:31 -0400)]
typing adjustments for composites
* if dataclass isn't used, columns have to be named
* _CompositeClassProto is not useful as dataclasses have no
methods / bases we can use, so composite is against Any
* Adjust session.get() feature to work w/ dataclass composites
Mike Bayer [Sat, 11 Jun 2022 15:33:46 +0000 (11:33 -0400)]
add auto_recurse option to selectinload, immediateload
Added very experimental feature to the :func:`_orm.selectinload` and
:func:`_orm.immediateload` loader options called
:paramref:`_orm.selectinload.auto_recurse` /
:paramref:`_orm.immediateload.auto_recurse` , which when set to True will
cause a self-referential relationship load to continue loading with
arbitrary depth until no further objects are found. This may be useful for
self-referential structures that must be loaded fully eagerly, such as when
using asyncio.
Mike Bayer [Fri, 10 Jun 2022 16:57:53 +0000 (12:57 -0400)]
resolve large ints to BigInteger
The in-place type detection for Python integers, as occurs with an
expression such as ``literal(25)``, will now apply value-based adaption as
well to accommodate Python large integers, where the datatype determined
will be :class:`.BigInteger` rather than :class:`.Integer`. This
accommodates for dialects such as that of asyncpg which both sends implicit
typing information to the driver as well as is sensitive to numeric scale.
Mike Bayer [Fri, 10 Jun 2022 16:42:54 +0000 (12:42 -0400)]
honor enum length in all cases
The :paramref:`.Enum.length` parameter, which sets the length of the
``VARCHAR`` column for non-native enumeration types, is now used
unconditionally when emitting DDL for the ``VARCHAR`` datatype, including
when the :paramref:`.Enum.native_enum` parameter is set to ``True`` for
target backends that continue to use ``VARCHAR``. Previously the parameter
would be erroneously ignored in this case. The warning previously emitted
for this case is now removed.
Mike Bayer [Fri, 10 Jun 2022 15:44:45 +0000 (11:44 -0400)]
remove "undefer_pks" as a strategy option
The behavior of :func:`_orm.defer` regarding primary key and "polymorphic
discriminator" columns is revised such that these columns are no longer
deferrable, either explicitly or when using a wildcard such as
``defer('*')``. Previously, a wildcard deferral would not load
PK/polymorphic columns which led to errors in all cases, as the ORM relies
upon these columns to produce object identities. The behavior of explicit
deferral of primary key columns is unchanged as these deferrals already
were implicitly ignored.
Mike Bayer [Tue, 7 Jun 2022 20:09:35 +0000 (16:09 -0400)]
update cx_Oracle / oracledb LOB handling
Adjustments made to the BLOB / CLOB / NCLOB datatypes in the cx_Oracle and
oracledb dialects, to improve performance based on recommendations from
Oracle developers.
Mike Bayer [Thu, 9 Jun 2022 01:35:02 +0000 (21:35 -0400)]
restore parameter escaping for public methods
Adjusted the fix made for :ticket:`8056` which adjusted the escaping of
bound parameter names with special characters such that the escaped names
were translated after the SQL compilation step, which broke a published
recipe on the FAQ illustrating how to merge parameter names into the string
output of a compiled SQL string. The change restores the escaped names that
come from ``compiled.params`` and adds a conditional parameter to
:meth:`.SQLCompiler.construct_params` named ``escape_names`` that defaults
to ``True``, restoring the old behavior by default.
Mike Bayer [Thu, 9 Jun 2022 13:53:43 +0000 (09:53 -0400)]
dont transfer __weakref__ to regenerated class
Repaired a deprecation warning class decorator that was preventing key
objects such as :class:`_engine.Connection` from having a proper
``__weakref__`` attribute, causing operations like Python standard library
``inspect.getmembers()`` to fail.
Fixed issue where a :func:`_orm.with_loader_criteria` option could not be
pickled, as is necessary when it is carried along for propagation to lazy
loaders in conjunction with a caching scheme. Currently, the only form that
is supported as picklable is to pass the "where criteria" as a fixed
module-level callable function that produces a SQL expression. An ad-hoc
"lambda" can't be pickled, and a SQL expression object is usually not fully
picklable directly.
Mike Bayer [Tue, 7 Jun 2022 19:00:20 +0000 (15:00 -0400)]
fix race conditions in lambda statements
Fixed multiple observed race conditions related to :func:`.lambda_stmt`,
including an initial "dogpile" issue when a new Python code object is
initially analyzed among multiple simultaneous threads which created both a
performance issue as well as some internal corruption of state.
Additionally repaired observed race condition which could occur when
"cloning" an expression construct that is also in the process of being
compiled or otherwise accessed in a different thread due to memoized
attributes altering the ``__dict__`` while iterated, for Python versions
prior to 3.10; in particular the lambda SQL construct is sensitive to this
as it holds onto a single statement object persistently. The iteration has
been refined to use ``dict.copy()`` with or without an additional iteration
instead.
Mike Bayer [Tue, 7 Jun 2022 13:40:26 +0000 (09:40 -0400)]
graceful degrade for FKs not reflectable
Fixed bugs involving the :paramref:`.Table.include_columns` and the
:paramref:`.Table.resolve_fks` parameters on :class:`.Table`; these
little-used parameters were apparently not working for columns that refer
to foreign key constraints.
In the first case, not-included columns that refer to foreign keys would
still attempt to create a :class:`.ForeignKey` object, producing errors
when attempting to resolve the columns for the foreign key constraint
within reflection; foreign key constraints that refer to skipped columns
are now omitted from the table reflection process in the same way as
occurs for :class:`.Index` and :class:`.UniqueConstraint` objects with the
same conditions. No warning is produced however, as we likely want to
remove the include_columns warnings for all constraints in 2.0.
In the latter case, the production of table aliases or subqueries would
fail on an FK related table not found despite the presence of
``resolve_fks=False``; the logic has been repaired so that if a related
table is not found, the :class:`.ForeignKey` object is still proxied to the
aliased table or subquery (these :class:`.ForeignKey` objects are normally
used in the production of join conditions), but it is sent with a flag that
it's not resolvable. The aliased table / subquery will then work normally,
with the exception that it cannot be used to generate a join condition
automatically, as the foreign key information is missing. This was already
the behavior for such foreign key constraints produced using non-reflection
methods, such as joining :class:`.Table` objects from different
:class:`.MetaData` collections.
Mike Bayer [Fri, 3 Jun 2022 14:34:19 +0000 (10:34 -0400)]
some typing fixes
* ClassVar for decl fields, add __tablename__
* dataclasses require annotations for all fields. For us,
if no annotation, then skip that field as part of what is
considered to be a "dataclass", as this matches the behavior
of pyright right now. We could alternatively raise on this
use, which is what dataclasses does. we should ask the pep
people
* plain field that's just "str", "int", etc., with no value.
Disallow it unless __allow_unmapped__ is set. If field
has dataclasses.field, Column, None, a value etc, it goes through,
and when using dataclasses mixin all such fields are considered
for the dataclass setup just like a dataclass. Hopefully this
does not have major backwards compat issues. __allow_unmapped__
can be set on the base class, mixins, etc., it's liberal for
now in case people have this problem.
* accommodate for ClassVar, these are not considered at all for
mapping.
Daniel Black [Tue, 28 Sep 2021 18:20:06 +0000 (14:20 -0400)]
Generalize RETURNING and suppor for MariaDB / SQLite
As almost every dialect supports RETURNING now, RETURNING
is also made more of a default assumption.
* the default compiler generates a RETURNING clause now
when specified; CompileError is no longer raised.
* The dialect-level implicit_returning parameter now has
no effect. It's not fully clear if there are real world
cases relying on the dialect-level parameter, so we will see
once 2.0 is released. ORM-level RETURNING can be disabled
at the table level, and perhaps "implicit returning" should
become an ORM-level option at some point as that's where
it applies.
* Altered ORM update() / delete() to respect table-level
implicit returning for fetch.
* Since MariaDB doesnt support UPDATE returning, "full_returning"
is now split into insert_returning, update_returning, delete_returning
* Crazy new thing. Dialects that have *both* cursor.lastrowid
*and* returning. so now we can pick between them for SQLite
and mariadb. so, we are trying to keep it on .lastrowid for
simple inserts with an autoincrement column, this helps with
some edge case test scenarios and i bet .lastrowid is faster
anyway. any return_defaults() / multiparams etc then we
use returning
* SQLite decided they dont want to return rows that match in
ON CONFLICT. this is flat out wrong, but for now we need to
work with it.
Mike Bayer [Fri, 11 Feb 2022 09:54:45 +0000 (04:54 -0500)]
add backend agnostic UUID datatype
Added new backend-agnostic :class:`_types.Uuid` datatype generalized from
the PostgreSQL dialects to now be a core type, as well as migrated
:class:`_types.UUID` from the PostgreSQL dialect. Thanks to Trevor Gross
for the help on this.
also includes:
* corrects some missing behaviors in the suite literal fixtures
test where row round trips weren't being correctly asserted.
* fixes some of the ISO literal date rendering added in 952383f9ee0 for #5052 to truncate datetime strings for date/time
datatypes in the same way that drivers typically do for bound
parameters; this was not working fully and wasn't caught by the
broken test fixture
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
