Mike Bayer [Tue, 14 Jun 2022 13:31:09 +0000 (09:31 -0400)]
typing adjustments for composites
* if dataclass isn't used, columns have to be named
* _CompositeClassProto is not useful as dataclasses have no
methods / bases we can use, so composite is against Any
* Adjust session.get() feature to work w/ dataclass composites
Mike Bayer [Sat, 11 Jun 2022 15:33:46 +0000 (11:33 -0400)]
add auto_recurse option to selectinload, immediateload
Added very experimental feature to the :func:`_orm.selectinload` and
:func:`_orm.immediateload` loader options called
:paramref:`_orm.selectinload.auto_recurse` /
:paramref:`_orm.immediateload.auto_recurse` , which when set to True will
cause a self-referential relationship load to continue loading with
arbitrary depth until no further objects are found. This may be useful for
self-referential structures that must be loaded fully eagerly, such as when
using asyncio.
Mike Bayer [Fri, 10 Jun 2022 16:57:53 +0000 (12:57 -0400)]
resolve large ints to BigInteger
The in-place type detection for Python integers, as occurs with an
expression such as ``literal(25)``, will now apply value-based adaption as
well to accommodate Python large integers, where the datatype determined
will be :class:`.BigInteger` rather than :class:`.Integer`. This
accommodates for dialects such as that of asyncpg which both sends implicit
typing information to the driver as well as is sensitive to numeric scale.
Mike Bayer [Fri, 10 Jun 2022 16:42:54 +0000 (12:42 -0400)]
honor enum length in all cases
The :paramref:`.Enum.length` parameter, which sets the length of the
``VARCHAR`` column for non-native enumeration types, is now used
unconditionally when emitting DDL for the ``VARCHAR`` datatype, including
when the :paramref:`.Enum.native_enum` parameter is set to ``True`` for
target backends that continue to use ``VARCHAR``. Previously the parameter
would be erroneously ignored in this case. The warning previously emitted
for this case is now removed.
Mike Bayer [Fri, 10 Jun 2022 15:44:45 +0000 (11:44 -0400)]
remove "undefer_pks" as a strategy option
The behavior of :func:`_orm.defer` regarding primary key and "polymorphic
discriminator" columns is revised such that these columns are no longer
deferrable, either explicitly or when using a wildcard such as
``defer('*')``. Previously, a wildcard deferral would not load
PK/polymorphic columns which led to errors in all cases, as the ORM relies
upon these columns to produce object identities. The behavior of explicit
deferral of primary key columns is unchanged as these deferrals already
were implicitly ignored.
Mike Bayer [Tue, 7 Jun 2022 20:09:35 +0000 (16:09 -0400)]
update cx_Oracle / oracledb LOB handling
Adjustments made to the BLOB / CLOB / NCLOB datatypes in the cx_Oracle and
oracledb dialects, to improve performance based on recommendations from
Oracle developers.
Mike Bayer [Thu, 9 Jun 2022 01:35:02 +0000 (21:35 -0400)]
restore parameter escaping for public methods
Adjusted the fix made for :ticket:`8056` which adjusted the escaping of
bound parameter names with special characters such that the escaped names
were translated after the SQL compilation step, which broke a published
recipe on the FAQ illustrating how to merge parameter names into the string
output of a compiled SQL string. The change restores the escaped names that
come from ``compiled.params`` and adds a conditional parameter to
:meth:`.SQLCompiler.construct_params` named ``escape_names`` that defaults
to ``True``, restoring the old behavior by default.
Mike Bayer [Thu, 9 Jun 2022 13:53:43 +0000 (09:53 -0400)]
dont transfer __weakref__ to regenerated class
Repaired a deprecation warning class decorator that was preventing key
objects such as :class:`_engine.Connection` from having a proper
``__weakref__`` attribute, causing operations like Python standard library
``inspect.getmembers()`` to fail.
Fixed issue where a :func:`_orm.with_loader_criteria` option could not be
pickled, as is necessary when it is carried along for propagation to lazy
loaders in conjunction with a caching scheme. Currently, the only form that
is supported as picklable is to pass the "where criteria" as a fixed
module-level callable function that produces a SQL expression. An ad-hoc
"lambda" can't be pickled, and a SQL expression object is usually not fully
picklable directly.
Mike Bayer [Tue, 7 Jun 2022 19:00:20 +0000 (15:00 -0400)]
fix race conditions in lambda statements
Fixed multiple observed race conditions related to :func:`.lambda_stmt`,
including an initial "dogpile" issue when a new Python code object is
initially analyzed among multiple simultaneous threads which created both a
performance issue as well as some internal corruption of state.
Additionally repaired observed race condition which could occur when
"cloning" an expression construct that is also in the process of being
compiled or otherwise accessed in a different thread due to memoized
attributes altering the ``__dict__`` while iterated, for Python versions
prior to 3.10; in particular the lambda SQL construct is sensitive to this
as it holds onto a single statement object persistently. The iteration has
been refined to use ``dict.copy()`` with or without an additional iteration
instead.
Mike Bayer [Tue, 7 Jun 2022 13:40:26 +0000 (09:40 -0400)]
graceful degrade for FKs not reflectable
Fixed bugs involving the :paramref:`.Table.include_columns` and the
:paramref:`.Table.resolve_fks` parameters on :class:`.Table`; these
little-used parameters were apparently not working for columns that refer
to foreign key constraints.
In the first case, not-included columns that refer to foreign keys would
still attempt to create a :class:`.ForeignKey` object, producing errors
when attempting to resolve the columns for the foreign key constraint
within reflection; foreign key constraints that refer to skipped columns
are now omitted from the table reflection process in the same way as
occurs for :class:`.Index` and :class:`.UniqueConstraint` objects with the
same conditions. No warning is produced however, as we likely want to
remove the include_columns warnings for all constraints in 2.0.
In the latter case, the production of table aliases or subqueries would
fail on an FK related table not found despite the presence of
``resolve_fks=False``; the logic has been repaired so that if a related
table is not found, the :class:`.ForeignKey` object is still proxied to the
aliased table or subquery (these :class:`.ForeignKey` objects are normally
used in the production of join conditions), but it is sent with a flag that
it's not resolvable. The aliased table / subquery will then work normally,
with the exception that it cannot be used to generate a join condition
automatically, as the foreign key information is missing. This was already
the behavior for such foreign key constraints produced using non-reflection
methods, such as joining :class:`.Table` objects from different
:class:`.MetaData` collections.
Mike Bayer [Fri, 3 Jun 2022 14:34:19 +0000 (10:34 -0400)]
some typing fixes
* ClassVar for decl fields, add __tablename__
* dataclasses require annotations for all fields. For us,
if no annotation, then skip that field as part of what is
considered to be a "dataclass", as this matches the behavior
of pyright right now. We could alternatively raise on this
use, which is what dataclasses does. we should ask the pep
people
* plain field that's just "str", "int", etc., with no value.
Disallow it unless __allow_unmapped__ is set. If field
has dataclasses.field, Column, None, a value etc, it goes through,
and when using dataclasses mixin all such fields are considered
for the dataclass setup just like a dataclass. Hopefully this
does not have major backwards compat issues. __allow_unmapped__
can be set on the base class, mixins, etc., it's liberal for
now in case people have this problem.
* accommodate for ClassVar, these are not considered at all for
mapping.
Daniel Black [Tue, 28 Sep 2021 18:20:06 +0000 (14:20 -0400)]
Generalize RETURNING and suppor for MariaDB / SQLite
As almost every dialect supports RETURNING now, RETURNING
is also made more of a default assumption.
* the default compiler generates a RETURNING clause now
when specified; CompileError is no longer raised.
* The dialect-level implicit_returning parameter now has
no effect. It's not fully clear if there are real world
cases relying on the dialect-level parameter, so we will see
once 2.0 is released. ORM-level RETURNING can be disabled
at the table level, and perhaps "implicit returning" should
become an ORM-level option at some point as that's where
it applies.
* Altered ORM update() / delete() to respect table-level
implicit returning for fetch.
* Since MariaDB doesnt support UPDATE returning, "full_returning"
is now split into insert_returning, update_returning, delete_returning
* Crazy new thing. Dialects that have *both* cursor.lastrowid
*and* returning. so now we can pick between them for SQLite
and mariadb. so, we are trying to keep it on .lastrowid for
simple inserts with an autoincrement column, this helps with
some edge case test scenarios and i bet .lastrowid is faster
anyway. any return_defaults() / multiparams etc then we
use returning
* SQLite decided they dont want to return rows that match in
ON CONFLICT. this is flat out wrong, but for now we need to
work with it.
Mike Bayer [Fri, 11 Feb 2022 09:54:45 +0000 (04:54 -0500)]
add backend agnostic UUID datatype
Added new backend-agnostic :class:`_types.Uuid` datatype generalized from
the PostgreSQL dialects to now be a core type, as well as migrated
:class:`_types.UUID` from the PostgreSQL dialect. Thanks to Trevor Gross
for the help on this.
also includes:
* corrects some missing behaviors in the suite literal fixtures
test where row round trips weren't being correctly asserted.
* fixes some of the ISO literal date rendering added in 952383f9ee0 for #5052 to truncate datetime strings for date/time
datatypes in the same way that drivers typically do for bound
parameters; this was not working fully and wasn't caught by the
broken test fixture
Patrick Gerken [Tue, 31 May 2022 19:42:35 +0000 (21:42 +0200)]
Update declarative_styles.rst: Update code example to attrs TNG usage. (#8072)
* Update declarative_styles.rst
Update docs to new style usage of attrs.
This is the default since December 2021.
While the old style still works, the newer one looks much nicer and is likely to be dominant pretty quickly. Imho.
Mike Bayer [Thu, 26 May 2022 18:35:03 +0000 (14:35 -0400)]
establish sessionmaker and async_sessionmaker as generic
This is so that custom Session and AsyncSession classes
can be typed for these factories. Added appropriate
typevars to `__call__()`, `__enter__()` and other methods
so that a custom Session or AsyncSession subclass is carried
through.
Mike Bayer [Mon, 30 May 2022 16:29:58 +0000 (12:29 -0400)]
Support handle_error for pre_ping
The :meth:`.DialectEvents.handle_error` event is now moved to the
:class:`.DialectEvents` suite from the :class:`.EngineEvents` suite, and
now participates in the connection pool "pre ping" event for those dialects
that make use of disconnect codes in order to detect if the database is
live. This allows end-user code to alter the state of "pre ping". Note that
this does not include dialects which contain a native "ping" method such as
that of psycopg2 or most MySQL dialects.
Mike Bayer [Tue, 31 May 2022 14:48:16 +0000 (10:48 -0400)]
raise informative error when selectable can't be extended
An informative error is raised for the use case where
:meth:`.Insert.from_select` is being passed a "compound select" object such
as a UNION, yet the INSERT statement needs to append additional columns to
support Python-side or explicit SQL defaults from the table metadata. In
this case a subquery of the compound object should be passed.
Mike Bayer [Fri, 27 May 2022 20:07:01 +0000 (16:07 -0400)]
remove "deannotate" from column_property expression
Fixed issue where using a :func:`_orm.column_property` construct containing
a subquery against an already-mapped column attribute would not correctly
apply ORM-compilation behaviors to the subquery, including that the "IN"
expression added for a single-table inherits expression would fail to be
included.
This fix involves a few tweaks in the ORM adaptation logic,
including a missing "parententity" adaptation on the mapper
side. The specific mechanics here have a lot of moving parts
so we will continue to add tests to assert these cases. In
particular a more complete test for issue #2316 is added
that was relying upon the deannotate happening here.
Mike Bayer [Sun, 29 May 2022 16:07:46 +0000 (12:07 -0400)]
move bindparam quote application from compiler to default
in 296c84313ab29bf9599634f3 for #5653 we generalized Oracle's
parameter escaping feature into the compiler, so that it could also
work for PostgreSQL. The compiler used quoted names within parameter
dictionaries, which then led to the complexity that all functions
which interpreted keys from the compiled_params dict had to
also quote the param names to use the dictionary. This
extra complexity was not added to the ORM peristence.py however,
which led to the versioning id feature being broken as well as
other areas where persistence.py relies on naming schemes present
in context.compiled_params. It also was not added to the
"processors" lookup which led to #8053, that added this escaping
to that part of the compiler.
To both solve the whole problem as well as simplify the compiler
quite a bit, move the actual application of the escaped names
to be as late as possible, when default.py builds the final list
of parameters. This is more similar to how it worked previously
where OracleExecutionContext would be late-applying these
escaped names. This re-establishes context.compiled_params as
deterministically named regardless of dialect in use and moves
out the complexity of the quoted param names to be only at the
cursor.execute stage.
Fixed bug, likely a regression from 1.3, where usage of column names that
require bound parameter escaping, more concretely when using Oracle with
column names that require quoting such as those that start with an
underscore, or in less common cases with some PostgreSQL drivers when using
column names that contain percent signs, would cause the ORM versioning
feature to not work correctly if the versioning column itself had such a
name, as the ORM assumes certain bound parameter naming conventions that
were being interfered with via the quotes. This issue is related to
:ticket:`8053` and essentially revises the approach towards fixing this,
revising the original issue :ticket:`5653` that created the initial
implementation for generalized bound-parameter name quoting.
Mike Bayer [Wed, 25 May 2022 12:47:29 +0000 (08:47 -0400)]
apply bindparam escape name to processors dictionary
Fixed SQL compiler issue where the "bind processing" function for a bound
parameter would not be correctly applied to a bound value if the bound
parameter's name were "escaped". Concretely, this applies, among other
cases, to Oracle when a :class:`.Column` has a name that itself requires
quoting, such that the quoting-required name is then used for the bound
parameters generated within DML statements, and the datatype in use
requires bind processing, such as the :class:`.Enum` datatype.
Andrew Brookins [Sun, 22 May 2022 20:24:15 +0000 (16:24 -0400)]
Add a note on using server_onupdate=FetchedValue()
Add a note on using `server_onupdate=FetchedValue()` when using SQL expressions with `onupdate`.
My team encountered an issue with using a SQL expression with `onupdate`.
Despite the dialect (PG) supporting `RETURNING`, we needed to mark the column with
`server_onupdate=FetchedValue()` in order to get the column used with `onupdate`
to appear in the `RETURNING` clause of `UPDATE` statements.
This was not clear from the documentation, so I want to make it crystal clear for other
folks defining similar columns.
Mike Bayer [Wed, 18 May 2022 20:06:29 +0000 (16:06 -0400)]
use plainto_tsquery for PG match
The :meth:`.Operators.match` operator now uses ``plainto_tsquery()`` for
PostgreSQL full text search, rather than ``to_tsquery()``. The rationale
for this change is to provide better cross-compatibility with match on
other database backends. Full support for all PostgreSQL full text
functions remains available through the use of :data:`.func` in
conjunction with :meth:`.Operators.bool_op` (an improved version of
:meth:`.Operators.op` for boolean operators).
Additional doc updates here apply to 1.4 so will backport these
out to a separate commit.
Mike Bayer [Fri, 20 May 2022 19:56:54 +0000 (15:56 -0400)]
render select froms first
The FROM clauses that are established on a :func:`_sql.select` construct
when using the :meth:`_sql.Select.select_from` method will now render first
in the FROM clause of the rendered SELECT, which serves to maintain the
ordering of clauses as was passed to the :meth:`_sql.Select.select_from`
method itself without being affected by the presence of those clauses also
being mentioned in other parts of the query. If other elements of the
:class:`_sql.Select` also generate FROM clauses, such as the columns clause
or WHERE clause, these will render after the clauses delivered by
:meth:`_sql.Select.select_from` assuming they were not explictly passed to
:meth:`_sql.Select.select_from` also. This improvement is useful in those
cases where a particular database generates a desirable query plan based on
a particular ordering of FROM clauses and allows full control over the
ordering of FROM clauses.
Mike Bayer [Sun, 22 May 2022 14:32:02 +0000 (10:32 -0400)]
disable psycopg-c from dbapimain builds
I don't know what this means.
ERROR:
Command errored out with exit status 1:
command: /home/classic/tmp/foo/.venv/bin/python /home/classic/tmp/foo/.venv/lib/python3.10/site-packages/pip/_vendor/pep517/in_process/_in_process.py get_requires_for_build_wheel /tmp/tmp1x3hi09h
cwd: /tmp/pip-install-yo6y28dm/psycopg-c_18eb2a0429dc438493454ed1051296ad
Complete output (14 lines):
error: Multiple top-level packages discovered in a flat-layout: ['psycopg', 'psycopg_c', 'psycopg_pool'].
To avoid accidental inclusion of unwanted files or directories,
setuptools will not proceed with this build.
If you are trying to create a single distribution with multiple packages
on purpose, you should not rely on automatic discovery.
Instead, consider the following options:
1. set up custom discovery (`find` directive with `include` or `exclude`)
2. use a `src-layout`
3. explicitly set `py_modules` or `packages` with a list of names
To find more information, look for "package discovery" on setuptools docs.
Mike Bayer [Fri, 18 Feb 2022 15:05:12 +0000 (10:05 -0500)]
implement dataclass_transforms
Implement a new means of creating a mapped dataclass where
instead of applying the `@dataclass` decorator distinctly,
the declarative process itself can create the dataclass.
MapperProperty and MappedColumn objects themselves take
the place of the dataclasses.Field object when constructing
the class.
The overall approach is made possible at the typing level
using pep-681 dataclass transforms [1].
This new approach should be able to completely supersede the
previous "dataclasses" approach of embedding metadata into
Field() objects, which remains a mutually exclusive declarative
setup style (mixing them introduces new issues that are not worth
solving).
valievkarim [Wed, 18 May 2022 20:24:41 +0000 (16:24 -0400)]
Include new MySQL error code 4031 for MySQL disconnect check
Added disconnect code for MySQL error 4031, introduced in MySQL >= 8.0.24,
indicating connection idle timeout exceeded. In particular this repairs an
issue where pre-ping could not reconnect on a timed-out connection. Pull
request courtesy valievkarim.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
