Mike Bayer [Thu, 5 Jan 2023 15:34:37 +0000 (10:34 -0500)]
warn and skip for FKs that refer to invisible cols for Oracle
Supported use case for foreign key constraints where the local column is
marked as "invisible". The errors normally generated when a
:class:`.ForeignKeyConstraint` is created that check for the target column
are disabled when reflecting, and the constraint is skipped with a warning
in the same way which already occurs for an :class:`.Index` with a similar
issue.
tests are added for indexes, unique constraints, and primary key
constraints, which were already working; indexes and uniques warn,
primary keys don't which we would assume is because we never see those
PK columns in the first place.
Constraints now raise an informative ConstraintColumnNotFoundError
in the general case for strings in the "pending colargs" collection
not being resolvable.
Mike Bayer [Mon, 9 Jan 2023 13:53:57 +0000 (08:53 -0500)]
accept TableClause through mapped selectable chain
type annotation somehow decided that TableClause doesn't have
primary key fields which is not the case at all. In particular
the "views" recipe relies on TableClause so adding a restriction
like this does not make any sense.
It seems the issue was to open this up for typing, by allowing
TableClause out as far as ddl.sort_tables() typing is passing
for now. Support it out in get_bind() etc.
Mike Bayer [Thu, 5 Jan 2023 04:32:23 +0000 (23:32 -0500)]
revert MySQL to use DESCRIBE for has_table()
Restored the behavior of :meth:`.Inspector.has_table` to report on
temporary tables for MySQL / MariaDB. This is currently the behavior for
all other included dialects, but was removed for MySQL in 1.4 due to no
longer using the DESCRIBE command; there was no documented support for temp
tables being reported by the :meth:`.Inspector.has_table` method in this
version or on any previous version, so the previous behavior was undefined.
As SQLAlchemy 2.0 has added formal support for temp table status via
:meth:`.Inspector.has_table`, the MySQL /MariaDB dialect has been reverted
to use the "DESCRIBE" statement as it did in the SQLAlchemy 1.3 series and
previously, and test support is added to include MySQL / MariaDB for
this behavior. The previous issues with ROLLBACK being emitted which
1.4 sought to improve upon don't apply in SQLAlchemy 2.0 due to
simplifications in how :class:`.Connection` handles transactions.
DESCRIBE is necessary as MariaDB in particular has no consistently
available public information schema of any kind in order to report on temp
tables other than DESCRIBE/SHOW COLUMNS, which rely on throwing an error
in order to report no results.
Michael Gorven [Wed, 4 Jan 2023 17:30:42 +0000 (12:30 -0500)]
[asyncpg] Extract rowcount for SELECT statements
Added support to the asyncpg dialect to return the ``cursor.rowcount``
value for SELECT statements when available. While this is not a typical use
for ``cursor.rowcount``, the other PostgreSQL dialects generally provide
this value. Pull request courtesy Michael Gorven.
Mike Bayer [Tue, 3 Jan 2023 23:21:25 +0000 (18:21 -0500)]
tutorial updates re: Core /ORM commonality
updates for Insert / bulk insert, executemanyvalues,
as well as beginning to describe Table / declared class more
closely together, mentioning typing support.
Fixed a long-standing issue where sphinx would complain about
the Insert symbol being ambiguous.
Gleb Kisenkov [Wed, 28 Dec 2022 19:23:23 +0000 (14:23 -0500)]
Type annotations for sqlalchemy.orm.events
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
An attempt to annotate `lib/sqlalchemy/orm/events.py` with type hints (issue #6810).
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [x] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Wed, 28 Dec 2022 17:04:07 +0000 (12:04 -0500)]
ensure whereclause, returning copied as tuples
Fixed issue in the internal SQL traversal for DML statements like
:class:`_dml.Update` and :class:`_dml.Delete` which would cause among other
potential issues, a specific issue using lambda statements with the ORM
update/delete feature.
Mike Bayer [Tue, 20 Dec 2022 17:48:08 +0000 (12:48 -0500)]
establish explicit join transaction modes
The behavior of "joining an external transaction into a Session" has been
revised and improved, allowing explicit control over how the
:class:`_orm.Session` will accommodate an incoming
:class:`_engine.Connection` that already has a transaction and possibly a
savepoint already established. The new parameter
:paramref:`_orm.Session.join_transaction_mode` includes a series of option
values which can accommodate the existing transaction in several ways, most
importantly allowing a :class:`_orm.Session` to operate in a fully
transactional style using savepoints exclusively, while leaving the
externally initiated transaction non-committed and active under all
circumstances, allowing test suites to rollback all changes that take place
within tests.
Additionally, revised the :meth:`_orm.Session.close` method to fully close
out savepoints that may still be present, which also allows the
"external transaction" recipe to proceed without warnings if the
:class:`_orm.Session` did not explicitly end its own SAVEPOINT
transactions.
Mike Bayer [Tue, 27 Dec 2022 17:29:38 +0000 (12:29 -0500)]
pass more contextual information to PyWrapper param create
Fixed issue in lambda SQL feature where the calculated type of a literal
value would not take into account the type coercion rules of the "compared
to type", leading to a lack of typing information for SQL expressions, such
as comparisons to :class:`.JSON` elements and similar.
Mike Bayer [Sun, 18 Dec 2022 21:33:22 +0000 (16:33 -0500)]
reorganize pre_session_exec around do_orm_execute
Allow do_orm_execute() events to both receive the complete
state of bind_argments, load_options, update_delete_options
as they do already, but also allow them to *change* all those
things via new execution options. Options like autoflush,
populate_existing etc. can now be updated within a
do_orm_execute() hook and those changes will take effect
all the way through.
Took a few tries to get something that covers every case here,
in particular horizontal sharding which is consuming those
options as well as using context.invoke(), without excess
complexity. The good news seems to be that a simple
reorg and replacing the "reentrant" boolean with
"is this before do_orm_execute is invoked" was all that was
needed.
As part of this we add a new "identity_token" option allowing
this option to be controlled from do_orm_execute() as well
as from the outside.
Mike Bayer [Fri, 23 Dec 2022 14:09:44 +0000 (09:09 -0500)]
improve async_sessionmaker docs
* illustrate patterns where the async_sessionmaker is being reused.
if you are not reusing it, there is no point to making it
* fix reference to async_sessionmaker
* add typing
Mike Bayer [Thu, 22 Dec 2022 23:14:31 +0000 (18:14 -0500)]
expand out Index if passed to "constraint"
Fixed bug where the PostgreSQL
:paramref:`_postgresql.OnConflictClause.constraint` parameter would accept
an :class:`.Index` object, however would not expand this index out into its
individual index expressions, instead rendering its name in an ON CONFLICT
ON CONSTRAINT clause, which is not accepted by PostgreSQL; the "constraint
name" form only accepts unique or exclude constraint names. The parameter
continues to accept the index but now expands it out into its component
expressions for the render.
Mike Bayer [Wed, 21 Dec 2022 21:10:34 +0000 (16:10 -0500)]
check for adapt to same class in AbstractRange
Fixed regression where newly revised PostgreSQL range types such as
:class:`_postgresql.INT4RANGE` could not be set up as the impl of a
:class:`.TypeDecorator` custom type, instead raising a ``TypeError``.
Mike Bayer [Mon, 19 Dec 2022 20:15:35 +0000 (15:15 -0500)]
add joins_implicitly to column_valued()
Added parameter
:paramref:`.FunctionElement.column_valued.joins_implicitly`, which is
useful in preventing the "cartesian product" warning when making use of
table-valued or column-valued functions. This parameter was already
introduced for :meth:`.FunctionElement.table_valued` in :ticket:`7845`,
however it failed to be added for :meth:`.FunctionElement.column_valued`
as well.
Mike Bayer [Sun, 18 Dec 2022 20:35:39 +0000 (15:35 -0500)]
remove __allow_unmapped__ requirement from dataclasses
Removed the requirement that the ``__allow_unmapped__`` attribute be used
on Declarative Dataclass Mapped class when non-``Mapped[]`` annotations are
detected; previously, an error message that was intended to support legacy
ORM typed mappings would be raised, which additionally did not mention
correct patterns to use with Dataclasses specifically. This error message
is now no longer raised if :meth:`_orm.registry.mapped_as_dataclass` or
:class:`_orm.MappedAsDataclass` is used.
Mike Bayer [Mon, 19 Dec 2022 13:34:51 +0000 (08:34 -0500)]
add exclusion for unusual chars in column names
Added new exclusion rule for third party dialects called
``unusual_column_name_characters``, which can be "closed" for third party
dialects that don't support column names with unusual characters such as
dots, slashes, or percent signs in them, even if the name is properly
quoted.
Mike Bayer [Fri, 16 Dec 2022 20:44:32 +0000 (15:44 -0500)]
include pk cols in refresh() if relationships are requested
A series of changes and improvements regarding
:meth:`_orm.Session.refresh`. The overall change is that primary key
attributes for an object are now included in a refresh operation
unconditionally when relationship-bound attributes are to be refreshed,
even if not expired and even if not specified in the refresh.
* Improved :meth:`_orm.Session.refresh` so that if autoflush is enabled
(as is the default for :class:`_orm.Session`), the autoflush takes place
at an earlier part of the refresh process so that pending primary key
changes are applied without errors being raised. Previously, this
autoflush took place too late in the process and the SELECT statement
would not use the correct key to locate the row and an
:class:`.InvalidRequestError` would be raised.
* When the above condition is present, that is, unflushed primary key
changes are present on the object, but autoflush is not enabled,
the refresh() method now explicitly disallows the operation to proceed,
and an informative :class:`.InvalidRequestError` is raised asking that
the pending primary key changes be flushed first. Previously,
this use case was simply broken and :class:`.InvalidRequestError`
would be raised anyway. This restriction is so that it's safe for the
primary key attributes to be refreshed, as is necessary for the case of
being able to refresh the object with relationship-bound secondary
eagerloaders also being emitted. This rule applies in all cases to keep
API behavior consistent regardless of whether or not the PK cols are
actually needed in the refresh, as it is unusual to be refreshing
some attributes on an object while keeping other attributes "pending"
in any case.
* The :meth:`_orm.Session.refresh` method has been enhanced such that
attributes which are :func:`_orm.relationship`-bound and linked to an
eager loader, either at mapping time or via last-used loader options,
will be refreshed in all cases even when a list of attributes is passed
that does not include any columns on the parent row. This builds upon the
feature first implemented for non-column attributes as part of
:ticket:`1763` fixed in 1.4 allowing eagerly-loaded relationship-bound
attributes to participate in the :meth:`_orm.Session.refresh` operation.
If the refresh operation does not indicate any columns on the parent row
to be refreshed, the primary key columns will nonetheless be included
in the refresh operation, which allows the load to proceed into the
secondary relationship loaders indicated as it does normally.
Previously an :class:`.InvalidRequestError` error would be raised
for this condition (:ticket:`8703`)
* Fixed issue where an unnecessary additional SELECT would be emitted in
the case where :meth:`_orm.Session.refresh` were called with a
combination of expired attributes, as well as an eager loader such as
:func:`_orm.selectinload` that emits a "secondary" query, if the primary
key attributes were also in an expired state. As the primary key
attributes are now included in the refresh automatically, there is no
additional load for these attributes when a relationship loader
goes to select for them (:ticket:`8997`)
* Fixed regression caused by :ticket:`8126` released in 2.0.0b1 where the
:meth:`_orm.Session.refresh` method would fail with an
``AttributeError``, if passed both an expired column name as well as the
name of a relationship-bound attribute that was linked to a "secondary"
eagerloader such as the :func:`_orm.selectinload` eager loader
(:ticket:`8996`)
Mike Bayer [Fri, 16 Dec 2022 19:05:48 +0000 (14:05 -0500)]
dont call platform.architecture()
Fixed regression where the base compat module was calling upon
``platform.architecture()`` in order to detect some system properties,
which results in an over-broad system call against the system-level
``file`` call that is unavailable under some circumstances, including
within some secure environment configurations.
Mike Bayer [Fri, 16 Dec 2022 17:16:21 +0000 (12:16 -0500)]
ensure all visit methods accept **kw
Added test support to ensure that all compiler ``visit_xyz()`` methods
across all :class:`.Compiler` implementations in SQLAlchemy accept a
``**kw`` parameter, so that all compilers accept additional keyword
arguments under all circumstances.
Mike Bayer [Fri, 16 Dec 2022 17:56:21 +0000 (12:56 -0500)]
make bind escape lookup extensible
To accommodate for third party dialects with different character escaping
needs regarding bound parameters, the system by which SQLAlchemy "escapes"
(i.e., replaces with another character in its place) special characters in
bound parameter names has been made extensible for third party dialects,
using the :attr:`.SQLCompiler.bindname_escape_chars` dictionary which can
be overridden at the class declaration level on any :class:`.SQLCompiler`
subclass. As part of this change, also added the dot ``"."`` as a default
"escaped" character.
Mike Bayer [Fri, 9 Dec 2022 00:31:37 +0000 (19:31 -0500)]
add eager_defaults="auto" for inserts
Added a new default value for the :paramref:`.Mapper.eager_defaults`
parameter "auto", which will automatically fetch table default values
during a unit of work flush, if the dialect supports RETURNING for the
INSERT being run, as well as
:ref:`insertmanyvalues <engine_insertmanyvalues>` available. Eager fetches
for server-side UPDATE defaults, which are very uncommon, continue to only
take place if :paramref:`.Mapper.eager_defaults` is set to ``True``, as
there is no batch-RETURNING form for UPDATE statements.
Mike Bayer [Wed, 14 Dec 2022 01:07:14 +0000 (20:07 -0500)]
add explicit REGCONFIG, pg full text functions
Added support for explicit use of PG full text functions with asyncpg and
psycopg (SQLAlchemy 2.0 only), with regards to the ``REGCONFIG`` type cast
for the first argument, which previously would be incorrectly cast to a
VARCHAR, causing failures on these dialects that rely upon explicit type
casts. This includes support for :class:`_postgresql.to_tsvector`,
:class:`_postgresql.to_tsquery`, :class:`_postgresql.plainto_tsquery`,
:class:`_postgresql.phraseto_tsquery`,
:class:`_postgresql.websearch_to_tsquery`,
:class:`_postgresql.ts_headline`, each of which will determine based on
number of arguments passed if the first string argument should be
interpreted as a PostgreSQL "REGCONFIG" value; if so, the argument is typed
using a newly added type object :class:`_postgresql.REGCONFIG` which is
then explicitly cast in the SQL expression.
Mike Bayer [Thu, 15 Dec 2022 15:22:36 +0000 (10:22 -0500)]
implement literal_binds with expanding + bind_expression
Fixed bug where SQL compilation would fail to make use of
:meth:`_types.TypeEngine.bind_expression` on a given type when used in the
context of an "expanding" (i.e. "IN") parameter with the ``literal_binds``
compiler parameter in use.
Mike Bayer [Wed, 14 Dec 2022 18:49:11 +0000 (13:49 -0500)]
warn when backref will replace existing userland descriptor
A warning is emitted if a backref name used in :func:`_orm.relationship`
names an attribute on the target class which already has a method or
attribute assigned to that name, as the backref declaration will replace
that attribute.
Mike Bayer [Mon, 12 Dec 2022 23:05:07 +0000 (18:05 -0500)]
check index_list pragma for number of columns returned
Fixed regression caused by new support for reflection of partial indexes on
SQLite added in 1.4.45 for :ticket:`8804`, where the ``index_list`` pragma
command in very old versions of SQLite (possibly prior to 3.8.9) does not
return the current expected number of columns, leading to exceptions raised
when reflecting tables and indexes.
Mike Bayer [Mon, 12 Dec 2022 18:47:27 +0000 (13:47 -0500)]
catch all BaseException in pool and revert failed checkouts
Fixed a long-standing race condition in the connection pool which could
occur under eventlet/gevent monkeypatching schemes in conjunction with the
use of eventlet/gevent ``Timeout`` conditions, where a connection pool
checkout that's interrupted due to the timeout would fail to clean up the
failed state, causing the underlying connection record and sometimes the
database connection itself to "leak", leaving the pool in an invalid state
with unreachable entries. This issue was first identified and fixed in
SQLAlchemy 1.2 for :ticket:`4225`, however the failure modes detected in
that fix failed to accommodate for ``BaseException``, rather than
``Exception``, which prevented eventlet/gevent ``Timeout`` from being
caught. In addition, a block within initial pool connect has also been
identified and hardened with a ``BaseException`` -> "clean failed connect"
block to accommodate for the same condition in this location.
Big thanks to Github user @niklaus for their tenacious efforts in
identifying and describing this intricate issue.
Mike Bayer [Sun, 11 Dec 2022 16:13:58 +0000 (11:13 -0500)]
adjust for tox changes to passenv
Fixed issue in tox.ini file where changes in the tox 4.0 series to the
format of "passenv" caused tox to not function correctly, in particular
raising an error as of tox 4.0.6.
Mike Bayer [Fri, 9 Dec 2022 20:56:15 +0000 (15:56 -0500)]
look out for extras=None in freeze
Fixed issue where :meth:`_engine.Result.freeze` method would not work for
textual SQL using either :func:`_sql.text` or
:meth:`_engine.Connection.exec_driver_sql`.
Better syncronize async result docs with plain ones.
Removed non-functional method ``merge`` from :class:`_asyncio.AsyncResult`.
This method was non-functional and non-testes since the first introduction
of asyncio in SQLAlchemy.
Mike Bayer [Thu, 8 Dec 2022 16:00:31 +0000 (11:00 -0500)]
fix construct_params() for render_postcompile; add new API
The :meth:`.SQLCompiler.construct_params` method, as well as the
:attr:`.SQLCompiler.params` accessor, will now return the
exact parameters that correspond to a compiled statement that used
the ``render_postcompile`` parameter to compile. Previously,
the method returned a parameter structure that by itself didn't correspond
to either the original parameters or the expanded ones.
Passing a new dictionary of parameters to
:meth:`.SQLCompiler.construct_params` for a :class:`.SQLCompiler` that was
constructed with ``render_postcompile`` is now disallowed; instead, to make
a new SQL string and parameter set for an alternate set of parameters, a
new method :meth:`.SQLCompiler.construct_expanded_state` is added which
will produce a new expanded form for the given parameter set, using the
:class:`.ExpandedState` container which includes a new SQL statement
and new parameter dictionary, as well as a positional parameter tuple.
Mike Bayer [Mon, 5 Dec 2022 21:11:59 +0000 (16:11 -0500)]
disable polymorphic adaption in most cases
Improved a fix first made in version 1.4 for :ticket:`8456` which scaled
back the usage of internal "polymorphic adapters", that are used to render
ORM queries when the :paramref:`_orm.Mapper.with_polymorphic` parameter is
used. These adapters, which are very complex and error prone, are now used
only in those cases where an explicit user-supplied subquery is used for
:paramref:`_orm.Mapper.with_polymorphic`, which includes only the use case
of concrete inheritance mappings that use the
:func:`_orm.polymorphic_union` helper, as well as the legacy use case of
using an aliased subquery for joined inheritance mappings, which is not
needed in modern use.
For the most common case of joined inheritance mappings that use the
built-in polymorphic loading scheme, which includes those which make use of
the :paramref:`_orm.Mapper.polymorphic_load` parameter set to ``inline``,
polymorphic adapters are now no longer used. This has both a positive
performance impact on the construction of queries as well as a
substantial simplification of the internal query rendering process.
The specific issue targeted was to allow a :func:`_orm.column_property`
to refer to joined-inheritance classes within a scalar subquery, which now
works as intuitively as is feasible.
ORM context, mapper, strategies now use ORMAdapter in all cases
instead of straight ColumnAdapter; added some more parameters
to ORMAdapter to make this possible. ORMAdapter now includes a
"trace" enumeration that identifies the use path for the
adapter and can aid in debugging.
implement __slots__ for the ExternalTraversal hierarchy up
to ORMAdapter. Within this change, we have to change the
ClauseAdapter.wrap() method, which is only used in one polymorphic
codepath, to use copy.copy() instead of
`__dict__` access (apparently `__reduce_ex__` is implemented for
objects with `__slots__`), and we also remove pickling ability,
which should not be needed for adapters (this might have been needed
for 1.3 and earlier in order for Query to be picklable, but none
of that state is present within Query / select() / etc. anymore).
Mike Bayer [Wed, 7 Dec 2022 20:54:59 +0000 (15:54 -0500)]
Oracle COLUMN_VALUE is a column name, not a keyword
Fixed issue in Oracle compiler where the syntax for
:meth:`.FunctionElement.column_valued` was incorrect, rendering the name
``COLUMN_VALUE`` without qualifying the source table correctly.
Mike Bayer [Wed, 7 Dec 2022 19:23:45 +0000 (14:23 -0500)]
move more Session weakref tests to test_memusage
py311 may be more sensitive here, or maybe the machines are
acting differently these days, in any case move
memory / GC sensitive tests to test_memusage.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
