Mike Bayer [Thu, 6 May 2021 16:24:00 +0000 (12:24 -0400)]
don't cache TypeDecorator by default
The :class:`.TypeDecorator` class will now emit a warning when used in SQL
compilation with caching unless the ``.cache_ok`` flag is set to ``True``
or ``False``. ``.cache_ok`` indicates that all the parameters passed to the
object are safe to be used as a cache key, ``False`` means they are not.
Mike Bayer [Sun, 2 May 2021 22:31:03 +0000 (18:31 -0400)]
unify transactional context managers
Applied consistent behavior to the use case of
calling ``.commit()`` or ``.rollback()`` inside of an existing
``.begin()`` context manager, with the addition of potentially
emitting SQL within the block subsequent to the commit or rollback.
This change continues upon the change first added in
:ticket:`6155` where the use case of calling "rollback" inside of
a ``.begin()`` contextmanager block was proposed:
* calling ``.commit()`` or ``.rollback()`` will now be allowed
without error or warning within all scopes, including
that of legacy and future :class:`_engine.Engine`, ORM
:class:`_orm.Session`, asyncio :class:`.AsyncEngine`. Previously,
the :class:`_orm.Session` disallowed this.
* The remaining scope of the context manager is then closed;
when the block ends, a check is emitted to see if the transaction
was already ended, and if so the block returns without action.
* It will now raise **an error** if subsequent SQL of any kind
is emitted within the block, **after** ``.commit()`` or
``.rollback()`` is called. The block should be closed as
the state of the executable object would otherwise be undefined
in this state.
Mike Bayer [Wed, 5 May 2021 12:38:54 +0000 (08:38 -0400)]
Parenthesize for empty not in
Fixed regression caused by the "empty in" change just made in
:ticket:`6397` 1.4.12 where the expression needs to be parenthesized for
the "not in" use case, otherwise the condition will interfere with the
other filtering criteria.
also amends StrSQLCompiler to use the newer "empty IN" style for
its compilation process.
mypy: load extra test files from environment variable
Add --mypy-extra-test-path parameter to pytest execution to list extra
directories to load test files from.
This enables the stubs repo to load this plugin and specify it's
own test directory to run mypy tests. Supports both single file tests
and incremental tests based on patch files.
Mike Bayer [Tue, 4 May 2021 17:31:51 +0000 (13:31 -0400)]
Establish deprecation path for CursorResult.keys()
Established a deprecation path for calling upon the
:meth:`_cursor.CursorResult.keys` method for a statement that returns no
rows to provide support for legacy patterns used by the "records" package
as well as any other non-migrated applications. Previously, this would
raise :class:`.ResourceClosedException` unconditionally in the same way as
it does when attempting to fetch rows. While this is the correct behavior
going forward, the :class:`_cursor.LegacyCursorResult` object will now in
this case return an empty list for ``.keys()`` as it did in 1.3, while also
emitting a 2.0 deprecation warning. The :class:`_cursor.CursorResult`, used
when using a 2.0-style "future" engine, will continue to raise as it does
now.
Mike Bayer [Tue, 4 May 2021 15:19:00 +0000 (11:19 -0400)]
Restore detached object logic for dynamic, but warn
Fixed regression involving ``lazy='dynamic'`` loader in conjunction with a
detached object. The previous behavior was that the dynamic loader upon
calling methods like ``.all()`` returns empty lists for detached objects
without error, this has been restored; however a warning is now emitted as
this is not the correct result. Other dynamic loader scenarios correctly
raise ``DetachedInstanceError``.
Mike Bayer [Mon, 3 May 2021 13:55:12 +0000 (09:55 -0400)]
loader strategy regression fixes
Fixed regression where using :func:`_orm.selectinload` and
:func:`_orm.subqueryload` to load a two-level-deep path would lead to an
attribute error.
Fixed regression where using the :func:`_orm.noload` loader strategy in
conjunction with a "dynamic" relationship would lead to an attribute error
as the noload strategy would attempt to apply itself to the dynamic loader.
Restored a legacy transactional behavior that was inadvertently removed
from the :class:`_engine.Connection` as it was never tested as a known use
case in previous versions, where calling upon the
:meth:`_engine.Connection.begin_nested` method, when no transaction were
present, would not create a SAVEPOINT at all, and would instead only start
the outermost transaction alone, and return that :class:`.RootTransaction`
object, acting like the outermost transaction. Committing the transaction
object returned by :meth:`_engine.Connection.begin_nested` would therefore
emit a real COMMIT on the database connection.
This behavior is not at all what the 2.0 style connection will do - in 2.0
style, calling :meth:`_future.Connection.begin_nested` will "autobegin" the
outer transaction, and then as instructed emit a SAVEPOINT, returning the
:class:`.NestedTransaction` object. The outer transaction is committed by
calling upon :meth:`_future.Connection.commit`, as is "commit-as-you-go"
style usage.
In non-"future" mode, while the old behavior is restored, it also
emits a 2.0 deprecation warning as this is a legacy behavior.
Additionally clarifies and reformats various engine-related
documentation, in particular future connection.begin() which
was a tire fire.
Fixed a regression introduced by :ticket:`6337` that would create an
``asyncio.Lock`` which could be attached to the wrong loop when
instantiating the async engine before any asyncio loop was started, leading
to an asyncio error message when attempting to use the engine under certain
circumstances.
Mike Bayer [Sat, 1 May 2021 12:41:09 +0000 (08:41 -0400)]
Support filter_by() from columns, functions, Core SQL
Fixed regression where :meth:`_orm.Query.filter_by` would not work if the
lead entity were a SQL function or other expression derived from the
primary entity in question, rather than a simple entity or column of that
entity. Additionally, improved the behavior of
:meth:`_sql.Select.filter_by` overall to work with column expressions even
in a non-ORM context.
Mike Bayer [Fri, 30 Apr 2021 15:54:52 +0000 (11:54 -0400)]
track_on needs to be a fixed size, support sub-tuples
Fixed regression in ``selectinload`` loader strategy that would cause it to
cache its internal state incorrectly when handling relationships that join
across more than one column, such as when using a composite foreign key.
The invalid caching would then cause other loader operations to fail.
Mike Bayer [Thu, 29 Apr 2021 20:38:03 +0000 (16:38 -0400)]
Ensure iterable passed to Select is not a mapped class
Fixed regression caused by :ticket:`5395` where tuning back the check for
sequences in :func:`_sql.select` now caused failures when doing 2.0-style
querying with a mapped class that also happens to have an ``__iter__()``
method. Tuned the check some more to accommodate this as well as some other
interesting ``__iter__()`` scenarios.
Fixed an issue with the (deprecated in 1.4)
:meth:`_schema.ForeignKeyConstraint.copy` method that caused an error when
invoked with the ``schema`` argument.
Mike Bayer [Wed, 28 Apr 2021 22:31:51 +0000 (18:31 -0400)]
Use non-subquery form for empty IN
Revised the "EMPTY IN" expression to no longer rely upon using a subquery,
as this was causing some compatibility and performance problems. The new
approach for selected databases takes advantage of using a NULL-returning
IN expression combined with the usual "1 != 1" or "1 = 1" expression
appended by AND or OR. The expression is now the default for all backends
other than SQLite, which still had some compatibility issues regarding
tuple "IN" for older SQLite versions.
Third party dialects can still override how the "empty set" expression
renders by implementing a new compiler method
``def visit_empty_set_op_expr(self, type_, expand_op)``, which takes
precedence over the existing
``def visit_empty_set_expr(self, element_types)`` which remains in place.
Tony Locke [Thu, 29 Apr 2021 18:25:09 +0000 (14:25 -0400)]
Support stream_results in the pg8000 dialect
### Description
This change adds support for stream_results for the pg8000 dialect by adding a server side cursor. The server-side cursor is a wrapper around a standard DBAPI cursor, and uses the SQL-level cursors. This is being discussed in issue https://github.com/sqlalchemy/sqlalchemy/issues/6198 and this pull request is really to give a concrete example of what I was suggesting.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [x] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Thu, 29 Apr 2021 16:03:47 +0000 (12:03 -0400)]
fix mapper._primary_key_propkeys
Fixed issue in :meth:`_orm.Session.bulk_save` when used with persistent
objects which would fail to track the primary key of mappings where the
column name of the primary key were different than the attribute name.
Mike Bayer [Thu, 29 Apr 2021 14:40:00 +0000 (10:40 -0400)]
accommodate HasEntityNamespace in context.all_selected_columns
Fixed regression in hybrid_property where a hybrid against a SQL function
would generate an ``AttributeError`` when attempting to generate an entry
for the ``.c`` collection of a subquery in some cases; among other things
this would impact its use in cases like that of ``Query.count()``.
Mike Bayer [Thu, 29 Apr 2021 12:42:40 +0000 (08:42 -0400)]
clarify individual isolation level section
also rewrite the "individual sessions" section
to not rely upon context from the previous section,
this is a bad habit and should be fixed whereever we can
find it becuase people click on individual sections out of
context.
Propertly ignore ``Identity`` in MySQL and MariaDb.
Ensure that the MySQL and MariaDB dialect ignore the
:class:`_sql.Identity` construct while rendering the
``AUTO_INCREMENT`` keyword in a create table.
The Oracle and PostgreSQL compiler was updated to not render
:class:`_sql.Identity` if the database version does not support it
(Oracle < 12 and PostgreSQL < 10). Previously it was rendered regardless
of the database version.
Use SingletonThreadPool for in-memory SQLite
database created using filename uri
Default to using ``SingletonThreadPool`` for in-memory SQLite databases
created using URI filenames. Previously the default pool used was the
``NullPool`` that precented sharing the same database between multiple
engines.
Mike Bayer [Wed, 28 Apr 2021 16:14:33 +0000 (12:14 -0400)]
accommodate for mutiple copies of bind in ckbm
Fixed critical regression where bound parameter tracking as used in the SQL
caching system could fail to track all parameters for the case where the
same SQL expression containing a parameter were used in an ORM-related
query using a feature such as class inheritance, which was then embedded in
an enclosing expression which would make use of that same expression
multiple times, such as a UNION. The ORM would individually copy the
individual SELECT statements as part of compilation with class inheritance,
which then embedded in the enclosing statement would fail to accommodate
for all parameters. The logic that tracks this condition has been adjusted
to work for multiple copies of a parameter.
Mike Bayer [Wed, 28 Apr 2021 13:56:15 +0000 (09:56 -0400)]
add optional proxy_class to track w/ proxy_key
Fixed regression in ORM where using hybrid property to indicate an
expression from a different entity would confuse the column-labeling logic
in the ORM and attempt to derive the name of the hybrid from that other
class, leading to an attribute error. The owning class of the hybrid
attribute is now tracked along with the name.
Mike Bayer [Wed, 28 Apr 2021 00:44:46 +0000 (20:44 -0400)]
don't mutate the statement in ORM compile
Fixed issue where using a :class:`_sql.Select` as a subquery in an ORM
context would modify the :class:`_sql.Select` in place to disable
eagerloads on that object, which would then cause that same
:class:`_sql.Select` to not eagerload if it were then re-used in a
top-level execution context.
Mike Bayer [Tue, 27 Apr 2021 17:09:04 +0000 (13:09 -0400)]
have SchemaType inherit schema from metadata
Fixed very old issue where the :class:`_types.Enum` datatype would not
inherit the :paramref:`_schema.MetaData.schema` parameter of a
:class:`_schema.MetaData` object when that object were passed to the
:class:`_types.Enum` using :paramref:`_types.Enum.metadata`.
Mike Bayer [Mon, 26 Apr 2021 23:17:06 +0000 (19:17 -0400)]
Pass all datetime values to pyodbc for timezone-naive column
Fixed regression caused by :ticket:`6306` which added support for
``DateTime(timezone=True)``, where the previous behavior of the pyodbc
driver of implicitly dropping the tzinfo from a timezone-aware date when
INSERTing into a timezone-naive DATETIME column were lost, leading to a SQL
Server error when inserting timezone-aware datetime objects into
timezone-native database columns.
Mike Bayer [Mon, 26 Apr 2021 17:30:29 +0000 (13:30 -0400)]
Handle Sequence in _process_multiparam_default_bind
Fixed issue where usage of an explicit :class:`.Sequence` would produce
inconsistent "inline" behavior for an :class:`.Insert` construct that
includes multiple values phrases; the first seq would be inline but
subsequent ones would be "pre-execute", leading to inconsistent sequence
ordering. The sequence expressions are now fully inline.
Mike Bayer [Mon, 26 Apr 2021 13:50:50 +0000 (09:50 -0400)]
Ensure autobegin occurs for attribute changes; Document autobegin
The Session autobegin feature was not anticipated as having
any behavioral changes other than the event hook being called
at a different time, however as autobegin impacts the behavior
of the commit() and rollback() methods in that they can now
be no-ops in non-autocommit mode, document the behavior fully.
Fixed issue where the new :ref:`session_autobegin` behavior failed to
"autobegin" in the case where an existing persistent object has an
attribute change, which would then impact the behavior of
:meth:`_orm.Session.rollback` in that no snapshot was created to be rolled
back. The "attribute modify" mechanics have been updated to ensure
"autobegin", which does not perform any database work, does occur when
persistent attributes change in the same manner as when
:meth:`_orm.Session.add` is called. This is a regression as in 1.3, the
rollback() method always had a transaction to roll back and would expire
every time.
Mike Bayer [Fri, 23 Apr 2021 01:45:10 +0000 (21:45 -0400)]
implement declared_attr superclass assignment check for dataclasses
Adjusted the declarative scan for dataclasses so that the inheritance
behavior of :func:`_orm.declared_attr` established on a mixin, when using
the new form of having it inside of a ``dataclasses.field()`` construct and
not actually a descriptor attribute on the class, correctly accommodates
the case when the target class to be mapped is a subclass of an existing
mapped class which has already mapped that :func:`_orm.declared_attr`, and
therefore should not be re-applied to this class.
Also, as changed in ed3f2c617239668d we now have an "is_dataclass"
boolean set as we iterate through attrs so we can remove this
from declared_attr.
Mike Bayer [Fri, 23 Apr 2021 14:19:39 +0000 (10:19 -0400)]
dont assume ClauseElement in attributes, coercions
Fixed two distinct issues, each of which would come into play under certain
circumstances, most likely however one which is a common mis-configuration
in :class:`_hybrid.hybrid_property`, where the "expression" implementation
would return a non :class:`_sql.ClauseElement` such as a boolean value.
For both issues, 1.3's behavior was to silently ignore the
mis-configuration and ultimately attempt to interpret the value as a
SQL expression, which would lead to an incorrect query.
* Fixed issue regarding interaction of the attribute system with
hybrid_property, where if the ``__clause_element__()`` method of the
attribute returned a non-:class:`_sql.ClauseElement` object, an internal
``AttributeError`` would lead the attribute to return the ``expression``
function on the hybrid_property itself, as the attribute error was
against the name ``.expression`` which would invoke the ``__getattr__()``
method as a fallback. This now raises explicitly. In 1.3 the
non-:class:`_sql.ClauseElement` was returned directly.
* Fixed issue in SQL argument coercions system where passing the wrong
kind of object to methods that expect column expressions would fail if
the object were altogether not a SQLAlchemy object, such as a Python
function, in cases where the object were not just coerced into a bound
value. Again 1.3 did not have a comprehensive argument coercion system
so this case would also pass silently.
Mike Bayer [Thu, 22 Apr 2021 20:05:16 +0000 (16:05 -0400)]
Break up data.rst; add unions, literal_column
This breaks data.rst into three separate sub-sections,
as SELECT is getting very long. It then adds sections
on select() + text/literal_column as well as unions
and set operations, and also tries to improve the
ORDER BY section a bit.
Mike Bayer [Thu, 22 Apr 2021 14:45:01 +0000 (10:45 -0400)]
omit text from selected_columns; clear memoizations
Fixed regression where usage of the :func:`_sql.text` construct inside the
columns clause of a :class:`_sql.Select` construct, which is better handled
by using a :func:`_sql.literal_column` construct, would nonetheless prevent
constructs like :func:`_sql.union` from working correctly. Other use cases,
such as constructing subuqeries, continue to work the same as in prior
versions where the :func:`_sql.text` construct is silently omitted from the
collection of exported columns. Also repairs similar use within the
ORM.
This adds a new internal method _all_selected_columns. The existing
"selected_columns" collection can't store a TextClause and this never
worked, so they are omitted. The TextClause is also not "exported",
i.e. available for SELECT from a subquery, as was already the case
in 1.3, so the "exported_columns" and "exported_columns_iterator"
accessors are where we now omit TextClause.
Fixed regression involving legacy methods such as
:meth:`_sql.Select.append_column` where internal assertions would fail.
Mike Bayer [Thu, 22 Apr 2021 18:33:36 +0000 (14:33 -0400)]
Ensure sparse backend selection sorts on driver also
Because we have --dbdriver now, we usually have
lots of db drivers queued up which affect the selection
of backend tests. the sort in possible_configs_for_cls()
is used so that when we are only picking some backends
due to the sparse flag, we get the same answer each time
otherwise pytest-xdist raises an error if two workers
have different tests.
adding the sort for driver fixes an issue on jenkins
master build where the memusage test run frequently failed
due to mismatched test collection.
Mike Bayer [Wed, 21 Apr 2021 18:44:45 +0000 (14:44 -0400)]
Add new "sync once" mode for pool.connect
Fixed critical regression caused by the change in :ticket`5497` where the
connection pool "init" phase no longer occurred within mutexed isolation,
allowing other threads to proceed with the dialect uninitialized, which
could then impact the compilation of SQL statements.
This issue is essentially the same regression which was fixed many years
ago in :ticket:`2964` in dd32540dabbee0678530fb1b0868d1eb41572dca,
which was missed this time as the test suite fo
that issue only tested the pool in isolation, and assumed the
"first_connect" event would be used by the Engine. However
:ticket:`5497` stopped using "first_connect" and no test detected
the lack of mutexing, that has been resolved here through
the addition of more tests.
This fix also identifies what is probably a bug in earlier versions
of SQLAlchemy where the "first_connect" handler would be cancelled
if the initializer failed; this is evidenced by
test_explode_in_initializer which was doing a reconnect due to
c.rollback() yet wasn't hanging. We now solve this issue by
preventing the manufactured Connection from ever reconnecting
inside the first_connect handler.
Also remove the "_sqla_unwrap" test attribute; this is almost
not used anymore however we can use a more targeted
wrapper supplied by the testing.engines.proxying_engine
function.
See if we can also open up Oracle for "ad hoc engines" tests
now that we have better connection management logic.
Mike Bayer [Wed, 21 Apr 2021 14:39:09 +0000 (10:39 -0400)]
Limit dc field logic to only fields that are definitely dc
Fixed regression where recent changes to support Python dataclasses had the
inadvertent effect that an ORM mapped class could not successfully override
the ``__new__()`` method.
In this case the "__new__" method comes out as staticmethod in
cls.__dict__ vs. a function in the metaclass dict_, so comparing
using identity fails. I was hoping not to have too much
"dataclass" hardcoded, the logic here if it were generalized
to other attribute declaration systems there
would still have a flag that indicates an attribute is part of
the "special declaration system".
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
