Mike Bayer [Tue, 4 Jan 2022 19:08:10 +0000 (14:08 -0500)]
improve custom operator for SQL types docs
introduce here that "custom ops" first come from
the .op() method in the usual case. then only if one wants
such an op to be pre-assocaited with particular types,
then the comparator may be used. Also clarify the individual
points regarding the comparator.
Mike Bayer [Sat, 1 Jan 2022 21:50:41 +0000 (16:50 -0500)]
adjust TypeError message yet again
in 924d38e660981ae2a7d55355603 we got it almost right in
all cases, Pypy seems to have an entirely different kind of message
so make this much more open ended.
Mike Bayer [Sat, 1 Jan 2022 20:30:15 +0000 (15:30 -0500)]
adjust error message assertion for TypeError
in eee7a3add99df2865e6d907f2 I added a few TypeError
tests, and some python interpreters produce a message like
"join() got an unexpected...", others seem to produce
"Query.join() got an unexpected..."
the pattern is strange, all Python 3.10's seem to return the
class name. For Python 3.8 and 3.9, my local intepreters
are doing the Python 3.10 behavior but github actions are doing
the older behavior. Python 3.7 both here and on github actions
are doing the older behavior.
Federico Caselli [Wed, 22 Dec 2021 20:45:45 +0000 (21:45 +0100)]
Properly type _generative, decorator, public_factory
Good new is that pylance likes it and copies over the
singature and everything.
Bad news is that mypy does not support this yet https://github.com/python/mypy/issues/8645
Other minor bad news is that non_generative is not typed. I've tried using a protocol
like the one in the comment but the signature is not ported over by pylance, so it's
probably best to just live without it to have the correct signature.
notes from mike: these three decorators are at the core of getting
the library to be typed, more good news is that pylance will
do all the things we like re: public_factory, see
https://github.com/microsoft/pyright/issues/2758#issuecomment-1002788656
.
For @_generative, we will likely move to using pep 673 once mypy
supports it which may be soon. but overall having the explicit
"return self" in the methods, while a little inconvenient, makes
the typing more straightforward and locally present in the files
rather than being decided at a distance. having "return self"
present, or not, both have problems, so maybe we will be able
to change it again if things change as far as decorator support.
As it is, I feel like we are barely squeaking by with our decorators,
the typing is already pretty out there.
Mike Bayer [Sat, 23 Oct 2021 15:26:45 +0000 (11:26 -0400)]
replace Variant with direct feature inside of TypeEngine
The :meth:`_sqltypes.TypeEngine.with_variant` method now returns a copy of
the original :class:`_sqltypes.TypeEngine` object, rather than wrapping it
inside the ``Variant`` class, which is effectively removed (the import
symbol remains for backwards compatibility with code that may be testing
for this symbol). While the previous approach maintained in-Python
behaviors, maintaining the original type allows for clearer type checking
and debugging.
Gord Thompson [Mon, 20 Dec 2021 21:37:13 +0000 (14:37 -0700)]
Reflect included columns as dialect_options
Fixed reflection of covering indexes to report ``include_columns`` as part
of the ``dialect_options`` entry in the reflected index dictionary, thereby
enabling round trips from reflection->create to be complete. Included
columns continue to also be present under the ``include_columns`` key for
backwards compatibility.
Mike Bayer [Thu, 2 Dec 2021 14:18:11 +0000 (09:18 -0500)]
propose concurrency check for SessionTransaction
the discussion at #7387 refers to a condition that seems
to happen in the wild also, such as [1] [2] [3], it's not
entirely clear why this specific spot is how this occurs,
however it's maybe that when the connection is being acquired
from the pool, under load there might be a wait on the connection
pool, leading to more time for another errant thread to be
calling .close(), just a theory.
in this patch we propose using decorators and context managers
along with declarative state declarations to block reentrant
or concurrent calls to methods that conflict with expected
state changes.
The :class:`_orm.Session` (and by extension :class:`.AsyncSession`) now has
new state-tracking functionality that will proactively trap any unexpected
state changes which occur as a particular transactional method proceeds.
This is to allow situations where the :class:`_orm.Session` is being used
in a thread-unsafe manner, where event hooks or similar may be calling
unexpected methods within operations, as well as potentially under other
concurrency situations such as asyncio or gevent to raise an informative
message when the illegal access first occurs, rather than passing silently
leading to secondary failures due to the :class:`_orm.Session` being in an
invalid state.
Mike Bayer [Thu, 9 Dec 2021 01:27:16 +0000 (20:27 -0500)]
factor out UnboundLoad and rearchitect strategy_options.py
The architecture of Load is mostly rewritten here.
The change includes removal of the "pluggable" aspect
of the loader options, which would patch new methods onto
Load. This has been replaced by normal methods that
respond normally to typing annotations. As part of this
change, the bake_loaders() and unbake_loaders() options,
which have no effect since 1.4 and were unlikely to be
in any common use, have been removed.
Additionally, to support annotations for methods that
make use of @decorator, @generative etc., modified
format_argspec_plus to no longer return "args", instead
returns "grouped_args" which is always grouped and
allows return annotations to format correctly.
Mike Bayer [Wed, 8 Dec 2021 13:57:44 +0000 (08:57 -0500)]
consider truediv as truediv; support floordiv operator
Implemented full support for "truediv" and "floordiv" using the
"/" and "//" operators. A "truediv" operation between two expressions
using :class:`_types.Integer` now considers the result to be
:class:`_types.Numeric`, and the dialect-level compilation will cast
the right operand to a numeric type on a dialect-specific basis to ensure
truediv is achieved. For floordiv, conversion is also added for those
databases that don't already do floordiv by default (MySQL, Oracle) and
the ``FLOOR()`` function is rendered in this case, as well as for
cases where the right operand is not an integer (needed for PostgreSQL,
others).
The change resolves issues both with inconsistent behavior of the
division operator on different backends and also fixes an issue where
integer division on Oracle would fail to be able to fetch a result due
to inappropriate outputtypehandlers.
Mike Bayer [Sun, 26 Dec 2021 17:13:19 +0000 (12:13 -0500)]
include empty intermediary tables in optimized get
Fixed issue in joined-inheritance load of additional attributes
functionality in deep multi-level inheritance where an intermediary table
that contained no columns would not be included in the tables joined,
instead linking those tables to their primary key identifiers. While this
works fine, it nonetheless in 1.4 began producing the cartesian product
compiler warning. The logic has been changed so that these intermediary
tables are included regardless. While this does include additional tables
in the query that are not technically necessary, this only occurs for the
highly unusual case of deep 3+ level inheritance with intermediary tables
that have no non primary key columns, potential performance impact is
therefore expected to be negligible.
Mike Bayer [Sun, 26 Dec 2021 16:25:00 +0000 (11:25 -0500)]
restore graceful degrade of subqueryload w from_statement
Fixed regression from 1.3 where the "subqueryload" loader strategy would
fail with a stack trace if used against a query that made use of
:meth:`_orm.Query.from_statement` or :meth:`_sql.Select.from_statement`. As
subqueryload requires modifying the original statement, it's not compatible
with the "from_statement" use case, especially for statements made against
the :func:`_sql.text` construct. The behavior now is equivalent to that of
1.3 and previously, which is that the loader strategy silently degrades to
not be used for such statements, typically falling back to using the
lazyload strategy.
Kai Mueller [Tue, 21 Dec 2021 20:00:30 +0000 (15:00 -0500)]
Fix missing class attributes when using __class_getitem__
Fixed issue where the ``__class_getitem__()`` method of the generated
declarative base class by :func:`_orm.as_declarative` would lead to
inaccessible class attributes such as ``__table__``, for cases where a
``Generic[T]`` style typing declaration were used in the class hierarchy.
This is in continuation from the basic addition of ``__class_getitem__()``
in :ticket:`7368`. Pull request courtesy Kai Mueller.
Mike Bayer [Wed, 22 Dec 2021 21:21:33 +0000 (16:21 -0500)]
use fully qualified, locatable names for all use of api.named_type()
Fixed mypy regression where the release of mypy 0.930 added additional
internal checks to the format of "named types", requiring that they be
fully qualified and locatable. This broke the mypy plugin for SQLAlchemy,
raising an assertion error, as there was use of symbols such as
``__builtins__`` and other un-locatable or unqualified names that
previously had not raised any assertions.
Mike Bayer [Wed, 22 Dec 2021 20:33:11 +0000 (15:33 -0500)]
add recursion check for with_loader_criteria() option
Fixed recursion overflow which could occur within ORM statement compilation
when using either the :func:`_orm.with_loader_criteria` feature or the the
:meth:`_orm.PropComparator.and_` method within a loader strategy in
conjunction with a subquery which referred to the same entity being altered
by the criteria option, or loaded by the loader strategy. A check for
coming across the same loader criteria option in a recursive fashion has
been added to accommodate for this scenario.
Mike Bayer [Wed, 22 Dec 2021 14:45:38 +0000 (09:45 -0500)]
remove all python_version<3 specs; update pytest
pytest 7 is on rc, so in order to get it installed
when I run tox, bump min version to 7.0.0rc1. The performance
issues with version 6 are not tolerable.
Additionally remove all python_version<3.7 specs from setup.cfg,
tox, these are no longer needed.
Mike Bayer [Tue, 21 Dec 2021 23:08:33 +0000 (18:08 -0500)]
accommodate for "clone" of ColumnClause
for use with the ClauseElement.params() method,
altered ColumnClause._clone() so that while the element
stays immutable, if the column is associated with a subquery,
it returns a new version of itself as corresponding to a
clone of the subquery. this allows processing functions
to access the parameters in the subquery and produce a
copy of it. The use case here is the expanded use of
.params() within loader strategies that use
HasCacheKey._apply_params_to_element().
Fixed issue in new "loader criteria" method
:meth:`_orm.PropComparator.and_` where usage with a loader strategy like
:func:`_orm.selectinload` against a column that was a member of the ``.c.``
collection of a subquery object, where the subquery would be dynamically
added to the FROM clause of the statement, would be subject to stale
parameter values within the subquery in the SQL statement cache, as the
process used by the loader strategy to replace the parameters at execution
time would fail to accommodate the subquery when received in this form.
Mike Bayer [Sun, 19 Dec 2021 20:59:55 +0000 (15:59 -0500)]
implement cython for cache_anon_map, prefix_anon_map
These are small bits where cache_anon_map in particular
is part of the cache key generation scheme which is a key
target for cython.
changing such a tiny element of the cache key gen is
doing basically nothing yet, as the cython impl is
mostly the exact same speed as the python one. I guess for
cython to be effective we'd need to redo the whole cache key
generation and possibly not use the same kinds of structures,
which might not be very easy to do.
Additionally, some cython runtime import errors are being
observed on jenkins, add an upfront check to the test suite
to indicate if the expected build succeeded when REQUIRE_SQLALCHEMY_CEXT
is set.
Mike Bayer [Wed, 15 Dec 2021 20:00:28 +0000 (15:00 -0500)]
use load_scalar_attributes() for undefer
At some point, undeferral of attributes started emitting
a full ORM query for the entity, including that subclass columns
would use a JOIN. Seems to be at least in 1.3 / 1.4 and possibly
earlier. This JOINs to the superclass table unnecessarily.
Use load_scalar_attributes() here which should handle the whole
thing and emits a more efficient query for joined inheritance.
As this behavior seems to have been throughout 1.3 and 1.4
at least, targeting at 2.0 is likely best.
Re-implement c version immutabledict / processors / resultproxy / utils with cython.
Performance is in general in par or better than the c version
Added a collection module that has cython version of OrderedSet and IdentitySet
Added a new test/perf file to compare the implementations.
Run ``python test/perf/compiled_extensions.py all`` to execute the comparison test.
See results here: https://docs.google.com/document/d/1nOcDGojHRtXEkuy4vNXcW_XOJd9gqKhSeALGG3kYr6A/edit?usp=sharing
Mike Bayer [Tue, 14 Dec 2021 21:46:50 +0000 (16:46 -0500)]
include InterfaceError for mariadb disconnect check
Corrected the error classes inspected for the "is_disconnect" check for the
``mariadbconnector`` dialect, which was failing for disconnects that
occurred due to common MySQL/MariaDB error codes such as 2006; the DBAPI
appears to currently use the ``mariadb.InterfaceError`` exception class for
disconnect errors such as error code 2006, which has been added to the list
of classes checked.
For the current "real reconnect test", shutting down the mariadb
connection from the client side produces
ProgrammingError("Connection isn't valid anymore") which we also
continue to intercept.
Nils Philippsen [Sun, 12 Dec 2021 23:35:03 +0000 (18:35 -0500)]
Ignore ephemeral classes in test_all_present()
Fixed a regression in the test suite where the test called
``CompareAndCopyTest::test_all_present`` would fail on some platforms due
to additional testing artifacts being detected. Pull request courtesy Nils
Philippsen.
In some circumstances, ephemeral class objects that are created within
the scope of a test method don't seem to be garbage collected directly
on exit. Filter out classes created in test modules.
Mike Bayer [Sun, 12 Dec 2021 18:37:21 +0000 (13:37 -0500)]
use the options from the cached statement for propagate_options
Fixed caching-related issue where the use of a loader option of the form
``lazyload(aliased(A).bs).joinedload(B.cs)`` would fail to result in the
joinedload being invoked for runs subsequent to the query being cached, due
to a mismatch for the options / object path applied to the objects loaded
for a query with a lead entity that used ``aliased()``.
Nils Philippsen [Sat, 13 Nov 2021 16:11:32 +0000 (11:11 -0500)]
Add async_engine_from_config()
Added :func:`_asyncio.async_engine_config` function to create
an async engine from a configuration dict. This otherwise
behaves the same as :func:`_sa.engine_from_config`.
Mike Bayer [Thu, 9 Dec 2021 19:23:42 +0000 (14:23 -0500)]
implement correct errors for Row immutability
Corrected the error message for the ``AttributeError`` that's raised when
attempting to write to an attribute on the :class:`_result.Row` class,
which is immutable. The previous message claimed the column didn't exist
which is misleading.
Mike Bayer [Thu, 9 Dec 2021 17:51:43 +0000 (12:51 -0500)]
implement attributes.Proxy._clone()
Fixed issue where the internal cloning used by the
:meth:`_orm.PropComparator.any` method on a :func:`_orm.relationship` in
the case where the related class also makes use of ORM polymorphic loading,
would fail if a hybrid property on the related, polymorphic class were used
within the criteria for the ``any()`` operation.
Mike Bayer [Wed, 8 Dec 2021 19:19:11 +0000 (14:19 -0500)]
Removals: strings for join(), loader_options().
* The :meth:`_orm.Query.join` method no longer accepts strings for
relationship names; the long-documented approach of using
``Class.attrname`` for join targets is now standard.
* Loader options no longer accept strings for attribute names. The
long-documented approach of using ``Class.attrname`` for loader option
targets is now standard.
It is hoped that a subsequent commit can refactor loader
options to no longer need "UnboundLoad" for most cases.
Mike Bayer [Wed, 8 Dec 2021 18:55:32 +0000 (13:55 -0500)]
include correct notes for 2.0
the changelog_20 file either started out wrong
or got changed at some point to include the wrong change notes,
so the unreleased_20/ notes were not being displayed.
Mike Bayer [Fri, 3 Dec 2021 19:04:05 +0000 (14:04 -0500)]
Warn when caching is disabled / document
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.
As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``. These have been amended to use inherit
cache where appropriate. "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.
The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.
Mike Bayer [Thu, 2 Dec 2021 00:27:25 +0000 (19:27 -0500)]
contextmanager skips rollback if trans says to skip it
Fixed issue where if an exception occurred when the :class:`_orm.Session`
were to close the connection within the :meth:`_orm.Session.commit` method,
when using a context manager for :meth:`_orm.Session.begin` , it would
attempt a rollback which would not be possible as the :class:`_orm.Session`
was in between where the transaction is committed and the connection is
then to be returned to the pool, raising the exception "this
sessiontransaction is in the committed state". This exception can occur
mostly in an asyncio context where CancelledError can be raised.
The where method of exists now accepts multiple cluase.
Support multiple clause elements in the :meth:`_sql.Exists.where` method,
unifying the api with the on presented by a normal :func:`_sql.select`
construct.
Mike Bayer [Thu, 2 Dec 2021 02:39:59 +0000 (21:39 -0500)]
copy list for __iadd__
Fixed issue where a list mapped with :func:`_orm.relationship` would go
into an endless loop if in-place added to itself, i.e. the ``+=`` operator
were used, as well as if ``.extend()`` were given the same list.
Kai Mueller [Wed, 1 Dec 2021 15:58:40 +0000 (10:58 -0500)]
Add __class_getitem__ to the declarative Base class
Fixed issue where the :func:`_orm.as_declarative` decorator and similar
functions used to generate the declarative base class would not copy the
``__class_getitem__()`` method from a given superclass, which prevented the
use of pep-484 generics in conjunction with the ``Base`` class. Pull
request courtesy Kai Mueller.
Federico Caselli [Sat, 27 Nov 2021 08:53:29 +0000 (09:53 +0100)]
adapt pytest plugin to support pytest v7
Implemented support for the test suite to run correctly under Pytest 7.
Previously, only Pytest 6.x was supported for Python 3, however the version
was not pinned on the upper bound in tox.ini. Pytest is not pinned in
tox.ini to be lower than version 8 so that SQLAlchemy versions released
with the current codebase will be able to be tested under tox without
changes to the environment. Much thanks to the Pytest developers for
their help with this issue.
Mike Bayer [Fri, 26 Nov 2021 15:17:38 +0000 (10:17 -0500)]
provide connectionfairy on initialize
This is so that dialect methods that are called within init
can assume the same argument structure as when they are called
in other places; we can nail down the type of object as well.
This change seems to mostly impact the isolation level routines
in the dialects, as these are called during initialize()
as well as on established connections. these methods can now
assume a non-proxied DBAPI connection object in all cases,
as it is commonly required that attributes like ".autocommit"
are set on the object which don't work well in a proxied
situation.
Other changes:
* adds an interface for the "connectionfairy" concept
called PoolProxiedConnection.
* Removes ``Connectable`` superclass of Connection.
``Connectable`` was originally meant to provide for the
"method which accepts connection or engine" theme. As this
pattern is greatly reduced in 2.0 and Engine no longer extends
from it, the ``Connectable`` superclass doesnt serve any real
purpose.
Leading from that, to set this in I also applied pep 484 annotations
to the Dialect base, and then in the interests of seeing some
of the typing information show up in my IDE did a little bit for Engine,
Connection and others. I hope that it's feasible that we can
add annotations to specific classes and attributes ahead of when we
actually try to mass-populate the whole library. This was
the original spirit of pep-484 that we can apply annotations
gradually. I do of course want to try to do a mass-populate
although i think even in that case we will end up doing a lot
of manual work anyway (in particular for the changes here which
are distinct from what the stubs have).
Mike Bayer [Fri, 26 Nov 2021 23:43:23 +0000 (18:43 -0500)]
"graceful fetch" test should....clean up gracefully!
this test was relying on gc to close out the connection.
this would lead to problems with aiosqlite as we invalidate
async connetions that aren't gracefully closed, and this test
suite was create tables per suite, so we'd get into a spot where
a new sqlite memory connection without the tables would get set up.
would only occur for full test run + -n2 + C extensions + python 3.7,
but we assume that is all related to just getting gc to trigger
or not trigger at exactly the right moment in this situation.
confirmed if we add a gc.collect() to the test without explcitly
closing out the conenction, the connection is gc'ed and detached,
and we get the error reproduced on the subsequent test.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
