Mike Bayer [Sun, 2 Jan 2022 22:35:43 +0000 (17:35 -0500)]
initial reorganize for static typing
start applying foundational annotations to key
elements.
two main elements addressed here:
1. removal of public_factory() and replacement with
explicit functions. this just works much better with
typing.
2. typing support for column expressions and operators.
The biggest part of this involves stubbing out all the
ColumnOperators methods under ColumnElement in a
TYPE_CHECKING section. Took me a while to see this
method vs. much more complicated things I thought
I needed.
Also for this version implementing #7519, ColumnElement
types against the Python type and not TypeEngine. it is
hoped this leads to easier transferrence between ORM/Core
as well as eventual support for result set typing.
Not clear yet how well this approach will work and what
new issues it may introduce.
given the current approach we now get full, rich typing for
scenarios like this:
from sqlalchemy import column, Integer, String, Boolean
Mike Bayer [Mon, 10 Jan 2022 21:48:05 +0000 (16:48 -0500)]
remove internal use of metaclasses
All but one metaclass used internally can now
be replaced using __init_subclass__(). Within this
patch we remove:
* events._EventMeta
* sql.visitors.TraversibleType
* sql.visitors.InternalTraversibleType
* testing.fixtures.FindFixture
* testing.fixtures.FindFixtureDeclarative
* langhelpers.EnsureKWArgType
* sql.functions._GenericMeta
* sql.type_api.VisitableCheckKWArg (was a mixture of TraversibleType
and EnsureKWArgType)
The remaining internal class is MetaOptions used by the
sql.Options object which is in turn currently mostly for
ORM internal use, as this type implements class level overrides
for the ``+`` operator.
For declarative, removing DeclarativeMeta in place of
an `__init_subclass__()` class would not be fully feasible as
it would break backwards compatibility with applications that
refer to this class explicitly, but also DeclarativeMeta intercepts
class-level attribute set and delete operations which is a widely
used pattern. An option for declarative base to use
`__init_subclass__()` should be provided but this is out of
scope for this particular change.
the description_encoding concept has been fully
removed in 2.0, but this logic was plainly broken
in any case. The same non-working logic exists in
1.4 as well, where description_encoding as a concept
is not removed, though it is likely that there are no
real-world use cases for it.
Mike Bayer [Tue, 11 Jan 2022 03:11:50 +0000 (22:11 -0500)]
change state.load_options to a tuple
having this be an immutable sequence is safer
and possibly lower overhead.
The change here went in with no issues save
for tests that asserted it was a set.
InstanceState.load_options is only referred
towards when the object is first loaded, and then
within the logic that emits an object refresh
as well as within a lazy loader. it's only
accessed as a whole collection.
Mike Bayer [Wed, 5 Jan 2022 17:20:46 +0000 (12:20 -0500)]
implement second-level type resolution for literals
Added additional rule to the system that determines ``TypeEngine``
implementations from Python literals to apply a second level of adjustment
to the type, so that a Python datetime with or without tzinfo can set the
``timezone=True`` parameter on the returned :class:`.DateTime` object, as
well as :class:`.Time`. This helps with some round-trip scenarios on
type-sensitive PostgreSQL dialects such as asyncpg, psycopg3 (2.0 only).
For 1.4 specifically, the backport improves support for asyncpg handling of
TIME WITH TIMEZONE, which was not fully implemented. 2.0's reworked
PostgreSQL architecture had this handled already.
Mike Bayer [Mon, 10 Jan 2022 19:59:59 +0000 (14:59 -0500)]
ensure with_options not switched to a list
Fixed regression which appeared in 1.4.23 which could cause loader options
to be mis-handled in some cases, in particular when using joined table
inheritance in combination with the ``polymorphic_load="selectin"`` option
as well as relationship lazy loading, leading to a ``TypeError``.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
There's a few bits and pieces of code to support Python <= 3.6 which are no longer needed and can be removed, to slightly simplify the codebase.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Mon, 3 Jan 2022 18:49:26 +0000 (13:49 -0500)]
Remove all remaining removed_in_20 warnings slated for removal
Finalize all remaining removed-in-2.0 changes so that we
can begin doing pep-484 typing without old things
getting in the way (we will also have to do public_factory).
note there are a few "moved_in_20()" and "became_legacy_in_20()"
warnings still in place. The SQLALCHEMY_WARN_20 variable
is now removed.
Also removed here are the legacy "in place mutators" for Select
statements, and some keyword-only argument signatures in Core
have been added.
Also in the big change department, the ORM mapper() function
is removed entirely; the Mapper class is otherwise unchanged,
just the public-facing API function. Mappers are now always
given a registry in which to participate, however the
argument signature of Mapper is not changed. ideally "registry"
would be the first positional argument.
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.
Also update Black and other pre-commit hooks and re-format with Black.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Tue, 4 Jan 2022 19:04:15 +0000 (14:04 -0500)]
implement python_impl to custom_op for basic ORM evaluator extensibility
Added new parameter :paramref:`_sql.Operators.op.python_impl`, available
from :meth:`_sql.Operators.op` and also when using the
:class:`_sql.Operators.custom_op` constructor directly, which allows an
in-Python evaluation function to be provided along with the custom SQL
operator. This evaluation function becomes the implementation used when the
operator object is used given plain Python objects as operands on both
sides, and in particular is compatible with the
``synchronize_session='evaluate'`` option used with
:ref:`orm_expression_update_delete`.
Mike Bayer [Tue, 4 Jan 2022 19:08:10 +0000 (14:08 -0500)]
improve custom operator for SQL types docs
introduce here that "custom ops" first come from
the .op() method in the usual case. then only if one wants
such an op to be pre-assocaited with particular types,
then the comparator may be used. Also clarify the individual
points regarding the comparator.
Mike Bayer [Mon, 3 Jan 2022 22:28:52 +0000 (17:28 -0500)]
ensure correlate_except is checked for empty tuple
Fixed issue where :meth:`_sql.Select.correlate_except` method, when passed
either the ``None`` value or no arguments, would not correlate any elements
when used in an ORM context (that is, passing ORM entities as FROM
clauses), rather than causing all FROM elements to be considered as
"correlated" in the same way which occurs when using Core-only constructs.
Federico Caselli [Thu, 30 Dec 2021 22:29:55 +0000 (23:29 +0100)]
Imrpove MySQL/MariaDB dialect initialization.
Replace ``SHOW VARIABLES LIKE`` statement with equivalent
``SELECT @@variable`` in MySQL and MariaDB dialect initialization.
This should avoid mutex contention caused by ``SHOW VARIABLES``,
improving initialization performance.
Mike Bayer [Sat, 1 Jan 2022 21:50:41 +0000 (16:50 -0500)]
adjust TypeError message yet again
in 924d38e660981ae2a7d55355603 we got it almost right in
all cases, Pypy seems to have an entirely different kind of message
so make this much more open ended.
Mike Bayer [Sat, 1 Jan 2022 20:30:15 +0000 (15:30 -0500)]
adjust error message assertion for TypeError
in eee7a3add99df2865e6d907f2 I added a few TypeError
tests, and some python interpreters produce a message like
"join() got an unexpected...", others seem to produce
"Query.join() got an unexpected..."
the pattern is strange, all Python 3.10's seem to return the
class name. For Python 3.8 and 3.9, my local intepreters
are doing the Python 3.10 behavior but github actions are doing
the older behavior. Python 3.7 both here and on github actions
are doing the older behavior.
Federico Caselli [Wed, 22 Dec 2021 20:45:45 +0000 (21:45 +0100)]
Properly type _generative, decorator, public_factory
Good new is that pylance likes it and copies over the
singature and everything.
Bad news is that mypy does not support this yet https://github.com/python/mypy/issues/8645
Other minor bad news is that non_generative is not typed. I've tried using a protocol
like the one in the comment but the signature is not ported over by pylance, so it's
probably best to just live without it to have the correct signature.
notes from mike: these three decorators are at the core of getting
the library to be typed, more good news is that pylance will
do all the things we like re: public_factory, see
https://github.com/microsoft/pyright/issues/2758#issuecomment-1002788656
.
For @_generative, we will likely move to using pep 673 once mypy
supports it which may be soon. but overall having the explicit
"return self" in the methods, while a little inconvenient, makes
the typing more straightforward and locally present in the files
rather than being decided at a distance. having "return self"
present, or not, both have problems, so maybe we will be able
to change it again if things change as far as decorator support.
As it is, I feel like we are barely squeaking by with our decorators,
the typing is already pretty out there.
Mike Bayer [Sat, 23 Oct 2021 15:26:45 +0000 (11:26 -0400)]
replace Variant with direct feature inside of TypeEngine
The :meth:`_sqltypes.TypeEngine.with_variant` method now returns a copy of
the original :class:`_sqltypes.TypeEngine` object, rather than wrapping it
inside the ``Variant`` class, which is effectively removed (the import
symbol remains for backwards compatibility with code that may be testing
for this symbol). While the previous approach maintained in-Python
behaviors, maintaining the original type allows for clearer type checking
and debugging.
Gord Thompson [Mon, 20 Dec 2021 21:37:13 +0000 (14:37 -0700)]
Reflect included columns as dialect_options
Fixed reflection of covering indexes to report ``include_columns`` as part
of the ``dialect_options`` entry in the reflected index dictionary, thereby
enabling round trips from reflection->create to be complete. Included
columns continue to also be present under the ``include_columns`` key for
backwards compatibility.
Mike Bayer [Thu, 2 Dec 2021 14:18:11 +0000 (09:18 -0500)]
propose concurrency check for SessionTransaction
the discussion at #7387 refers to a condition that seems
to happen in the wild also, such as [1] [2] [3], it's not
entirely clear why this specific spot is how this occurs,
however it's maybe that when the connection is being acquired
from the pool, under load there might be a wait on the connection
pool, leading to more time for another errant thread to be
calling .close(), just a theory.
in this patch we propose using decorators and context managers
along with declarative state declarations to block reentrant
or concurrent calls to methods that conflict with expected
state changes.
The :class:`_orm.Session` (and by extension :class:`.AsyncSession`) now has
new state-tracking functionality that will proactively trap any unexpected
state changes which occur as a particular transactional method proceeds.
This is to allow situations where the :class:`_orm.Session` is being used
in a thread-unsafe manner, where event hooks or similar may be calling
unexpected methods within operations, as well as potentially under other
concurrency situations such as asyncio or gevent to raise an informative
message when the illegal access first occurs, rather than passing silently
leading to secondary failures due to the :class:`_orm.Session` being in an
invalid state.
Mike Bayer [Thu, 9 Dec 2021 01:27:16 +0000 (20:27 -0500)]
factor out UnboundLoad and rearchitect strategy_options.py
The architecture of Load is mostly rewritten here.
The change includes removal of the "pluggable" aspect
of the loader options, which would patch new methods onto
Load. This has been replaced by normal methods that
respond normally to typing annotations. As part of this
change, the bake_loaders() and unbake_loaders() options,
which have no effect since 1.4 and were unlikely to be
in any common use, have been removed.
Additionally, to support annotations for methods that
make use of @decorator, @generative etc., modified
format_argspec_plus to no longer return "args", instead
returns "grouped_args" which is always grouped and
allows return annotations to format correctly.
Mike Bayer [Wed, 8 Dec 2021 13:57:44 +0000 (08:57 -0500)]
consider truediv as truediv; support floordiv operator
Implemented full support for "truediv" and "floordiv" using the
"/" and "//" operators. A "truediv" operation between two expressions
using :class:`_types.Integer` now considers the result to be
:class:`_types.Numeric`, and the dialect-level compilation will cast
the right operand to a numeric type on a dialect-specific basis to ensure
truediv is achieved. For floordiv, conversion is also added for those
databases that don't already do floordiv by default (MySQL, Oracle) and
the ``FLOOR()`` function is rendered in this case, as well as for
cases where the right operand is not an integer (needed for PostgreSQL,
others).
The change resolves issues both with inconsistent behavior of the
division operator on different backends and also fixes an issue where
integer division on Oracle would fail to be able to fetch a result due
to inappropriate outputtypehandlers.
Mike Bayer [Sun, 26 Dec 2021 17:13:19 +0000 (12:13 -0500)]
include empty intermediary tables in optimized get
Fixed issue in joined-inheritance load of additional attributes
functionality in deep multi-level inheritance where an intermediary table
that contained no columns would not be included in the tables joined,
instead linking those tables to their primary key identifiers. While this
works fine, it nonetheless in 1.4 began producing the cartesian product
compiler warning. The logic has been changed so that these intermediary
tables are included regardless. While this does include additional tables
in the query that are not technically necessary, this only occurs for the
highly unusual case of deep 3+ level inheritance with intermediary tables
that have no non primary key columns, potential performance impact is
therefore expected to be negligible.
Mike Bayer [Sun, 26 Dec 2021 16:25:00 +0000 (11:25 -0500)]
restore graceful degrade of subqueryload w from_statement
Fixed regression from 1.3 where the "subqueryload" loader strategy would
fail with a stack trace if used against a query that made use of
:meth:`_orm.Query.from_statement` or :meth:`_sql.Select.from_statement`. As
subqueryload requires modifying the original statement, it's not compatible
with the "from_statement" use case, especially for statements made against
the :func:`_sql.text` construct. The behavior now is equivalent to that of
1.3 and previously, which is that the loader strategy silently degrades to
not be used for such statements, typically falling back to using the
lazyload strategy.
Kai Mueller [Tue, 21 Dec 2021 20:00:30 +0000 (15:00 -0500)]
Fix missing class attributes when using __class_getitem__
Fixed issue where the ``__class_getitem__()`` method of the generated
declarative base class by :func:`_orm.as_declarative` would lead to
inaccessible class attributes such as ``__table__``, for cases where a
``Generic[T]`` style typing declaration were used in the class hierarchy.
This is in continuation from the basic addition of ``__class_getitem__()``
in :ticket:`7368`. Pull request courtesy Kai Mueller.
Mike Bayer [Wed, 22 Dec 2021 21:21:33 +0000 (16:21 -0500)]
use fully qualified, locatable names for all use of api.named_type()
Fixed mypy regression where the release of mypy 0.930 added additional
internal checks to the format of "named types", requiring that they be
fully qualified and locatable. This broke the mypy plugin for SQLAlchemy,
raising an assertion error, as there was use of symbols such as
``__builtins__`` and other un-locatable or unqualified names that
previously had not raised any assertions.
Mike Bayer [Wed, 22 Dec 2021 20:33:11 +0000 (15:33 -0500)]
add recursion check for with_loader_criteria() option
Fixed recursion overflow which could occur within ORM statement compilation
when using either the :func:`_orm.with_loader_criteria` feature or the the
:meth:`_orm.PropComparator.and_` method within a loader strategy in
conjunction with a subquery which referred to the same entity being altered
by the criteria option, or loaded by the loader strategy. A check for
coming across the same loader criteria option in a recursive fashion has
been added to accommodate for this scenario.
Mike Bayer [Wed, 22 Dec 2021 14:45:38 +0000 (09:45 -0500)]
remove all python_version<3 specs; update pytest
pytest 7 is on rc, so in order to get it installed
when I run tox, bump min version to 7.0.0rc1. The performance
issues with version 6 are not tolerable.
Additionally remove all python_version<3.7 specs from setup.cfg,
tox, these are no longer needed.
Mike Bayer [Tue, 21 Dec 2021 23:08:33 +0000 (18:08 -0500)]
accommodate for "clone" of ColumnClause
for use with the ClauseElement.params() method,
altered ColumnClause._clone() so that while the element
stays immutable, if the column is associated with a subquery,
it returns a new version of itself as corresponding to a
clone of the subquery. this allows processing functions
to access the parameters in the subquery and produce a
copy of it. The use case here is the expanded use of
.params() within loader strategies that use
HasCacheKey._apply_params_to_element().
Fixed issue in new "loader criteria" method
:meth:`_orm.PropComparator.and_` where usage with a loader strategy like
:func:`_orm.selectinload` against a column that was a member of the ``.c.``
collection of a subquery object, where the subquery would be dynamically
added to the FROM clause of the statement, would be subject to stale
parameter values within the subquery in the SQL statement cache, as the
process used by the loader strategy to replace the parameters at execution
time would fail to accommodate the subquery when received in this form.
Mike Bayer [Sun, 19 Dec 2021 20:59:55 +0000 (15:59 -0500)]
implement cython for cache_anon_map, prefix_anon_map
These are small bits where cache_anon_map in particular
is part of the cache key generation scheme which is a key
target for cython.
changing such a tiny element of the cache key gen is
doing basically nothing yet, as the cython impl is
mostly the exact same speed as the python one. I guess for
cython to be effective we'd need to redo the whole cache key
generation and possibly not use the same kinds of structures,
which might not be very easy to do.
Additionally, some cython runtime import errors are being
observed on jenkins, add an upfront check to the test suite
to indicate if the expected build succeeded when REQUIRE_SQLALCHEMY_CEXT
is set.
Mike Bayer [Wed, 15 Dec 2021 20:00:28 +0000 (15:00 -0500)]
use load_scalar_attributes() for undefer
At some point, undeferral of attributes started emitting
a full ORM query for the entity, including that subclass columns
would use a JOIN. Seems to be at least in 1.3 / 1.4 and possibly
earlier. This JOINs to the superclass table unnecessarily.
Use load_scalar_attributes() here which should handle the whole
thing and emits a more efficient query for joined inheritance.
As this behavior seems to have been throughout 1.3 and 1.4
at least, targeting at 2.0 is likely best.
Re-implement c version immutabledict / processors / resultproxy / utils with cython.
Performance is in general in par or better than the c version
Added a collection module that has cython version of OrderedSet and IdentitySet
Added a new test/perf file to compare the implementations.
Run ``python test/perf/compiled_extensions.py all`` to execute the comparison test.
See results here: https://docs.google.com/document/d/1nOcDGojHRtXEkuy4vNXcW_XOJd9gqKhSeALGG3kYr6A/edit?usp=sharing
Mike Bayer [Tue, 14 Dec 2021 21:46:50 +0000 (16:46 -0500)]
include InterfaceError for mariadb disconnect check
Corrected the error classes inspected for the "is_disconnect" check for the
``mariadbconnector`` dialect, which was failing for disconnects that
occurred due to common MySQL/MariaDB error codes such as 2006; the DBAPI
appears to currently use the ``mariadb.InterfaceError`` exception class for
disconnect errors such as error code 2006, which has been added to the list
of classes checked.
For the current "real reconnect test", shutting down the mariadb
connection from the client side produces
ProgrammingError("Connection isn't valid anymore") which we also
continue to intercept.
Nils Philippsen [Sun, 12 Dec 2021 23:35:03 +0000 (18:35 -0500)]
Ignore ephemeral classes in test_all_present()
Fixed a regression in the test suite where the test called
``CompareAndCopyTest::test_all_present`` would fail on some platforms due
to additional testing artifacts being detected. Pull request courtesy Nils
Philippsen.
In some circumstances, ephemeral class objects that are created within
the scope of a test method don't seem to be garbage collected directly
on exit. Filter out classes created in test modules.
Mike Bayer [Sun, 12 Dec 2021 18:37:21 +0000 (13:37 -0500)]
use the options from the cached statement for propagate_options
Fixed caching-related issue where the use of a loader option of the form
``lazyload(aliased(A).bs).joinedload(B.cs)`` would fail to result in the
joinedload being invoked for runs subsequent to the query being cached, due
to a mismatch for the options / object path applied to the objects loaded
for a query with a lead entity that used ``aliased()``.
Nils Philippsen [Sat, 13 Nov 2021 16:11:32 +0000 (11:11 -0500)]
Add async_engine_from_config()
Added :func:`_asyncio.async_engine_config` function to create
an async engine from a configuration dict. This otherwise
behaves the same as :func:`_sa.engine_from_config`.
Mike Bayer [Thu, 9 Dec 2021 19:23:42 +0000 (14:23 -0500)]
implement correct errors for Row immutability
Corrected the error message for the ``AttributeError`` that's raised when
attempting to write to an attribute on the :class:`_result.Row` class,
which is immutable. The previous message claimed the column didn't exist
which is misleading.
Mike Bayer [Thu, 9 Dec 2021 17:51:43 +0000 (12:51 -0500)]
implement attributes.Proxy._clone()
Fixed issue where the internal cloning used by the
:meth:`_orm.PropComparator.any` method on a :func:`_orm.relationship` in
the case where the related class also makes use of ORM polymorphic loading,
would fail if a hybrid property on the related, polymorphic class were used
within the criteria for the ``any()`` operation.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
