Mike Bayer [Thu, 20 Jan 2022 20:21:17 +0000 (15:21 -0500)]
re-enable tests for asyncmy; fix Binary
Fixed regression in asyncmy dialect caused by :ticket:`7567` where removal
of the PyMySQL dependency broke binary columns, due to the asyncmy dialect
not being properly included within CI tests.
Also repairs mariadbconnector isolation level for 2.0.
basically tox config was failing to include additional
drivers.
Mike Bayer [Thu, 20 Jan 2022 20:01:47 +0000 (15:01 -0500)]
restore empty list logic to ARRAY of ENUM parsing
Fixed regression where the change in :ticket:`7148` to repair ENUM handling
in PostgreSQL broke the use case of an empty ARRAY of ENUM, preventing rows
that contained an empty array from being handled correctly when fetching
results.
Mike Bayer [Thu, 20 Jan 2022 14:31:42 +0000 (09:31 -0500)]
repair mapper sort
Fixed issue in :meth:`_orm.Session.bulk_save_mappings` where the sorting
that takes place when the ``preserve_order`` parameter is set to False
would sort partially on ``Mapper`` objects, which is rejected in Python
3.11.
Also uses typing_extensions for NotRequired as this symbol
does not seem to be in Python 3.11.0a4 yet.
For interim 3.11 support, adds the git main build of greenlet
for Python 3.11
Mike Bayer [Thu, 20 Jan 2022 17:26:36 +0000 (12:26 -0500)]
repair broken truediv test suite; memusage
the truediv test suite didn't have __backend__ so wasn't running
for every DB except in the main build. Repaired this as well
as truediv support to preserve the right-hand side type
when casting to numeric, if the right type is already a
numeric type.
also fixed a memusage test that relies on savepoints so was
not running under gerrit runs.
Mike Bayer [Wed, 19 Jan 2022 19:31:52 +0000 (14:31 -0500)]
Add AdaptedConnection.run_async
Added new method :meth:`.AdaptedConnection.run_async` to the DBAPI
connection interface used by asyncio drivers, which allows methods to be
called against the underlying "driver" connection directly within a
sync-style function where the ``await`` keyword can't be used, such as
within SQLAlchemy event handler functions. The method is analogous to the
:meth:`_asyncio.AsyncConnection.run_sync` method which translates
async-style calls to sync-style. The method is useful for things like
connection-pool on-connect handlers that need to invoke awaitable methods
on the driver connection when it's first created.
Mike Bayer [Tue, 18 Jan 2022 22:19:24 +0000 (17:19 -0500)]
reject methods as lambda SQL callables
Added an informative error message when a method object is passed to a SQL
construct. Previously, when such a callable were passed, as is a common
typographical error when dealing with method-chained SQL constructs, they
were interpreted as "lambda SQL" targets to be invoked at compilation time,
which would lead to silent failures. As this feature was not intended to be
used with methods, method objects are now rejected.
Mike Bayer [Sun, 16 Jan 2022 15:21:45 +0000 (10:21 -0500)]
enhance double-aliased table logic to handle more cases
Fixed ORM regression where calling the :func:`_orm.aliased` function
against an existing :func:`_orm.aliased` construct would fail to produce
correct SQL if the existing construct were against a fixed table. The fix
allows that the original :func:`_orm.aliased` construct is disregarded if
it were only against a table that's now being replaced. It also allows for
correct behavior when constructing a :func:`_orm.aliased` without a
selectable argument against a :func:`_orm.aliased` that's against a
subuquery, to create an alias of that subquery (i.e. to change its name).
The nesting behavior of :func:`_orm.aliased` remains in place for the case
where the outer :func:`_orm.aliased` object is against a subquery which in
turn refers to the inner :func:`_orm.aliased` object. This is a relatively
new 1.4 feature that helps to suit use cases that were previously served by
the deprecated ``Query.from_self()`` method.
Mike Bayer [Tue, 18 Jan 2022 16:02:57 +0000 (11:02 -0500)]
detect map_imperatively() called twice
Fixed issue where calling upon :meth:`_orm.regsitry.map_imperatively` more
than once for the same class would produce an unexpected error, rather than
an informative error that the target class is already mapped. This behavior
differed from that of the :func:`_orm.mapper` function which does report an
informative message already.
For 2.0, this change also cleans up the logic that detects against
`Mapper()` or `_mapper()` being invoked directly. 1.4's backport will
take on a different format as `mapper()` is still public API in that
release.
Mike Bayer [Fri, 14 Jan 2022 22:42:44 +0000 (17:42 -0500)]
remove dbapi_proxy module
This module was not documented nor part of any test suite,
and it's unlikely it was working correctly. It's not likely
that this module was ever used after the first year or so
of SQLAlchemy, and it's stayed around because it is so
obscure that I never remembered to remove it.
Mike Bayer [Fri, 17 Dec 2021 23:04:47 +0000 (18:04 -0500)]
track item schema names to identify name collisions w/ default schema
Added an additional lookup step to the compiler which will track all FROM
clauses which are tables, that may have the same name shared in multiple
schemas where one of the schemas is the implicit "default" schema; in this
case, the table name when referring to that name without a schema
qualification will be rendered with an anonymous alias name at the compiler
level in order to disambiguate the two (or more) names. The approach of
schema-qualifying the normally unqualified name with the server-detected
"default schema name" value was also considered, however this approach
doesn't apply to Oracle nor is it accepted by SQL Server, nor would it work
with multiple entries in the PostgreSQL search path. The name collision
issue resolved here has been identified as affecting at least Oracle,
PostgreSQL, SQL Server, MySQL and MariaDB.
José Duarte [Thu, 13 Jan 2022 22:20:06 +0000 (17:20 -0500)]
Fixes(#7561) Add support for postgres.UUID literal_binds compilation
Added string rendering to the :class:`.postgresql.UUID` datatype, so that
stringifying a statement with "literal_binds" that uses this type will
render an appropriate string value for the PostgreSQL backend. Pull request
courtesy José Duarte.
Mike Bayer [Sun, 9 Jan 2022 16:49:02 +0000 (11:49 -0500)]
Initial ORM typing layout
introduces:
1. new mapped_column() helper
2. DeclarativeBase helper
3. declared_attr has been re-typed
4. rework of Mapped[] to return InstrumentedAtribute for
class get, so works without Mapped itself having expression
methods
5. ORM constructs now generic on [_T]
also includes some early typing work, most of which will
be in later commits:
1. URL and History become typing.NamedTuple
2. come up with type-checking friendly way of type
checking cy extensions, where type checking will be applied
to the py versions, just needed to come up with a succinct
conditional pattern for the imports
Mike Bayer [Sun, 2 Jan 2022 22:35:43 +0000 (17:35 -0500)]
initial reorganize for static typing
start applying foundational annotations to key
elements.
two main elements addressed here:
1. removal of public_factory() and replacement with
explicit functions. this just works much better with
typing.
2. typing support for column expressions and operators.
The biggest part of this involves stubbing out all the
ColumnOperators methods under ColumnElement in a
TYPE_CHECKING section. Took me a while to see this
method vs. much more complicated things I thought
I needed.
Also for this version implementing #7519, ColumnElement
types against the Python type and not TypeEngine. it is
hoped this leads to easier transferrence between ORM/Core
as well as eventual support for result set typing.
Not clear yet how well this approach will work and what
new issues it may introduce.
given the current approach we now get full, rich typing for
scenarios like this:
from sqlalchemy import column, Integer, String, Boolean
Mike Bayer [Mon, 10 Jan 2022 21:48:05 +0000 (16:48 -0500)]
remove internal use of metaclasses
All but one metaclass used internally can now
be replaced using __init_subclass__(). Within this
patch we remove:
* events._EventMeta
* sql.visitors.TraversibleType
* sql.visitors.InternalTraversibleType
* testing.fixtures.FindFixture
* testing.fixtures.FindFixtureDeclarative
* langhelpers.EnsureKWArgType
* sql.functions._GenericMeta
* sql.type_api.VisitableCheckKWArg (was a mixture of TraversibleType
and EnsureKWArgType)
The remaining internal class is MetaOptions used by the
sql.Options object which is in turn currently mostly for
ORM internal use, as this type implements class level overrides
for the ``+`` operator.
For declarative, removing DeclarativeMeta in place of
an `__init_subclass__()` class would not be fully feasible as
it would break backwards compatibility with applications that
refer to this class explicitly, but also DeclarativeMeta intercepts
class-level attribute set and delete operations which is a widely
used pattern. An option for declarative base to use
`__init_subclass__()` should be provided but this is out of
scope for this particular change.
the description_encoding concept has been fully
removed in 2.0, but this logic was plainly broken
in any case. The same non-working logic exists in
1.4 as well, where description_encoding as a concept
is not removed, though it is likely that there are no
real-world use cases for it.
Mike Bayer [Tue, 11 Jan 2022 03:11:50 +0000 (22:11 -0500)]
change state.load_options to a tuple
having this be an immutable sequence is safer
and possibly lower overhead.
The change here went in with no issues save
for tests that asserted it was a set.
InstanceState.load_options is only referred
towards when the object is first loaded, and then
within the logic that emits an object refresh
as well as within a lazy loader. it's only
accessed as a whole collection.
Mike Bayer [Wed, 5 Jan 2022 17:20:46 +0000 (12:20 -0500)]
implement second-level type resolution for literals
Added additional rule to the system that determines ``TypeEngine``
implementations from Python literals to apply a second level of adjustment
to the type, so that a Python datetime with or without tzinfo can set the
``timezone=True`` parameter on the returned :class:`.DateTime` object, as
well as :class:`.Time`. This helps with some round-trip scenarios on
type-sensitive PostgreSQL dialects such as asyncpg, psycopg3 (2.0 only).
For 1.4 specifically, the backport improves support for asyncpg handling of
TIME WITH TIMEZONE, which was not fully implemented. 2.0's reworked
PostgreSQL architecture had this handled already.
Mike Bayer [Mon, 10 Jan 2022 19:59:59 +0000 (14:59 -0500)]
ensure with_options not switched to a list
Fixed regression which appeared in 1.4.23 which could cause loader options
to be mis-handled in some cases, in particular when using joined table
inheritance in combination with the ``polymorphic_load="selectin"`` option
as well as relationship lazy loading, leading to a ``TypeError``.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
There's a few bits and pieces of code to support Python <= 3.6 which are no longer needed and can be removed, to slightly simplify the codebase.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Mon, 3 Jan 2022 18:49:26 +0000 (13:49 -0500)]
Remove all remaining removed_in_20 warnings slated for removal
Finalize all remaining removed-in-2.0 changes so that we
can begin doing pep-484 typing without old things
getting in the way (we will also have to do public_factory).
note there are a few "moved_in_20()" and "became_legacy_in_20()"
warnings still in place. The SQLALCHEMY_WARN_20 variable
is now removed.
Also removed here are the legacy "in place mutators" for Select
statements, and some keyword-only argument signatures in Core
have been added.
Also in the big change department, the ORM mapper() function
is removed entirely; the Mapper class is otherwise unchanged,
just the public-facing API function. Mappers are now always
given a registry in which to participate, however the
argument signature of Mapper is not changed. ideally "registry"
would be the first positional argument.
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.
Also update Black and other pre-commit hooks and re-format with Black.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Tue, 4 Jan 2022 19:04:15 +0000 (14:04 -0500)]
implement python_impl to custom_op for basic ORM evaluator extensibility
Added new parameter :paramref:`_sql.Operators.op.python_impl`, available
from :meth:`_sql.Operators.op` and also when using the
:class:`_sql.Operators.custom_op` constructor directly, which allows an
in-Python evaluation function to be provided along with the custom SQL
operator. This evaluation function becomes the implementation used when the
operator object is used given plain Python objects as operands on both
sides, and in particular is compatible with the
``synchronize_session='evaluate'`` option used with
:ref:`orm_expression_update_delete`.
Mike Bayer [Tue, 4 Jan 2022 19:08:10 +0000 (14:08 -0500)]
improve custom operator for SQL types docs
introduce here that "custom ops" first come from
the .op() method in the usual case. then only if one wants
such an op to be pre-assocaited with particular types,
then the comparator may be used. Also clarify the individual
points regarding the comparator.
Mike Bayer [Mon, 3 Jan 2022 22:28:52 +0000 (17:28 -0500)]
ensure correlate_except is checked for empty tuple
Fixed issue where :meth:`_sql.Select.correlate_except` method, when passed
either the ``None`` value or no arguments, would not correlate any elements
when used in an ORM context (that is, passing ORM entities as FROM
clauses), rather than causing all FROM elements to be considered as
"correlated" in the same way which occurs when using Core-only constructs.
Federico Caselli [Thu, 30 Dec 2021 22:29:55 +0000 (23:29 +0100)]
Imrpove MySQL/MariaDB dialect initialization.
Replace ``SHOW VARIABLES LIKE`` statement with equivalent
``SELECT @@variable`` in MySQL and MariaDB dialect initialization.
This should avoid mutex contention caused by ``SHOW VARIABLES``,
improving initialization performance.
Mike Bayer [Sat, 1 Jan 2022 21:50:41 +0000 (16:50 -0500)]
adjust TypeError message yet again
in 924d38e660981ae2a7d55355603 we got it almost right in
all cases, Pypy seems to have an entirely different kind of message
so make this much more open ended.
Mike Bayer [Sat, 1 Jan 2022 20:30:15 +0000 (15:30 -0500)]
adjust error message assertion for TypeError
in eee7a3add99df2865e6d907f2 I added a few TypeError
tests, and some python interpreters produce a message like
"join() got an unexpected...", others seem to produce
"Query.join() got an unexpected..."
the pattern is strange, all Python 3.10's seem to return the
class name. For Python 3.8 and 3.9, my local intepreters
are doing the Python 3.10 behavior but github actions are doing
the older behavior. Python 3.7 both here and on github actions
are doing the older behavior.
Federico Caselli [Wed, 22 Dec 2021 20:45:45 +0000 (21:45 +0100)]
Properly type _generative, decorator, public_factory
Good new is that pylance likes it and copies over the
singature and everything.
Bad news is that mypy does not support this yet https://github.com/python/mypy/issues/8645
Other minor bad news is that non_generative is not typed. I've tried using a protocol
like the one in the comment but the signature is not ported over by pylance, so it's
probably best to just live without it to have the correct signature.
notes from mike: these three decorators are at the core of getting
the library to be typed, more good news is that pylance will
do all the things we like re: public_factory, see
https://github.com/microsoft/pyright/issues/2758#issuecomment-1002788656
.
For @_generative, we will likely move to using pep 673 once mypy
supports it which may be soon. but overall having the explicit
"return self" in the methods, while a little inconvenient, makes
the typing more straightforward and locally present in the files
rather than being decided at a distance. having "return self"
present, or not, both have problems, so maybe we will be able
to change it again if things change as far as decorator support.
As it is, I feel like we are barely squeaking by with our decorators,
the typing is already pretty out there.
Mike Bayer [Sat, 23 Oct 2021 15:26:45 +0000 (11:26 -0400)]
replace Variant with direct feature inside of TypeEngine
The :meth:`_sqltypes.TypeEngine.with_variant` method now returns a copy of
the original :class:`_sqltypes.TypeEngine` object, rather than wrapping it
inside the ``Variant`` class, which is effectively removed (the import
symbol remains for backwards compatibility with code that may be testing
for this symbol). While the previous approach maintained in-Python
behaviors, maintaining the original type allows for clearer type checking
and debugging.
Gord Thompson [Mon, 20 Dec 2021 21:37:13 +0000 (14:37 -0700)]
Reflect included columns as dialect_options
Fixed reflection of covering indexes to report ``include_columns`` as part
of the ``dialect_options`` entry in the reflected index dictionary, thereby
enabling round trips from reflection->create to be complete. Included
columns continue to also be present under the ``include_columns`` key for
backwards compatibility.
Mike Bayer [Thu, 2 Dec 2021 14:18:11 +0000 (09:18 -0500)]
propose concurrency check for SessionTransaction
the discussion at #7387 refers to a condition that seems
to happen in the wild also, such as [1] [2] [3], it's not
entirely clear why this specific spot is how this occurs,
however it's maybe that when the connection is being acquired
from the pool, under load there might be a wait on the connection
pool, leading to more time for another errant thread to be
calling .close(), just a theory.
in this patch we propose using decorators and context managers
along with declarative state declarations to block reentrant
or concurrent calls to methods that conflict with expected
state changes.
The :class:`_orm.Session` (and by extension :class:`.AsyncSession`) now has
new state-tracking functionality that will proactively trap any unexpected
state changes which occur as a particular transactional method proceeds.
This is to allow situations where the :class:`_orm.Session` is being used
in a thread-unsafe manner, where event hooks or similar may be calling
unexpected methods within operations, as well as potentially under other
concurrency situations such as asyncio or gevent to raise an informative
message when the illegal access first occurs, rather than passing silently
leading to secondary failures due to the :class:`_orm.Session` being in an
invalid state.
Mike Bayer [Thu, 9 Dec 2021 01:27:16 +0000 (20:27 -0500)]
factor out UnboundLoad and rearchitect strategy_options.py
The architecture of Load is mostly rewritten here.
The change includes removal of the "pluggable" aspect
of the loader options, which would patch new methods onto
Load. This has been replaced by normal methods that
respond normally to typing annotations. As part of this
change, the bake_loaders() and unbake_loaders() options,
which have no effect since 1.4 and were unlikely to be
in any common use, have been removed.
Additionally, to support annotations for methods that
make use of @decorator, @generative etc., modified
format_argspec_plus to no longer return "args", instead
returns "grouped_args" which is always grouped and
allows return annotations to format correctly.
Mike Bayer [Wed, 8 Dec 2021 13:57:44 +0000 (08:57 -0500)]
consider truediv as truediv; support floordiv operator
Implemented full support for "truediv" and "floordiv" using the
"/" and "//" operators. A "truediv" operation between two expressions
using :class:`_types.Integer` now considers the result to be
:class:`_types.Numeric`, and the dialect-level compilation will cast
the right operand to a numeric type on a dialect-specific basis to ensure
truediv is achieved. For floordiv, conversion is also added for those
databases that don't already do floordiv by default (MySQL, Oracle) and
the ``FLOOR()`` function is rendered in this case, as well as for
cases where the right operand is not an integer (needed for PostgreSQL,
others).
The change resolves issues both with inconsistent behavior of the
division operator on different backends and also fixes an issue where
integer division on Oracle would fail to be able to fetch a result due
to inappropriate outputtypehandlers.
Mike Bayer [Sun, 26 Dec 2021 17:13:19 +0000 (12:13 -0500)]
include empty intermediary tables in optimized get
Fixed issue in joined-inheritance load of additional attributes
functionality in deep multi-level inheritance where an intermediary table
that contained no columns would not be included in the tables joined,
instead linking those tables to their primary key identifiers. While this
works fine, it nonetheless in 1.4 began producing the cartesian product
compiler warning. The logic has been changed so that these intermediary
tables are included regardless. While this does include additional tables
in the query that are not technically necessary, this only occurs for the
highly unusual case of deep 3+ level inheritance with intermediary tables
that have no non primary key columns, potential performance impact is
therefore expected to be negligible.
Mike Bayer [Sun, 26 Dec 2021 16:25:00 +0000 (11:25 -0500)]
restore graceful degrade of subqueryload w from_statement
Fixed regression from 1.3 where the "subqueryload" loader strategy would
fail with a stack trace if used against a query that made use of
:meth:`_orm.Query.from_statement` or :meth:`_sql.Select.from_statement`. As
subqueryload requires modifying the original statement, it's not compatible
with the "from_statement" use case, especially for statements made against
the :func:`_sql.text` construct. The behavior now is equivalent to that of
1.3 and previously, which is that the loader strategy silently degrades to
not be used for such statements, typically falling back to using the
lazyload strategy.
Kai Mueller [Tue, 21 Dec 2021 20:00:30 +0000 (15:00 -0500)]
Fix missing class attributes when using __class_getitem__
Fixed issue where the ``__class_getitem__()`` method of the generated
declarative base class by :func:`_orm.as_declarative` would lead to
inaccessible class attributes such as ``__table__``, for cases where a
``Generic[T]`` style typing declaration were used in the class hierarchy.
This is in continuation from the basic addition of ``__class_getitem__()``
in :ticket:`7368`. Pull request courtesy Kai Mueller.
Mike Bayer [Wed, 22 Dec 2021 21:21:33 +0000 (16:21 -0500)]
use fully qualified, locatable names for all use of api.named_type()
Fixed mypy regression where the release of mypy 0.930 added additional
internal checks to the format of "named types", requiring that they be
fully qualified and locatable. This broke the mypy plugin for SQLAlchemy,
raising an assertion error, as there was use of symbols such as
``__builtins__`` and other un-locatable or unqualified names that
previously had not raised any assertions.
Mike Bayer [Wed, 22 Dec 2021 20:33:11 +0000 (15:33 -0500)]
add recursion check for with_loader_criteria() option
Fixed recursion overflow which could occur within ORM statement compilation
when using either the :func:`_orm.with_loader_criteria` feature or the the
:meth:`_orm.PropComparator.and_` method within a loader strategy in
conjunction with a subquery which referred to the same entity being altered
by the criteria option, or loaded by the loader strategy. A check for
coming across the same loader criteria option in a recursive fashion has
been added to accommodate for this scenario.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
