Mike Bayer [Mon, 24 Jan 2022 23:13:05 +0000 (18:13 -0500)]
restore set-as-superclass for OrderedSet
OrderedSet again subclasses set, spent some time
with the stubs at
https://github.com/python/typeshed/blob/master/stdlib/builtins.pyi#L887
to more deeply understand what they are doing here
so that we can type check fully.
Mike Bayer [Tue, 25 Jan 2022 16:01:25 +0000 (11:01 -0500)]
join to existing mark expr with "and"
ca48f461b2dcac2970829e4e0 considered an existing mark expression
plus legacy tags to be an error condition; however these can
be joined by "and" and will in our use case do the right thing.
The github action scripts make use of legacy tags. We can change
that also but I want to just make sure this combination works
fully as well.
Mike Bayer [Tue, 25 Jan 2022 05:45:30 +0000 (00:45 -0500)]
replace test tags with pytest.mark
replaced the __tags__ class attribute and the
--exclude-tags / --include-tags test runner options
with regular pytest.mark names
so that we can take advantage of mark expressions.
options --nomemory, --notimingintensive, --backend-only,
--exclude-tags, --include-tags remain as legacy but
make use of pytest mark for implemementation.
Added a "mypy" mark for the section of tests that are doing mypy
integration tests.
The __backend__ and __sparse_backend__ class attributes also
use pytest marks for their implementation, which also allows
the marks "backend" and "sparse_backend" to be used explicitly.
Also removed the no longer used "--cdecimal" option as this was
python 2 specific.
in theory, the usage of pytest marks could expand such that
the whole exclusions system would be based on it, but this
does not seem to have any advantage at the moment.
Mike Bayer [Sun, 9 Jan 2022 16:49:02 +0000 (11:49 -0500)]
mypy: sqlalchemy.util
Starting to set up practices and conventions to
get the library typed.
Key goals for typing are:
1. whole library can pass mypy without any strict
turned on.
2. we can incrementally turn on some strict flags on a per-package/
module basis, as here we turn on more strictness for sqlalchemy.util, exc,
and log
3. mypy ORM plugin tests work fully without sqlalchemy2-stubs
installed
4. public facing methods all have return types, major parameter
signatures filled in also
5. Foundational elements like util etc. are typed enough so that
we can use them in fully typed internals higher up the stack.
Conventions set up here:
1. we can use lots of config in setup.cfg to limit where mypy
is throwing errors and how detailed it should be in different
packages / modules. We can use this to push up gerrits
that will pass tests fully without everything being typed.
2. a new tox target pep484 is added. this links to a new jenkins
pep484 job that works across all projects (alembic, dogpile, etc.)
We've worked around some mypy bugs that will likely
be around for awhile, and also set up some core practices
for how to deal with certain things such as public_factory
modules (mypy won't accept a module from a callable at all,
so need to use simple type checking conditionals).
Mike Bayer [Sun, 23 Jan 2022 15:51:55 +0000 (10:51 -0500)]
after all that, use pytest warnings plugin
The warnings plugin lets us set the filters up
in the config, and as our filter requirements are now
simple we can just set this up.
additionally pytest now recommends pyproject.toml, since
we fully include this now, let's move it there.
the pytest logging plugin seems to not be any problem either
at the moment, so re-enable that. if it becomes apparent
whatever the problem was (which was probably that it was just
surprising, or something) we can disable it again and comment
what the reason was.
Mike Bayer [Sun, 23 Jan 2022 14:28:22 +0000 (09:28 -0500)]
dont test squelched warnings against the filter
I spent days on Ibcf09af25228d39ee5a943fda82d8a9302433726
reading it over and over again and noticed this slight inaccuracy
10 seconds after I merged it.
the assert_warns_message() and assert_warns() functions should
not consider a mismatched warning class as valid for a match.
Mike Bayer [Sun, 23 Jan 2022 14:14:00 +0000 (09:14 -0500)]
make the autoincrement doc a lot more clear
I saw it saying "this parameter has no effect when
Identity is in use", which is not accurate, so I've expanded
this way out with bullets to make the current situation
as clear as possible.
Mike Bayer [Fri, 21 Jan 2022 23:46:37 +0000 (18:46 -0500)]
dont use exception catches for warnings; modernize xdist detection
Improvements to the test suite's integration with pytest such that the
"warnings" plugin, if manually enabled, will not interfere with the test
suite, such that third parties can enable the warnings plugin or make use
of the ``-W`` parameter and SQLAlchemy's test suite will continue to pass.
Additionally, modernized the detection of the "pytest-xdist" plugin so that
plugins can be globally disabled using PYTEST_DISABLE_PLUGIN_AUTOLOAD=1
without breaking the test suite if xdist were still installed. Warning
filters that promote deprecation warnings to errors are now localized to
SQLAlchemy-specific warnings, or within SQLAlchemy-specific sources for
general Python deprecation warnings, so that non-SQLAlchemy deprecation
warnings emitted from pytest plugins should also not impact the test suite.
Mike Bayer [Fri, 21 Jan 2022 15:19:02 +0000 (10:19 -0500)]
Skip PK returned as None for RETURNING, server side default
Fixed regression where the ORM exception that is to be raised when an
INSERT silently fails to actually insert a row (such as from a trigger)
would not be reached, due to a runtime exception raised ahead of time due
to the missing primary key value, thus raising an uninformative exception
rather than the correct one. For 1.4 and above, a new ``FlushError`` is
added for this case that's raised earlier than the previous "null identity"
exception was for 1.3, as a situation where the number of rows actually
INSERTed does not match what was expected is a more critical situation in
1.4 as it prevents batching of multiple objects from working correctly.
This is separate from the case where a newly fetched primary key is
fetched as NULL, which continues to raise the existing "null identity"
exception.
Mike Bayer [Tue, 18 Jan 2022 22:00:16 +0000 (17:00 -0500)]
Add new infrastructure to support greater use of __slots__
* Changed AliasedInsp to use __slots__
* Migrated all of strategy_options to use __slots__ for objects.
Adds new infrastructure to traversals to support shallow
copy, to dict and from dict based on internal traversal
attributes. Load / _LoadElement then leverage this to
provide clone / generative / getstate without the need
for __dict__ or explicit attribute lists.
Doing this change revealed that there are lots of things that
trigger off of whether or not a class has a __visit_name__ attribute.
so to suit that we've gone back to having Visitable, which is
a better name than Traversible at this point (I think
Traversible is mis-spelled too).
Mike Bayer [Thu, 20 Jan 2022 20:21:17 +0000 (15:21 -0500)]
re-enable tests for asyncmy; fix Binary
Fixed regression in asyncmy dialect caused by :ticket:`7567` where removal
of the PyMySQL dependency broke binary columns, due to the asyncmy dialect
not being properly included within CI tests.
Also repairs mariadbconnector isolation level for 2.0.
basically tox config was failing to include additional
drivers.
Mike Bayer [Thu, 20 Jan 2022 20:01:47 +0000 (15:01 -0500)]
restore empty list logic to ARRAY of ENUM parsing
Fixed regression where the change in :ticket:`7148` to repair ENUM handling
in PostgreSQL broke the use case of an empty ARRAY of ENUM, preventing rows
that contained an empty array from being handled correctly when fetching
results.
Mike Bayer [Thu, 20 Jan 2022 14:31:42 +0000 (09:31 -0500)]
repair mapper sort
Fixed issue in :meth:`_orm.Session.bulk_save_mappings` where the sorting
that takes place when the ``preserve_order`` parameter is set to False
would sort partially on ``Mapper`` objects, which is rejected in Python
3.11.
Also uses typing_extensions for NotRequired as this symbol
does not seem to be in Python 3.11.0a4 yet.
For interim 3.11 support, adds the git main build of greenlet
for Python 3.11
Mike Bayer [Thu, 20 Jan 2022 17:26:36 +0000 (12:26 -0500)]
repair broken truediv test suite; memusage
the truediv test suite didn't have __backend__ so wasn't running
for every DB except in the main build. Repaired this as well
as truediv support to preserve the right-hand side type
when casting to numeric, if the right type is already a
numeric type.
also fixed a memusage test that relies on savepoints so was
not running under gerrit runs.
Mike Bayer [Wed, 19 Jan 2022 19:31:52 +0000 (14:31 -0500)]
Add AdaptedConnection.run_async
Added new method :meth:`.AdaptedConnection.run_async` to the DBAPI
connection interface used by asyncio drivers, which allows methods to be
called against the underlying "driver" connection directly within a
sync-style function where the ``await`` keyword can't be used, such as
within SQLAlchemy event handler functions. The method is analogous to the
:meth:`_asyncio.AsyncConnection.run_sync` method which translates
async-style calls to sync-style. The method is useful for things like
connection-pool on-connect handlers that need to invoke awaitable methods
on the driver connection when it's first created.
Mike Bayer [Tue, 18 Jan 2022 22:19:24 +0000 (17:19 -0500)]
reject methods as lambda SQL callables
Added an informative error message when a method object is passed to a SQL
construct. Previously, when such a callable were passed, as is a common
typographical error when dealing with method-chained SQL constructs, they
were interpreted as "lambda SQL" targets to be invoked at compilation time,
which would lead to silent failures. As this feature was not intended to be
used with methods, method objects are now rejected.
Mike Bayer [Sun, 16 Jan 2022 15:21:45 +0000 (10:21 -0500)]
enhance double-aliased table logic to handle more cases
Fixed ORM regression where calling the :func:`_orm.aliased` function
against an existing :func:`_orm.aliased` construct would fail to produce
correct SQL if the existing construct were against a fixed table. The fix
allows that the original :func:`_orm.aliased` construct is disregarded if
it were only against a table that's now being replaced. It also allows for
correct behavior when constructing a :func:`_orm.aliased` without a
selectable argument against a :func:`_orm.aliased` that's against a
subuquery, to create an alias of that subquery (i.e. to change its name).
The nesting behavior of :func:`_orm.aliased` remains in place for the case
where the outer :func:`_orm.aliased` object is against a subquery which in
turn refers to the inner :func:`_orm.aliased` object. This is a relatively
new 1.4 feature that helps to suit use cases that were previously served by
the deprecated ``Query.from_self()`` method.
Mike Bayer [Tue, 18 Jan 2022 16:02:57 +0000 (11:02 -0500)]
detect map_imperatively() called twice
Fixed issue where calling upon :meth:`_orm.regsitry.map_imperatively` more
than once for the same class would produce an unexpected error, rather than
an informative error that the target class is already mapped. This behavior
differed from that of the :func:`_orm.mapper` function which does report an
informative message already.
For 2.0, this change also cleans up the logic that detects against
`Mapper()` or `_mapper()` being invoked directly. 1.4's backport will
take on a different format as `mapper()` is still public API in that
release.
Mike Bayer [Fri, 14 Jan 2022 22:42:44 +0000 (17:42 -0500)]
remove dbapi_proxy module
This module was not documented nor part of any test suite,
and it's unlikely it was working correctly. It's not likely
that this module was ever used after the first year or so
of SQLAlchemy, and it's stayed around because it is so
obscure that I never remembered to remove it.
Mike Bayer [Fri, 17 Dec 2021 23:04:47 +0000 (18:04 -0500)]
track item schema names to identify name collisions w/ default schema
Added an additional lookup step to the compiler which will track all FROM
clauses which are tables, that may have the same name shared in multiple
schemas where one of the schemas is the implicit "default" schema; in this
case, the table name when referring to that name without a schema
qualification will be rendered with an anonymous alias name at the compiler
level in order to disambiguate the two (or more) names. The approach of
schema-qualifying the normally unqualified name with the server-detected
"default schema name" value was also considered, however this approach
doesn't apply to Oracle nor is it accepted by SQL Server, nor would it work
with multiple entries in the PostgreSQL search path. The name collision
issue resolved here has been identified as affecting at least Oracle,
PostgreSQL, SQL Server, MySQL and MariaDB.
José Duarte [Thu, 13 Jan 2022 22:20:06 +0000 (17:20 -0500)]
Fixes(#7561) Add support for postgres.UUID literal_binds compilation
Added string rendering to the :class:`.postgresql.UUID` datatype, so that
stringifying a statement with "literal_binds" that uses this type will
render an appropriate string value for the PostgreSQL backend. Pull request
courtesy José Duarte.
Mike Bayer [Sun, 9 Jan 2022 16:49:02 +0000 (11:49 -0500)]
Initial ORM typing layout
introduces:
1. new mapped_column() helper
2. DeclarativeBase helper
3. declared_attr has been re-typed
4. rework of Mapped[] to return InstrumentedAtribute for
class get, so works without Mapped itself having expression
methods
5. ORM constructs now generic on [_T]
also includes some early typing work, most of which will
be in later commits:
1. URL and History become typing.NamedTuple
2. come up with type-checking friendly way of type
checking cy extensions, where type checking will be applied
to the py versions, just needed to come up with a succinct
conditional pattern for the imports
Mike Bayer [Sun, 2 Jan 2022 22:35:43 +0000 (17:35 -0500)]
initial reorganize for static typing
start applying foundational annotations to key
elements.
two main elements addressed here:
1. removal of public_factory() and replacement with
explicit functions. this just works much better with
typing.
2. typing support for column expressions and operators.
The biggest part of this involves stubbing out all the
ColumnOperators methods under ColumnElement in a
TYPE_CHECKING section. Took me a while to see this
method vs. much more complicated things I thought
I needed.
Also for this version implementing #7519, ColumnElement
types against the Python type and not TypeEngine. it is
hoped this leads to easier transferrence between ORM/Core
as well as eventual support for result set typing.
Not clear yet how well this approach will work and what
new issues it may introduce.
given the current approach we now get full, rich typing for
scenarios like this:
from sqlalchemy import column, Integer, String, Boolean
Mike Bayer [Mon, 10 Jan 2022 21:48:05 +0000 (16:48 -0500)]
remove internal use of metaclasses
All but one metaclass used internally can now
be replaced using __init_subclass__(). Within this
patch we remove:
* events._EventMeta
* sql.visitors.TraversibleType
* sql.visitors.InternalTraversibleType
* testing.fixtures.FindFixture
* testing.fixtures.FindFixtureDeclarative
* langhelpers.EnsureKWArgType
* sql.functions._GenericMeta
* sql.type_api.VisitableCheckKWArg (was a mixture of TraversibleType
and EnsureKWArgType)
The remaining internal class is MetaOptions used by the
sql.Options object which is in turn currently mostly for
ORM internal use, as this type implements class level overrides
for the ``+`` operator.
For declarative, removing DeclarativeMeta in place of
an `__init_subclass__()` class would not be fully feasible as
it would break backwards compatibility with applications that
refer to this class explicitly, but also DeclarativeMeta intercepts
class-level attribute set and delete operations which is a widely
used pattern. An option for declarative base to use
`__init_subclass__()` should be provided but this is out of
scope for this particular change.
the description_encoding concept has been fully
removed in 2.0, but this logic was plainly broken
in any case. The same non-working logic exists in
1.4 as well, where description_encoding as a concept
is not removed, though it is likely that there are no
real-world use cases for it.
Mike Bayer [Tue, 11 Jan 2022 03:11:50 +0000 (22:11 -0500)]
change state.load_options to a tuple
having this be an immutable sequence is safer
and possibly lower overhead.
The change here went in with no issues save
for tests that asserted it was a set.
InstanceState.load_options is only referred
towards when the object is first loaded, and then
within the logic that emits an object refresh
as well as within a lazy loader. it's only
accessed as a whole collection.
Mike Bayer [Wed, 5 Jan 2022 17:20:46 +0000 (12:20 -0500)]
implement second-level type resolution for literals
Added additional rule to the system that determines ``TypeEngine``
implementations from Python literals to apply a second level of adjustment
to the type, so that a Python datetime with or without tzinfo can set the
``timezone=True`` parameter on the returned :class:`.DateTime` object, as
well as :class:`.Time`. This helps with some round-trip scenarios on
type-sensitive PostgreSQL dialects such as asyncpg, psycopg3 (2.0 only).
For 1.4 specifically, the backport improves support for asyncpg handling of
TIME WITH TIMEZONE, which was not fully implemented. 2.0's reworked
PostgreSQL architecture had this handled already.
Mike Bayer [Mon, 10 Jan 2022 19:59:59 +0000 (14:59 -0500)]
ensure with_options not switched to a list
Fixed regression which appeared in 1.4.23 which could cause loader options
to be mis-handled in some cases, in particular when using joined table
inheritance in combination with the ``polymorphic_load="selectin"`` option
as well as relationship lazy loading, leading to a ``TypeError``.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
There's a few bits and pieces of code to support Python <= 3.6 which are no longer needed and can be removed, to slightly simplify the codebase.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Mon, 3 Jan 2022 18:49:26 +0000 (13:49 -0500)]
Remove all remaining removed_in_20 warnings slated for removal
Finalize all remaining removed-in-2.0 changes so that we
can begin doing pep-484 typing without old things
getting in the way (we will also have to do public_factory).
note there are a few "moved_in_20()" and "became_legacy_in_20()"
warnings still in place. The SQLALCHEMY_WARN_20 variable
is now removed.
Also removed here are the legacy "in place mutators" for Select
statements, and some keyword-only argument signatures in Core
have been added.
Also in the big change department, the ORM mapper() function
is removed entirely; the Mapper class is otherwise unchanged,
just the public-facing API function. Mappers are now always
given a registry in which to participate, however the
argument signature of Mapper is not changed. ideally "registry"
would be the first positional argument.
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Black's `target-version` was still set to `['py27', 'py36']`. Set it to `[py37]` instead.
Also update Black and other pre-commit hooks and re-format with Black.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Tue, 4 Jan 2022 19:04:15 +0000 (14:04 -0500)]
implement python_impl to custom_op for basic ORM evaluator extensibility
Added new parameter :paramref:`_sql.Operators.op.python_impl`, available
from :meth:`_sql.Operators.op` and also when using the
:class:`_sql.Operators.custom_op` constructor directly, which allows an
in-Python evaluation function to be provided along with the custom SQL
operator. This evaluation function becomes the implementation used when the
operator object is used given plain Python objects as operands on both
sides, and in particular is compatible with the
``synchronize_session='evaluate'`` option used with
:ref:`orm_expression_update_delete`.
Mike Bayer [Tue, 4 Jan 2022 19:08:10 +0000 (14:08 -0500)]
improve custom operator for SQL types docs
introduce here that "custom ops" first come from
the .op() method in the usual case. then only if one wants
such an op to be pre-assocaited with particular types,
then the comparator may be used. Also clarify the individual
points regarding the comparator.
Mike Bayer [Mon, 3 Jan 2022 22:28:52 +0000 (17:28 -0500)]
ensure correlate_except is checked for empty tuple
Fixed issue where :meth:`_sql.Select.correlate_except` method, when passed
either the ``None`` value or no arguments, would not correlate any elements
when used in an ORM context (that is, passing ORM entities as FROM
clauses), rather than causing all FROM elements to be considered as
"correlated" in the same way which occurs when using Core-only constructs.
Federico Caselli [Thu, 30 Dec 2021 22:29:55 +0000 (23:29 +0100)]
Imrpove MySQL/MariaDB dialect initialization.
Replace ``SHOW VARIABLES LIKE`` statement with equivalent
``SELECT @@variable`` in MySQL and MariaDB dialect initialization.
This should avoid mutex contention caused by ``SHOW VARIABLES``,
improving initialization performance.
Mike Bayer [Sat, 1 Jan 2022 21:50:41 +0000 (16:50 -0500)]
adjust TypeError message yet again
in 924d38e660981ae2a7d55355603 we got it almost right in
all cases, Pypy seems to have an entirely different kind of message
so make this much more open ended.
Mike Bayer [Sat, 1 Jan 2022 20:30:15 +0000 (15:30 -0500)]
adjust error message assertion for TypeError
in eee7a3add99df2865e6d907f2 I added a few TypeError
tests, and some python interpreters produce a message like
"join() got an unexpected...", others seem to produce
"Query.join() got an unexpected..."
the pattern is strange, all Python 3.10's seem to return the
class name. For Python 3.8 and 3.9, my local intepreters
are doing the Python 3.10 behavior but github actions are doing
the older behavior. Python 3.7 both here and on github actions
are doing the older behavior.
Federico Caselli [Wed, 22 Dec 2021 20:45:45 +0000 (21:45 +0100)]
Properly type _generative, decorator, public_factory
Good new is that pylance likes it and copies over the
singature and everything.
Bad news is that mypy does not support this yet https://github.com/python/mypy/issues/8645
Other minor bad news is that non_generative is not typed. I've tried using a protocol
like the one in the comment but the signature is not ported over by pylance, so it's
probably best to just live without it to have the correct signature.
notes from mike: these three decorators are at the core of getting
the library to be typed, more good news is that pylance will
do all the things we like re: public_factory, see
https://github.com/microsoft/pyright/issues/2758#issuecomment-1002788656
.
For @_generative, we will likely move to using pep 673 once mypy
supports it which may be soon. but overall having the explicit
"return self" in the methods, while a little inconvenient, makes
the typing more straightforward and locally present in the files
rather than being decided at a distance. having "return self"
present, or not, both have problems, so maybe we will be able
to change it again if things change as far as decorator support.
As it is, I feel like we are barely squeaking by with our decorators,
the typing is already pretty out there.
Mike Bayer [Sat, 23 Oct 2021 15:26:45 +0000 (11:26 -0400)]
replace Variant with direct feature inside of TypeEngine
The :meth:`_sqltypes.TypeEngine.with_variant` method now returns a copy of
the original :class:`_sqltypes.TypeEngine` object, rather than wrapping it
inside the ``Variant`` class, which is effectively removed (the import
symbol remains for backwards compatibility with code that may be testing
for this symbol). While the previous approach maintained in-Python
behaviors, maintaining the original type allows for clearer type checking
and debugging.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
