Re-implement c version immutabledict / processors / resultproxy / utils with cython.
Performance is in general in par or better than the c version
Added a collection module that has cython version of OrderedSet and IdentitySet
Added a new test/perf file to compare the implementations.
Run ``python test/perf/compiled_extensions.py all`` to execute the comparison test.
See results here: https://docs.google.com/document/d/1nOcDGojHRtXEkuy4vNXcW_XOJd9gqKhSeALGG3kYr6A/edit?usp=sharing
Mike Bayer [Tue, 14 Dec 2021 21:46:50 +0000 (16:46 -0500)]
include InterfaceError for mariadb disconnect check
Corrected the error classes inspected for the "is_disconnect" check for the
``mariadbconnector`` dialect, which was failing for disconnects that
occurred due to common MySQL/MariaDB error codes such as 2006; the DBAPI
appears to currently use the ``mariadb.InterfaceError`` exception class for
disconnect errors such as error code 2006, which has been added to the list
of classes checked.
For the current "real reconnect test", shutting down the mariadb
connection from the client side produces
ProgrammingError("Connection isn't valid anymore") which we also
continue to intercept.
Nils Philippsen [Sun, 12 Dec 2021 23:35:03 +0000 (18:35 -0500)]
Ignore ephemeral classes in test_all_present()
Fixed a regression in the test suite where the test called
``CompareAndCopyTest::test_all_present`` would fail on some platforms due
to additional testing artifacts being detected. Pull request courtesy Nils
Philippsen.
In some circumstances, ephemeral class objects that are created within
the scope of a test method don't seem to be garbage collected directly
on exit. Filter out classes created in test modules.
Mike Bayer [Sun, 12 Dec 2021 18:37:21 +0000 (13:37 -0500)]
use the options from the cached statement for propagate_options
Fixed caching-related issue where the use of a loader option of the form
``lazyload(aliased(A).bs).joinedload(B.cs)`` would fail to result in the
joinedload being invoked for runs subsequent to the query being cached, due
to a mismatch for the options / object path applied to the objects loaded
for a query with a lead entity that used ``aliased()``.
Nils Philippsen [Sat, 13 Nov 2021 16:11:32 +0000 (11:11 -0500)]
Add async_engine_from_config()
Added :func:`_asyncio.async_engine_config` function to create
an async engine from a configuration dict. This otherwise
behaves the same as :func:`_sa.engine_from_config`.
Mike Bayer [Thu, 9 Dec 2021 19:23:42 +0000 (14:23 -0500)]
implement correct errors for Row immutability
Corrected the error message for the ``AttributeError`` that's raised when
attempting to write to an attribute on the :class:`_result.Row` class,
which is immutable. The previous message claimed the column didn't exist
which is misleading.
Mike Bayer [Thu, 9 Dec 2021 17:51:43 +0000 (12:51 -0500)]
implement attributes.Proxy._clone()
Fixed issue where the internal cloning used by the
:meth:`_orm.PropComparator.any` method on a :func:`_orm.relationship` in
the case where the related class also makes use of ORM polymorphic loading,
would fail if a hybrid property on the related, polymorphic class were used
within the criteria for the ``any()`` operation.
Mike Bayer [Wed, 8 Dec 2021 19:19:11 +0000 (14:19 -0500)]
Removals: strings for join(), loader_options().
* The :meth:`_orm.Query.join` method no longer accepts strings for
relationship names; the long-documented approach of using
``Class.attrname`` for join targets is now standard.
* Loader options no longer accept strings for attribute names. The
long-documented approach of using ``Class.attrname`` for loader option
targets is now standard.
It is hoped that a subsequent commit can refactor loader
options to no longer need "UnboundLoad" for most cases.
Mike Bayer [Wed, 8 Dec 2021 18:55:32 +0000 (13:55 -0500)]
include correct notes for 2.0
the changelog_20 file either started out wrong
or got changed at some point to include the wrong change notes,
so the unreleased_20/ notes were not being displayed.
Mike Bayer [Fri, 3 Dec 2021 19:04:05 +0000 (14:04 -0500)]
Warn when caching is disabled / document
This patch adds new warnings for all elements that
don't indicate their caching behavior, including user-defined
ClauseElement subclasses and third party dialects.
it additionally adds new documentation to discuss an apparent
performance degradation in 1.4 when caching is disabled as a
result in the significant expense incurred by ORM
lazy loaders, which in 1.3 used BakedQuery so were actually
cached.
As a result of adding the warnings, a fair degree of
lesser used SQL expression objects identified that they did not
define caching behavior so would have been producing
``[no key]``, including PostgreSQL constructs ``hstore``
and ``array``. These have been amended to use inherit
cache where appropriate. "on conflict" constructs in
PostgreSQL, MySQL, SQLite still explicitly don't generate
a cache key at this time.
The change also adds a test for all constructs via
assert_compile() to assert they will not generate cache
warnings.
Mike Bayer [Thu, 2 Dec 2021 00:27:25 +0000 (19:27 -0500)]
contextmanager skips rollback if trans says to skip it
Fixed issue where if an exception occurred when the :class:`_orm.Session`
were to close the connection within the :meth:`_orm.Session.commit` method,
when using a context manager for :meth:`_orm.Session.begin` , it would
attempt a rollback which would not be possible as the :class:`_orm.Session`
was in between where the transaction is committed and the connection is
then to be returned to the pool, raising the exception "this
sessiontransaction is in the committed state". This exception can occur
mostly in an asyncio context where CancelledError can be raised.
The where method of exists now accepts multiple cluase.
Support multiple clause elements in the :meth:`_sql.Exists.where` method,
unifying the api with the on presented by a normal :func:`_sql.select`
construct.
Mike Bayer [Thu, 2 Dec 2021 02:39:59 +0000 (21:39 -0500)]
copy list for __iadd__
Fixed issue where a list mapped with :func:`_orm.relationship` would go
into an endless loop if in-place added to itself, i.e. the ``+=`` operator
were used, as well as if ``.extend()`` were given the same list.
Kai Mueller [Wed, 1 Dec 2021 15:58:40 +0000 (10:58 -0500)]
Add __class_getitem__ to the declarative Base class
Fixed issue where the :func:`_orm.as_declarative` decorator and similar
functions used to generate the declarative base class would not copy the
``__class_getitem__()`` method from a given superclass, which prevented the
use of pep-484 generics in conjunction with the ``Base`` class. Pull
request courtesy Kai Mueller.
Federico Caselli [Sat, 27 Nov 2021 08:53:29 +0000 (09:53 +0100)]
adapt pytest plugin to support pytest v7
Implemented support for the test suite to run correctly under Pytest 7.
Previously, only Pytest 6.x was supported for Python 3, however the version
was not pinned on the upper bound in tox.ini. Pytest is not pinned in
tox.ini to be lower than version 8 so that SQLAlchemy versions released
with the current codebase will be able to be tested under tox without
changes to the environment. Much thanks to the Pytest developers for
their help with this issue.
Mike Bayer [Fri, 26 Nov 2021 15:17:38 +0000 (10:17 -0500)]
provide connectionfairy on initialize
This is so that dialect methods that are called within init
can assume the same argument structure as when they are called
in other places; we can nail down the type of object as well.
This change seems to mostly impact the isolation level routines
in the dialects, as these are called during initialize()
as well as on established connections. these methods can now
assume a non-proxied DBAPI connection object in all cases,
as it is commonly required that attributes like ".autocommit"
are set on the object which don't work well in a proxied
situation.
Other changes:
* adds an interface for the "connectionfairy" concept
called PoolProxiedConnection.
* Removes ``Connectable`` superclass of Connection.
``Connectable`` was originally meant to provide for the
"method which accepts connection or engine" theme. As this
pattern is greatly reduced in 2.0 and Engine no longer extends
from it, the ``Connectable`` superclass doesnt serve any real
purpose.
Leading from that, to set this in I also applied pep 484 annotations
to the Dialect base, and then in the interests of seeing some
of the typing information show up in my IDE did a little bit for Engine,
Connection and others. I hope that it's feasible that we can
add annotations to specific classes and attributes ahead of when we
actually try to mass-populate the whole library. This was
the original spirit of pep-484 that we can apply annotations
gradually. I do of course want to try to do a mass-populate
although i think even in that case we will end up doing a lot
of manual work anyway (in particular for the changes here which
are distinct from what the stubs have).
Mike Bayer [Fri, 26 Nov 2021 23:43:23 +0000 (18:43 -0500)]
"graceful fetch" test should....clean up gracefully!
this test was relying on gc to close out the connection.
this would lead to problems with aiosqlite as we invalidate
async connetions that aren't gracefully closed, and this test
suite was create tables per suite, so we'd get into a spot where
a new sqlite memory connection without the tables would get set up.
would only occur for full test run + -n2 + C extensions + python 3.7,
but we assume that is all related to just getting gc to trigger
or not trigger at exactly the right moment in this situation.
confirmed if we add a gc.collect() to the test without explcitly
closing out the conenction, the connection is gc'ed and detached,
and we get the error reproduced on the subsequent test.
Mike Bayer [Mon, 22 Nov 2021 19:28:26 +0000 (14:28 -0500)]
propose emulated setinputsizes embedded in the compiler
Add a new system so that PostgreSQL and other dialects have a
reliable way to add casts to bound parameters in SQL statements,
replacing previous use of setinputsizes() for PG dialects.
rationale:
1. psycopg3 will be using the same SQLAlchemy-side "setinputsizes"
as asyncpg, so we will be seeing a lot more of this
2. the full rendering that SQLAlchemy's compilation is performing
is in the engine log as well as error messages. Without this,
we introduce three levels of SQL rendering, the compiler, the
hidden "setinputsizes" in SQLAlchemy, and then whatever the DBAPI
driver does. With this new approach, users reporting bugs etc.
will be less confused that there are as many as two separate
layers of "hidden rendering"; SQLAlchemy's rendering is again
fully transparent
3. calling upon a setinputsizes() method for every statement execution
is expensive. this way, the work is done behind the caching layer
4. for "fast insertmany()", I also want there to be a fast approach
towards setinputsizes. As it was, we were going to be taking
a SQL INSERT with thousands of bound parameter placeholders and
running a whole second pass on it to apply typecasts. this way,
we will at least be able to build the SQL string once without a huge
second pass over the whole string
5. psycopg2 can use this same system for its ARRAY casts
6. the general need for PostgreSQL to have lots of type casts
is now mostly in the base PostgreSQL dialect and works independently
of a DBAPI being present. dependence on DBAPI symbols that aren't
complete / consistent / hashable is removed
I was originally going to try to build this into bind_expression(),
but it was revealed this worked poorly with custom bind_expression()
as well as empty sets. the current impl also doesn't need to
run a second expression pass over the POSTCOMPILE sections, which
came out better than I originally thought it would.
Mike Bayer [Mon, 22 Nov 2021 15:59:06 +0000 (10:59 -0500)]
Support lightweight compiler column elements w/ slots
the _CompileLabel class included ``__slots__`` but these
weren't used as the superclasses included slots.
Create a ``__slots__`` superclass for ``ClauseElement``,
creating a new class of compilable SQL elements that don't
include heavier features like caching, annotations and
cloning, which are meant to be used only in an ad-hoc
compiler fashion. Create new ``CompilerColumnElement``
from that which serves in column-oriented contexts, but
similarly does not include any expression operator support
as it is intended to be used only to generate a string.
Apply this to both
``_CompileLabel`` as well as PostgreSQL ``_ColonCast``,
which does not actually subclass ``ColumnElement`` as this
class has memoized attributes that aren't worth changing,
and does not include SQL operator capabilities as these
are not needed for these compiler-only objects.
this allows us to more inexpensively add new ad-hoc
labels / casts etc. at compile time, as we will be seeking
to expand out the typecasts that are needed for PostgreSQL
dialects in a subsequent patch.
Mike Bayer [Sun, 21 Nov 2021 17:58:11 +0000 (12:58 -0500)]
provide space for new 2.0 changes not part of upgrade path
As we are going to be adding new improvements such as
variant, cython, psycopg3, more RETURNING support,
fast execution helpers, and probably a lot more, the
2.0 migration doc needs to also still have the "what's new in 2.0?"
thing going on as well. Organize the document so it has space
to introduce these sections, as well as the existing
1.4->2.0 migration sections.
If the "What's New" document gets large, we will break up this
page into two separate pages, it's already quite large, so
the proposal would be the exising migration doc becomes
a separate special migration document.
Mike Bayer [Thu, 18 Nov 2021 18:39:54 +0000 (13:39 -0500)]
use typing.Dict
newer Pythons seem to accept ``dict[Any, Any]`` which is why
this wasn't noticed. Revise fix for #7321 made in
I55656e867876677c5c55143449db371344be8600.
Mike Bayer [Sat, 6 Nov 2021 17:00:43 +0000 (13:00 -0400)]
fully support isolation_level parameter in base dialect
Generalized the :paramref:`_sa.create_engine.isolation_level` parameter to
the base dialect so that it is no longer dependent on individual dialects
to be present. This parameter sets up the "isolation level" setting to
occur for all new database connections as soon as they are created by the
connection pool, where the value then stays set without being reset on
every checkin.
The :paramref:`_sa.create_engine.isolation_level` parameter is essentially
equivalent in functionality to using the
:paramref:`_engine.Engine.execution_options.isolation_level` parameter via
:meth:`_engine.Engine.execution_options` for an engine-wide setting. The
difference is in that the former setting assigns the isolation level just
once when a connection is created, the latter sets and resets the given
level on each connection checkout.
Mike Bayer [Thu, 18 Nov 2021 17:46:25 +0000 (12:46 -0500)]
disable raiseerr for refresh state loader options
Fixed ORM regression where the new behavior of "eager loaders run on
unexpire" added in :ticket:`1763` would lead to loader option errors being
raised inappropriately for the case where a single :class:`_orm.Query` or
:class:`_sql.Select` were used to load multiple kinds of entities, along
with loader options that apply to just one of those kinds of entity like a
:func:`_orm.joinedload`, and later the objects would be refreshed from
expiration, where the loader options would attempt to be applied to the
mismatched object type and then raise an exception. The check for this
mismatch now bypasses raising an error for this case.
Mike Bayer [Mon, 8 Nov 2021 17:20:23 +0000 (12:20 -0500)]
remove "native decimal" warning
Removed the warning that emits from the :class:`_types.Numeric` type about
DBAPIs not supporting Decimal values natively. This warning was oriented
towards SQLite, which does not have any real way without additional
extensions or workarounds of handling precision numeric values more than 15
significant digits as it only uses floating point math to represent
numbers. As this is a known and documented limitation in SQLite itself, and
not a quirk of the pysqlite driver, there's no need for SQLAlchemy to warn
for this. The change does not otherwise modify how precision numerics are
handled. Values can continue to be handled as ``Decimal()`` or ``float()``
as configured with the :class:`_types.Numeric`, :class:`_types.Float` , and
related datatypes, just without the ability to maintain precision beyond 15
significant digits when using SQLite, unless alternate representations such
as strings are used.
Mike Bayer [Mon, 15 Nov 2021 20:06:06 +0000 (15:06 -0500)]
favor setuptools imports over distutils
Python 3.10 has deprecated "distutils" in favor of explicit use of
"setuptools" in :pep:`632`; SQLAlchemy's setup.py has replaced imports
accordingly. However, since setuptools itself only recently added the
replacement symbols mentioned in pep-632 as of November of 2022 in version
59.0.1, ``setup.py`` still has fallback imports to distutils, as SQLAlchemy
1.4 does not have a hard setuptools versioning requirement at this time.
SQLAlchemy 2.0 is expected to use a full :pep:`517` installation layout
which will indicate appropriate setuptools versioning up front.
Mike Bayer [Mon, 15 Nov 2021 01:02:10 +0000 (20:02 -0500)]
handle dunder names in @declared_attr separately
Fixed Mypy crash which would occur when using Mypy plugin against code
which made use of :class:`_orm.declared_attr` methods for non-mapped names
like ``__mapper_args__``, ``__table_args__``, or other dunder names, as the
plugin would try to interpret these as mapped attributes which would then
be later mis-handled. As part of this change, the decorated function is
still converted by the plugin into a generic assignment statement (e.g.
``__mapper_args__: Any``) so that the argument signature can continue to be
annotated in the same way one would for any other ``@classmethod`` without
Mypy complaining about the wrong argument type for a method that isn't
explicitly ``@classmethod``.
jonathan vanasco [Mon, 27 Sep 2021 16:41:24 +0000 (12:41 -0400)]
Add new sections regarding schemas and reflection
* add a new section to reflection.rst `Schemas and Reflection`.
* this contains some text from the ticket
* migrate some text from `Specifying the Schema Name` to new section
* migrate some text from PostgreSQL dialect to new section
* target text is made more generic
* cross-reference the postgres and new sections to one another, to avoid duplication of docs
* update some docs 'meta' to 'metadata_obj'
Fixes: #4387 Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com>
Change-Id: I2b08672753fb2575d30ada07ead77587468fdade
Mike Bayer [Wed, 17 Nov 2021 20:08:29 +0000 (15:08 -0500)]
generalize cache_ok to UserDefinedType
Extended the ``cache_ok`` flag and corresponding warning message if this
flag is not defined, a behavior first established for
:class:`.TypeDecorator` as part of :ticket:`6436`, to also take place for
:class:`.UserDefinedType`, by generalizing the flag and associated caching
logic to a new common base for these two types, :class:`.ExternalType`.
The change means any current :class:`.UserDefinedType` will now cause SQL
statement caching to no longer take place for statements which make use of
the datatype, along with a warning being emitted, unless the class defines
the :attr:`.UserDefinedType.cache_ok` flag as True. If the datatype cannot
form a deterministic, hashable cache key derived from its arguments, it may return
False which will continue to keep caching disabled but will suppress the
warning. In particular, custom datatypes currently used in packages such as
SQLAlchemy-utils will need to implement this flag. The issue was observed
as a result of a SQLAlchemy-utils datatype that is not currently cacheable.
Eric Masseran [Tue, 2 Nov 2021 20:40:04 +0000 (16:40 -0400)]
Add Non linear CTE support
"Compound select" methods like :meth:`_sql.Select.union`,
:meth:`_sql.Select.intersect_all` etc. now accept ``*other`` as an argument
rather than ``other`` to allow for multiple additional SELECTs to be
compounded with the parent statement at once. In particular, the change as
applied to :meth:`_sql.CTE.union` and :meth:`_sql.CTE.union_all` now allow
for a so-called "non-linear CTE" to be created with the :class:`_sql.CTE`
construct, whereas previously there was no way to have more than two CTE
sub-elements in a UNION together while still correctly calling upon the CTE
in recursive fashion. Pull request courtesy Eric Masseran.
Allow:
```sql
WITH RECURSIVE nodes(x) AS (
SELECT 59
UNION
SELECT aa FROM edge JOIN nodes ON bb=x
UNION
SELECT bb FROM edge JOIN nodes ON aa=x
)
SELECT x FROM nodes;
```
Based on @zzzeek suggestion: https://github.com/sqlalchemy/sqlalchemy/pull/7133#issuecomment-933882348
Mike Bayer [Sun, 7 Nov 2021 20:47:15 +0000 (15:47 -0500)]
removals: all unicode encoding / decoding
Removed here includes:
* convert_unicode parameters
* encoding create_engine() parameter
* description encoding support
* "non-unicode fallback" modes under Python 2
* String symbols regarding Python 2 non-unicode fallbacks
* any concept of DBAPIs that don't accept unicode
statements, unicode bound parameters, or that return bytes
for strings anywhere except an explicit Binary / BLOB
type
* unicode processors in Python / C
Risk factors:
* Whether all DBAPIs do in fact return Unicode objects for
all entries in cursor.description now
* There was logic for mysql-connector trying to determine
description encoding. A quick test shows Unicode coming
back but it's not clear if there are still edge cases where
they return bytes. if so, these are bugs in that driver,
and at most we would only work around it in the mysql-connector
DBAPI itself (but we won't do that either).
* It seems like Oracle 8 was not expecting unicode bound parameters.
I'm assuming this was all Python 2 stuff and does not apply
for modern cx_Oracle under Python 3.
* third party dialects relying upon built in unicode encoding/decoding
but it's hard to imagine any non-SQLAlchemy database driver not
dealing exclusively in Python unicode strings in Python 3
Mike Bayer [Wed, 10 Nov 2021 14:52:18 +0000 (09:52 -0500)]
qualify asyncpg API tests for python 3.8
Getting
TypeError: object MagicMock can't be used in 'await' expression
for Python 3.7 and earlier. this test is not needed
on all platforms it's confirming that two methods
are present.
Mike Bayer [Mon, 8 Nov 2021 23:29:16 +0000 (18:29 -0500)]
change the POSTCOMPILE/ SCHEMA symbols to not conflict w mssql quoting
Adjusted the compiler's generation of "post compile" symbols including
those used for "expanding IN" as well as for the "schema translate map" to
not be based directly on plain bracketed strings with underscores, as this
conflicts directly with SQL Server's quoting format of also using brackets,
which produces false matches when the compiler replaces "post compile" and
"schema translate" symbols. The issue created easy to reproduce examples
both with the :meth:`.Inspector.get_schema_names` method when used in
conjunction with the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature, as well in the unlikely case that a symbol overlapping with the
internal name "POSTCOMPILE" would be used with a feature like "expanding
in".
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
