Mike Bayer [Thu, 7 Jan 2021 03:56:14 +0000 (22:56 -0500)]
Implement connection binding for AsyncSession
Implemented "connection-binding" for :class:`.AsyncSession`, the ability to
pass an :class:`.AsyncConnection` to create an :class:`.AsyncSession`.
Previously, this use case was not implemented and would use the associated
engine when the connection were passed. This fixes the issue where the
"join a session to an external transaction" use case would not work
correctly for the :class:`.AsyncSession`. Additionally, added methods
:meth:`.AsyncConnection.in_transaction`,
:meth:`.AsyncConnection.in_nested_transaction`,
:meth:`.AsyncConnection.get_transaction`.
The :class:`.AsyncEngine`, :class:`.AsyncConnection` and
:class:`.AsyncTransaction` objects may be compared using Python ``==`` or
``!=``, which will compare the two given objects based on the "sync" object
they are proxying towards. This is useful as there are cases particularly
for :class:`.AsyncTransaction` where multiple instances of
:class:`.AsyncTransaction` can be proxying towards the same sync
:class:`_engine.Transaction`, and are actually equivalent. The
:meth:`.AsyncConnection.get_transaction` method will currently return a new
proxying :class:`.AsyncTransaction` each time as the
:class:`.AsyncTransaction` is not otherwise statefully associated with its
originating :class:`.AsyncConnection`.
Mike Bayer [Thu, 7 Jan 2021 17:20:51 +0000 (12:20 -0500)]
Update connect args for pymysql 1.0.0; aiomysql fixes
Fixed deprecation warnings that arose as a result of the release of PyMySQL
1.0, including deprecation warnings for the "db" and "passwd" parameters
now replaced with "database" and "password".
For the 1.4 version of this patch, we are also changing tox.ini
to refer to a local branch of aiomysql that fixes pymysql
compatibility issues.
Mike Bayer [Wed, 6 Jan 2021 17:50:22 +0000 (12:50 -0500)]
repair ORM tests that use deferrable FKs
after the bound metadata merge the unitofwork.TransactionTest
became deprecations.AutocommitClosesOnFail and was found to
be not actually running correctly on SQLite. This test
should only run on a backend that has deferrable FKs,
which includes PostgreSQL; so add that to requirements. this
in turn requires clarification of test_naturalpks that the
"deferrable FKs" req there is more about Oracle and its lack
of ON UPDATE cascades.
Mike Bayer [Mon, 4 Jan 2021 20:18:25 +0000 (15:18 -0500)]
remove more bound metadata
in Iae6ab95938a7e92b6d42086aec534af27b5577d3 I missed
that the "bind" was being stuck onto the MetaData in
TablesTest, which led thousands of ORM tests to still use
bound metadata. Keep looking for bound metadata.
standardize all ORM tests on a single means of getting a
Session when the Session API isn't the thing we are directly
testing, using a new function fixture_session() that replaces
create_session() and uses modern defaults.
Mike Bayer [Tue, 5 Jan 2021 13:48:36 +0000 (08:48 -0500)]
Remove special rule for TypeDecorator of TypeDecorator
Removing this check for "TypeDecorator" in impl seems to not
break anything and allows TypeDecorator.with_variant() to
work correctly. The line has been traced back to 2007 and
does not appear to have relevance today.
Fixed bug where making use of the :meth:`.TypeEngine.with_variant` method
on a :class:`.TypeDecorator` type would fail to take into account the
dialect-specific mappings in use, due to a rule in :class:`.TypeDecorator`
that was instead attempting to check for chains of :class:`.TypeDecorator`
instances.
Mike Bayer [Mon, 4 Jan 2021 22:05:46 +0000 (17:05 -0500)]
Check for column expr in Oracle RETURNING check
Fixed regression in Oracle dialect introduced by :ticket:`4894` in
SQLAlchemy 1.3.11 where use of a SQL expression in RETURNING for an UPDATE
would fail to compile, due to a check for "server_default" when an
arbitrary SQL expression is not a column.
Mike Bayer [Mon, 21 Dec 2020 15:22:43 +0000 (10:22 -0500)]
remove metadata.bind use from test suite
importantly this means we can remove bound metadata from
the fixtures that are used by Alembic's test suite.
hopefully this is the last one that has to happen to allow
Alembic to be fully 1.4/2.0.
Start moving from @testing.provide_metadata to a pytest
metadata fixture. This does not seem to have any negative
effects even though TablesTest uses a "self.metadata" attribute.
Mike Bayer [Fri, 1 Jan 2021 18:21:55 +0000 (13:21 -0500)]
Cache asyngpg prepared statements
Enhanced the performance of the asyncpg dialect by caching the asyncpg
PreparedStatement objects on a per-connection basis. For a test case that
makes use of the same statement on a set of pooled connections this appears
to grant a 10-20% speed improvement. The cache size is adjustable and may
also be disabled.
Unfortunately the caching gets more complicated when there are
schema changes present. An invalidation scheme is now also added
to accommodate for prepared statements as well as asyncpg cached OIDs.
However, the exception raises cannot be prevented if DDL has changed
database structures that were cached for a particular asyncpg
connection. Logic is added to clear the caches when these errors occur.
Mike Bayer [Sat, 2 Jan 2021 15:55:21 +0000 (10:55 -0500)]
Repair async test refactor
in I4940d184a4dc790782fcddfb9873af3cca844398 we reworked how async
tests run but apparently the async tests in test/ext/asyncio
are reporting success without being run. This patch pushes
pytestplugin further so that it won't instrument any test
or function overall that declares itself async. This removes
the need for the __async_wrap__ flag and also allows us to
use a more strict "run_async_test" function that always
runs the asyncio event loop from the top.
Also start working asyncio into main testing suite.
Mike Bayer [Wed, 30 Dec 2020 18:56:20 +0000 (13:56 -0500)]
Support TypeDecorator.get_dbapi_type() for setinpusizes
Adjusted the "setinputsizes" logic relied upon by the cx_Oracle, asyncpg
and pg8000 dialects to support a :class:`.TypeDecorator` that includes
an override the :meth:`.TypeDecorator.get_dbapi_type()` method.
Mike Bayer [Wed, 30 Dec 2020 02:06:15 +0000 (21:06 -0500)]
disambiguate SQL server temp table constraint names
This seems to be raising errors lately which was not the
case earlier, however SQL Server seems to produce name conflicts
for named constraints against temp tables in different databases.
I can raise the error every time here running two tests
from ComponentReflectionTest with -n2. Unknown why the issue
is happening now and didn't occur for several months.
https://www.arbinada.com/en/node/1645 has some background.
Mike Bayer [Sat, 26 Dec 2020 16:46:42 +0000 (11:46 -0500)]
implement sessionmaker.begin(), scalar() for async session
Added :meth:`_asyncio.AsyncSession.scalar` as well as support for
:meth:`_orm.sessionmaker.begin` to work as an async context manager with
:class:`_asyncio.AsyncSession`. Also added
:meth:`_asyncio.AsyncSession.in_transaction` accessor.
Mike Bayer [Sat, 26 Dec 2020 16:16:51 +0000 (11:16 -0500)]
narrow the check for double-paren exprs in mysql create_index
Fixed regression from SQLAlchemy 1.3.20 caused by the fix for
:ticket:`5462` which adds double-parenthesis for MySQL functional
expressions in indexes, as is required by the backend, this inadvertently
extended to include arbitrary :func:`_sql.text` expressions as well as
Alembic's internal textual component, which are required by Alembic for
arbitrary index expressions which don't imply double parenthesis. The
check has been narrowed to include only binary/ unary/functional
expressions directly.
Mike Bayer [Wed, 23 Dec 2020 15:43:51 +0000 (10:43 -0500)]
Add ORMExecuteState mapper accessors
Added :attr:`_orm.ORMExecuteState.bind_mapper` and
:attr:`_orm.ORMExecuteState.all_mappers` accessors to
:class:`_orm.ORMExecuteState` event object, so that handlers can respond to
the target mapper and/or mapped class or classes involved in an ORM
statement execution.
Gord Thompson [Sun, 20 Dec 2020 17:20:10 +0000 (10:20 -0700)]
Fix issues with JSON and float/numeric
Decimal accuracy and behavior has been improved when extracting floating
point and/or decimal values from JSON strings using the
:meth:`_sql.sqltypes.JSON.Comparator.as_float` method, when the numeric
value inside of the JSON string has many significant digits; previously,
MySQL backends would truncate values with many significant digits and SQL
Server backends would raise an exception due to a DECIMAL cast with
insufficient significant digits. Both backends now use a FLOAT-compatible
approach that does not hardcode significant digits for floating point
values. For precision numerics, a new method
:meth:`_sql.sqltypes.JSON.Comparator.as_numeric` has been added which
accepts arguments for precision and scale, and will return values as Python
``Decimal`` objects with no floating point conversion assuming the DBAPI
supports it (all but pysqlite).
Mike Bayer [Thu, 3 Dec 2020 19:35:30 +0000 (14:35 -0500)]
Allow Declarative to extract class attr from field
Added an alternate resolution scheme to Declarative that will extract the
SQLAlchemy column or mapped property from the "metadata" dictionary of a
dataclasses.Field object. This allows full declarative mappings to be
combined with dataclass fields.
esoh [Sat, 19 Dec 2020 02:59:31 +0000 (21:59 -0500)]
Repair and cover adaption call w/ ORM having()
Fixed 1.4 regression where the use of :meth:`_orm.Query.having` in
conjunction with queries with internally adapted SQL elements (common in
inheritance scenarios) would fail due to an incorrect function call. Pull
request courtesy esoh.
Mike Bayer [Sat, 19 Dec 2020 15:59:42 +0000 (10:59 -0500)]
Repair mssql dep tests; have __only_on__ imply __backend__
CI missed a few SQL Server tests because we run mssql-backendonly
in the gerrit job. As there was a test that was "only on"
mssql but didn't have backendonly, it never got run and then
fails in master where we run mssql fully.
Any suite that has an __only_on__ is inherently specific to
a backend, so if present this should imply __backend__ so that
it definitely runs when we have that backend present.
This in turn meant we had to fix a few sqlite_file tests that
weren't cleaning up or sharing well as they suddenly became
backend tests under sqlite_file. Added a sqlite_file cleanup
to test class cleanup for now.
Mike Bayer [Fri, 18 Dec 2020 17:22:12 +0000 (12:22 -0500)]
Improve type detection for Values / Tuple
Fixed issue in new :class:`_sql.Values` construct where passing tuples of
objects would fall back to per-value type detection rather than making use
of the :class:`_schema.Column` objects passed directly to
:class:`_sql.Values` that tells SQLAlchemy what the expected type is. This
would lead to issues for objects such as enumerations and numpy strings
that are not actually necessary since the expected type is given.
note this changes NullType() to raise CompileError for
literal_processor; NullType() does not imply the actual value
NULL as much as it does "unknown type" so this should make failure
modes more clear.
Mike Bayer [Fri, 18 Dec 2020 14:28:06 +0000 (09:28 -0500)]
Gracefully degrade on v$transaction not readable
Fixed regression which occured due to [ticket:5755] which implemented
isolation level support for Oracle. It has been reported that many Oracle
accounts don't actually have permission to query the ``v$transaction``
view so this feature has been altered to gracefully fallback when it fails
upon database connect, where the dialect will assume "READ COMMITTED" is
the default isolation level as was the case prior to SQLAlchemy 1.3.21.
However, explicit use of the :meth:`_engine.Connection.get_isolation_level`
method must now necessarily raise an exception, as Oracle databases with
this restriction explicitly disallow the user from reading the current
isolation level.
Mike Bayer [Sat, 12 Dec 2020 23:56:58 +0000 (18:56 -0500)]
Major revisals to lambdas
1. Improve coercions._deep_is_literal to check sequences
for clause elements, thus allowing a phrase like
lambda: col.in_([literal("x"), literal("y")]) to be handled
2. revise closure variable caching completely. All variables
entering must be part of a closure cache key or rejected.
only objects that can be resolved to HasCacheKey or FunctionType
are accepted; all other types are rejected. This adds a high
degree of strictness to lambdas and will make them a little more
awkward to use in some cases, however prevents several classes
of critical issues:
a. previously, a lambda that had an expression derived from
some kind of state, like "self.x", or "execution_context.session.foo"
would produce a closure cache key from "self" or "execution_context",
objects that can very well be per-execution and would therefore
cause a AnalyzedFunction objects to overflow. (memory won't leak
as it looks like an LRUCache is already used for these)
b. a lambda, such as one used within DeferredLamdaElement, that
produces different SQL expressions based on the arguments
(which is in fact what it's supposed to do), however it would
through the use of conditionals produce different bound parameter
combinations, leading to literal parameters not tracked properly.
These are now rejected as uncacheable whereas previously they would
again be part of the closure cache key, causing an overflow of
AnalyizedFunction objects.
3. Ensure non-mapped mixins are handled correctly by
with_loader_criteria().
4. Fixed bug in lambda SQL system where we are not supposed to allow a Python
function to be embedded in the lambda, since we can't predict a bound value
from it. While there was an error condition added for this, it was not
tested and wasn't working; an informative error is now raised.
Michael Oliver [Wed, 16 Dec 2020 16:49:07 +0000 (11:49 -0500)]
Improved PEP-484 for AsyncSession.run_sync
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
I don't think this needs any test added.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Tue, 15 Dec 2020 14:45:48 +0000 (09:45 -0500)]
Check explicitly for mapped class as secondary
Added a comprehensive check and an informative error message for the case
where a mapped class, or a string mapped class name, is passed to
:paramref:`_orm.relationship.secondary`. This is an extremely common error
which warrants a clear message.
Additionally, added a new rule to the class registry resolution such that
with regards to the :paramref:`_orm.relationship.secondary` parameter, if a
mapped class and its table are of the identical string name, the
:class:`.Table` will be favored when resolving this parameter. In all
other cases, the class continues to be favored if a class and table
share the identical name.
Mike Bayer [Tue, 15 Dec 2020 13:52:15 +0000 (08:52 -0500)]
Use .expression accessor for hybrid example
hybrids since 1.1 use InstrumentedAttribute at the expression
level, so this doc needed to illustrate how to get at the
SQL expression. Also added a docstring for
QueryableAttribute.expression.
RamonWill [Wed, 21 Oct 2020 12:24:27 +0000 (08:24 -0400)]
Support IF EXISTS/IF NOT EXISTS for DDL constructs
Added parameters :paramref:`_ddl.CreateTable.if_not_exists`,
:paramref:`_ddl.CreateIndex.if_not_exists`,
:paramref:`_ddl.DropTable.if_exists` and
:paramref:`_ddl.DropIndex.if_exists` to the :class:`_ddl.CreateTable`,
:class:`_ddl.DropTable`, :class:`_ddl.CreateIndex` and
:class:`_ddl.DropIndex` constructs which result in "IF NOT EXISTS" / "IF
EXISTS" DDL being added to the CREATE/DROP. These phrases are not accepted
by all databases and the operation will fail on a database that does not
support it as there is no similarly compatible fallback within the scope of
a single DDL statement. Pull request courtesy Ramon Williams.
Gord Thompson [Sat, 12 Dec 2020 16:29:12 +0000 (09:29 -0700)]
Remove autoload=True references
Remove references to deprecated ``autoload=True`` in docs and code samples.
Also remove test/dialect/test_firebird.py and test/dialect/test_sybase.py
as those tests have already been incorporated into the external dialects.
Mike Bayer [Sat, 12 Dec 2020 00:01:12 +0000 (19:01 -0500)]
Fixes for lambda expressions and relationship loaders
Fixed bug in lambda SQL feature, used by ORM
:meth:`_orm.with_loader_criteria` as well as available generally in the SQL
expression language, where assigning a boolean value True/False to a
variable would cause the query-time expression calculation to fail, as it
would produce a SQL expression not compatible with a bound value.
Fixed issue where the :attr:`_orm.ORMExecuteState.is_relationship_load`
parameter would not be set correctly for many lazy loads, all
selectinloads, etc. The flag is essential in order to test if options
should be added to statements or if they would already have been propagated
via relationship loads.
Mike Bayer [Mon, 30 Nov 2020 17:50:51 +0000 (12:50 -0500)]
Send deterministic ordering into unit of work topological
Improved the unit of work topological sorting system such that the
toplogical sort is now deterministic based on the sorting of the input set,
which itself is now sorted at the level of mappers, so that the same inputs
of affected mappers should produce the same output every time, among
mappers / tables that don't have any dependency on each other. This further
reduces the chance of deadlocks as can be observed in a flush that UPDATEs
among multiple, unrelated tables such that row locks are generated.
topological.sort() has been made "deterministic" in all cases by
using a separate list + set.
Mike Bayer [Wed, 9 Dec 2020 16:39:45 +0000 (11:39 -0500)]
Emit deprecation warnings for plain text under Session
Deprecation warnings are emitted under "SQLALCHEMY_WARN_20" mode when
passing a plain string to :meth:`_orm.Session.execute`.
It was also considered to have DDL string expressions to include
this as well, however this leaves us with no backwards-compatible
way of handling reflection of elemens, such as an Index() which
reflects "postgresql_where='x > 5'", there's no place for a rule
that will turn those into text() within the reflection process
that would be separate from when the user passes postgresql_where
to the Index. Not worth it right now.
Mike Bayer [Sun, 15 Nov 2020 21:58:50 +0000 (16:58 -0500)]
correct for "autocommit" deprecation warning
Ensure no autocommit warnings occur internally or
within tests.
Also includes fixes for SQL Server full text tests
which apparently have not been working at all for a long
time, as it used long removed APIs. CI has not had
fulltext running for some years and is now installed.
Mike Bayer [Fri, 11 Dec 2020 15:34:44 +0000 (10:34 -0500)]
Revise attribute refresh for with_loader_criteria, related
Added new attribute :attr:`_orm.ORMExecuteState.is_column_load` to indicate
that a :meth:`_orm.SessionEvents.do_orm_execute` handler that a particular
operation is a primary-key-directed column attribute load, such as from an
expiration or a deferred attribute, and that WHERE criteria or additional
loader options should not be added to the query. This has been added to
the examples which illustrate the :func:`_orm.with_loader_criteria` option.
The :func:`_orm.with_loader_criteria` option has been modified so that it
will never apply its criteria to the SELECT statement for an ORM refresh
operation, such as that invoked by :meth:`_orm.Session.refresh` or whenever
an expired attribute is loaded. These queries are only against the
primary key row of the object that is already present in memory so there
should not be additional criteria added.
Added doc caveats for using lambdas.
Added test coverage for most ORMExecuteState flags and fixed a few
basic access issues.
Mike Bayer [Wed, 9 Dec 2020 03:07:48 +0000 (22:07 -0500)]
Implement Oracle SERIALIZABLE + real read of isolation level
There's some significant awkwardness in that we can't
read the level unless a transaction is started, which normally
does not occur unless DML is emitted. The implementation
uses the local_transaction_id function to start a transaction.
It is not known what the performance impact of this might
have, however by default the function is called only once
on first connect and later only if the get_isolation_level()
method is used.
The crashes that occur in jenkins have not been solved and are
now impacting master. I am not able to reproduce the failure,
including running on the CI machines directly, and a few runs
where I sat there for 20 minutes and watched, it didn't happen.
it is the ultimate heisenbug.
Additionally, there's a reference to "arraysize" that doesn't
exist in fetchmany() and there seem to be no tests that exercise
this for any DBAPI which is also a major bug to be fixed.
Gord Thompson [Mon, 7 Dec 2020 23:37:29 +0000 (18:37 -0500)]
Implement `TypeEngine.as_generic`
Added :meth:`_types.TypeEngine.as_generic` to map dialect-specific types,
such as :class:`sqlalchemy.dialects.mysql.INTEGER`, with the "best match"
generic SQLAlchemy type, in this case :class:`_types.Integer`. Pull
request courtesy Andrew Hannigan.
Abstract away how we check for "overridden methods" so it is more
clear what the intent is and that the methodology can be
independently tested.
The SQLAlchemy async mode now detects and raises an informative
error when an non asyncio compatible :term:`DBAPI` is used.
Using a standard ``DBAPI`` with async SQLAlchemy will cause
it to block like any sync call, interrupting the executing asyncio
loop.
Mike Bayer [Mon, 7 Dec 2020 18:40:48 +0000 (13:40 -0500)]
fix non-ascii quotes in sqlite on update feature
Fixed a SQLite source file that had non-ascii characters inside of its
docstring without a source encoding, introduced within the "INSERT..ON
CONFLICT" feature, which would cause failures under Python 2.
Mike Bayer [Thu, 3 Dec 2020 16:17:08 +0000 (11:17 -0500)]
Reflect decimal points in MariaDB non-quoted numeric defaults
Fixed issue where reflecting a server default on MariaDB only that
contained a decimal point in the value would fail to be reflected
correctly, leading towards a reflected table that lacked any server
default.