Mike Bayer [Sat, 12 Dec 2020 23:56:58 +0000 (18:56 -0500)]
Major revisals to lambdas
1. Improve coercions._deep_is_literal to check sequences
for clause elements, thus allowing a phrase like
lambda: col.in_([literal("x"), literal("y")]) to be handled
2. revise closure variable caching completely. All variables
entering must be part of a closure cache key or rejected.
only objects that can be resolved to HasCacheKey or FunctionType
are accepted; all other types are rejected. This adds a high
degree of strictness to lambdas and will make them a little more
awkward to use in some cases, however prevents several classes
of critical issues:
a. previously, a lambda that had an expression derived from
some kind of state, like "self.x", or "execution_context.session.foo"
would produce a closure cache key from "self" or "execution_context",
objects that can very well be per-execution and would therefore
cause a AnalyzedFunction objects to overflow. (memory won't leak
as it looks like an LRUCache is already used for these)
b. a lambda, such as one used within DeferredLamdaElement, that
produces different SQL expressions based on the arguments
(which is in fact what it's supposed to do), however it would
through the use of conditionals produce different bound parameter
combinations, leading to literal parameters not tracked properly.
These are now rejected as uncacheable whereas previously they would
again be part of the closure cache key, causing an overflow of
AnalyizedFunction objects.
3. Ensure non-mapped mixins are handled correctly by
with_loader_criteria().
4. Fixed bug in lambda SQL system where we are not supposed to allow a Python
function to be embedded in the lambda, since we can't predict a bound value
from it. While there was an error condition added for this, it was not
tested and wasn't working; an informative error is now raised.
Mike Bayer [Tue, 15 Dec 2020 14:45:48 +0000 (09:45 -0500)]
Check explicitly for mapped class as secondary
Added a comprehensive check and an informative error message for the case
where a mapped class, or a string mapped class name, is passed to
:paramref:`_orm.relationship.secondary`. This is an extremely common error
which warrants a clear message.
Additionally, added a new rule to the class registry resolution such that
with regards to the :paramref:`_orm.relationship.secondary` parameter, if a
mapped class and its table are of the identical string name, the
:class:`.Table` will be favored when resolving this parameter. In all
other cases, the class continues to be favored if a class and table
share the identical name.
Mike Bayer [Tue, 15 Dec 2020 13:52:15 +0000 (08:52 -0500)]
Use .expression accessor for hybrid example
hybrids since 1.1 use InstrumentedAttribute at the expression
level, so this doc needed to illustrate how to get at the
SQL expression. Also added a docstring for
QueryableAttribute.expression.
RamonWill [Wed, 21 Oct 2020 12:24:27 +0000 (08:24 -0400)]
Support IF EXISTS/IF NOT EXISTS for DDL constructs
Added parameters :paramref:`_ddl.CreateTable.if_not_exists`,
:paramref:`_ddl.CreateIndex.if_not_exists`,
:paramref:`_ddl.DropTable.if_exists` and
:paramref:`_ddl.DropIndex.if_exists` to the :class:`_ddl.CreateTable`,
:class:`_ddl.DropTable`, :class:`_ddl.CreateIndex` and
:class:`_ddl.DropIndex` constructs which result in "IF NOT EXISTS" / "IF
EXISTS" DDL being added to the CREATE/DROP. These phrases are not accepted
by all databases and the operation will fail on a database that does not
support it as there is no similarly compatible fallback within the scope of
a single DDL statement. Pull request courtesy Ramon Williams.
Gord Thompson [Sat, 12 Dec 2020 16:29:12 +0000 (09:29 -0700)]
Remove autoload=True references
Remove references to deprecated ``autoload=True`` in docs and code samples.
Also remove test/dialect/test_firebird.py and test/dialect/test_sybase.py
as those tests have already been incorporated into the external dialects.
Mike Bayer [Sat, 12 Dec 2020 00:01:12 +0000 (19:01 -0500)]
Fixes for lambda expressions and relationship loaders
Fixed bug in lambda SQL feature, used by ORM
:meth:`_orm.with_loader_criteria` as well as available generally in the SQL
expression language, where assigning a boolean value True/False to a
variable would cause the query-time expression calculation to fail, as it
would produce a SQL expression not compatible with a bound value.
Fixed issue where the :attr:`_orm.ORMExecuteState.is_relationship_load`
parameter would not be set correctly for many lazy loads, all
selectinloads, etc. The flag is essential in order to test if options
should be added to statements or if they would already have been propagated
via relationship loads.
Mike Bayer [Mon, 30 Nov 2020 17:50:51 +0000 (12:50 -0500)]
Send deterministic ordering into unit of work topological
Improved the unit of work topological sorting system such that the
toplogical sort is now deterministic based on the sorting of the input set,
which itself is now sorted at the level of mappers, so that the same inputs
of affected mappers should produce the same output every time, among
mappers / tables that don't have any dependency on each other. This further
reduces the chance of deadlocks as can be observed in a flush that UPDATEs
among multiple, unrelated tables such that row locks are generated.
topological.sort() has been made "deterministic" in all cases by
using a separate list + set.
Mike Bayer [Wed, 9 Dec 2020 16:39:45 +0000 (11:39 -0500)]
Emit deprecation warnings for plain text under Session
Deprecation warnings are emitted under "SQLALCHEMY_WARN_20" mode when
passing a plain string to :meth:`_orm.Session.execute`.
It was also considered to have DDL string expressions to include
this as well, however this leaves us with no backwards-compatible
way of handling reflection of elemens, such as an Index() which
reflects "postgresql_where='x > 5'", there's no place for a rule
that will turn those into text() within the reflection process
that would be separate from when the user passes postgresql_where
to the Index. Not worth it right now.
Mike Bayer [Sun, 15 Nov 2020 21:58:50 +0000 (16:58 -0500)]
correct for "autocommit" deprecation warning
Ensure no autocommit warnings occur internally or
within tests.
Also includes fixes for SQL Server full text tests
which apparently have not been working at all for a long
time, as it used long removed APIs. CI has not had
fulltext running for some years and is now installed.
Mike Bayer [Fri, 11 Dec 2020 15:34:44 +0000 (10:34 -0500)]
Revise attribute refresh for with_loader_criteria, related
Added new attribute :attr:`_orm.ORMExecuteState.is_column_load` to indicate
that a :meth:`_orm.SessionEvents.do_orm_execute` handler that a particular
operation is a primary-key-directed column attribute load, such as from an
expiration or a deferred attribute, and that WHERE criteria or additional
loader options should not be added to the query. This has been added to
the examples which illustrate the :func:`_orm.with_loader_criteria` option.
The :func:`_orm.with_loader_criteria` option has been modified so that it
will never apply its criteria to the SELECT statement for an ORM refresh
operation, such as that invoked by :meth:`_orm.Session.refresh` or whenever
an expired attribute is loaded. These queries are only against the
primary key row of the object that is already present in memory so there
should not be additional criteria added.
Added doc caveats for using lambdas.
Added test coverage for most ORMExecuteState flags and fixed a few
basic access issues.
Mike Bayer [Wed, 9 Dec 2020 03:07:48 +0000 (22:07 -0500)]
Implement Oracle SERIALIZABLE + real read of isolation level
There's some significant awkwardness in that we can't
read the level unless a transaction is started, which normally
does not occur unless DML is emitted. The implementation
uses the local_transaction_id function to start a transaction.
It is not known what the performance impact of this might
have, however by default the function is called only once
on first connect and later only if the get_isolation_level()
method is used.
The crashes that occur in jenkins have not been solved and are
now impacting master. I am not able to reproduce the failure,
including running on the CI machines directly, and a few runs
where I sat there for 20 minutes and watched, it didn't happen.
it is the ultimate heisenbug.
Additionally, there's a reference to "arraysize" that doesn't
exist in fetchmany() and there seem to be no tests that exercise
this for any DBAPI which is also a major bug to be fixed.
Gord Thompson [Mon, 7 Dec 2020 23:37:29 +0000 (18:37 -0500)]
Implement `TypeEngine.as_generic`
Added :meth:`_types.TypeEngine.as_generic` to map dialect-specific types,
such as :class:`sqlalchemy.dialects.mysql.INTEGER`, with the "best match"
generic SQLAlchemy type, in this case :class:`_types.Integer`. Pull
request courtesy Andrew Hannigan.
Abstract away how we check for "overridden methods" so it is more
clear what the intent is and that the methodology can be
independently tested.
The SQLAlchemy async mode now detects and raises an informative
error when an non asyncio compatible :term:`DBAPI` is used.
Using a standard ``DBAPI`` with async SQLAlchemy will cause
it to block like any sync call, interrupting the executing asyncio
loop.
Mike Bayer [Mon, 7 Dec 2020 18:40:48 +0000 (13:40 -0500)]
fix non-ascii quotes in sqlite on update feature
Fixed a SQLite source file that had non-ascii characters inside of its
docstring without a source encoding, introduced within the "INSERT..ON
CONFLICT" feature, which would cause failures under Python 2.
Mike Bayer [Thu, 3 Dec 2020 16:17:08 +0000 (11:17 -0500)]
Reflect decimal points in MariaDB non-quoted numeric defaults
Fixed issue where reflecting a server default on MariaDB only that
contained a decimal point in the value would fail to be reflected
correctly, leading towards a reflected table that lacked any server
default.
Mike Bayer [Wed, 2 Dec 2020 14:26:18 +0000 (09:26 -0500)]
modernize contains_eager() docs
Along with other loader options that are likely
to require usage of populate_existing(), make sure
contains_eager() documentation makes it absolutely
clear that already-loaded collections are not overwritten.
consolidate contains_eager() documentation into the narrative docs.
Additionally, remove the "arbitrary statements" section, this is
not a useful case and seems to be left over.
Mike Bayer [Wed, 2 Dec 2020 01:55:06 +0000 (20:55 -0500)]
Make sure asyncpgfallback has __slots__
the test suite failed to find that we started accessing
a non-existent slot "_isolation_setting" added by me in 9b779611f9, as the test suite makes use of the
AsyncAdaptFallback_asyncpg_connection subclass, which didn't
include __slots__ and therefore didn't catch that
_isolation_setting wasn't added to slots.
Mike Bayer [Tue, 1 Dec 2020 21:03:35 +0000 (16:03 -0500)]
Pass along other keyword args in _EventsHold.populate
Fixed bug involving the ``restore_load_context`` option of ORM events such
as :meth:`_orm.InstanceEvents.load` such that the flag would not be carried
along to subclasses which were mapped after the event handler were first
established.
Mike Bayer [Thu, 26 Nov 2020 17:10:25 +0000 (12:10 -0500)]
Don't discard leftovers from surface_selectables
Fixed regression introduced in 1.3.2 for the PostgreSQL dialect, also
copied out to the MySQL dialect's feature in 1.3.18, where usage of a non
:class:`_schema.Table` construct such as :func:`_sql.text` as the argument
to :paramref:`_sql.Select.with_for_update.of` would fail to be accommodated
correctly within the PostgreSQL or MySQL compilers.
Brendan McCollam [Thu, 26 Nov 2020 16:04:42 +0000 (11:04 -0500)]
Fix typo in sessionmaker example: s/wih/with
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
Fix typo in sessionmaker documentation example: s/wih/with
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Tue, 24 Nov 2020 01:13:20 +0000 (20:13 -0500)]
Support Column objects in the SET clause for upsert
Established support for :class:`_schema.Column` objects as well as ORM
instrumented attributes as keys in the ``set_`` dictionary passed to the
:meth:`_postgresql.Insert.on_conflict_do_update` and
:meth:`_sqlite.Insert.on_conflict_do_update` methods, which match to the
:class:`_schema.Column` objects in the ``.c`` collection of the target
:class:`_schema.Table`. Previously, only string column names were
expected; a column expression would be assumed to be an out-of-table
expression that would render fully along with a warning.
Federico Caselli [Tue, 10 Nov 2020 22:13:55 +0000 (23:13 +0100)]
Add summary table for orm 2.0 migration
A new formatting option is added to zzzeeksphinx
to allow the table to slide for horizontal overflow
without blowing up the page. overall though trying
to keep the table narrow as the code examples are side by
side, other configurations didn't work as well.
Jordan Pittier [Wed, 18 Nov 2020 14:57:43 +0000 (09:57 -0500)]
QueuePool: support subsecond timeout
Fixes: #5719
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
Make it explicit in the documentation and in the default value for the 'timeout'
parameter that `timeout` can be a float. Because Python timing is not
very accurate, warn about the precision.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Fri, 20 Nov 2020 15:12:18 +0000 (10:12 -0500)]
Don't emit warnings on descriptor access
This commit is revising 5162f2bc5fc0ac239f26a76fc9f0c2, which
when I did it felt a little rushed but I couldn't find anything
wrong. Well here we are :).
Fixed issue where a :class:`.RemovedIn20Warning` would erroneously emit
when the ``.bind`` attribute were accessed internally on objects,
particularly when stringifying a SQL construct.
Alter the deprecated() decorator so that we can use it just to add
docstring warnings but not actually warn when the function is
accessed, adding new argument enable_warnings that can be
set to False.
Added a safety feature to deprecated_20() that will disallow an
":attr:" from proceeding if enable_warnings=False isn't present,
unless there's an extra flag
warn_on_attribute_access, since we want Session.transaction to
emit a deprecation warning. This is a little hacky but it's essentially
modifying the decorator to require a positive assertion that
a deprecation decorator on a descriptor should actually warn
on access.
Remove the warning filter for session.transaction and get
tests to pass to ensure this is not also being called
internally.
Added tests to ensure that common places .bind can be passed
as a parameter definitely warn as I was not able to find
this otherwise.
Fantix King [Thu, 19 Nov 2020 16:53:50 +0000 (11:53 -0500)]
Support PEP-567 context variables
Invoke the given function within a copy of the current PEP-567
context in `greenlet_spawn()`, so that subsequent sync methods could
retrieve the correct context variable.
Adjusted the greenlet integration, which provides support for Python asyncio
in SQLAlchemy, to accommodate for the handling of Python ``contextvars``
(introduced in Python 3.7) for ``greenlet`` versions greater than 0.4.17.
Greenlet version 0.4.17 added automatic handling of contextvars in a
backwards-incompatible way; we've coordinated with the greenlet authors to
add a preferred API for this in versions subsequent to 0.4.17 which is now
supported by SQLAlchemy's greenlet integration. For greenlet versions prior
to 0.4.17 no behavioral change is needed, version 0.4.17 itself is blocked
from the dependencies.
Federico Caselli [Sun, 15 Nov 2020 15:58:46 +0000 (16:58 +0100)]
Use ``re.search`` instead of ``re.match`` in sqlite
Use python ``re.search()`` instead of ``re.match()`` as the operation
used by the :meth:`Column.regexp_match` method when using sqlite.
This matches the behavior of regular expressions on other databases
as well as that of well-known SQLite plugins.
Mike Bayer [Tue, 17 Nov 2020 22:13:24 +0000 (17:13 -0500)]
Support pool.connect() event firing before all else
Fixed regression where a connection pool event specified with a keyword,
most notably ``insert=True``, would be lost when the event were set up.
This would prevent startup events that need to fire before dialect-level
events from working correctly.
The internal mechanics of the engine connection routine has been altered
such that it's now guaranteed that a user-defined event handler for the
:meth:`_pool.PoolEvents.connect` handler, when established using
``insert=True``, will allow an event handler to run that is definitely
invoked **before** any dialect-specific initialization starts up, most
notably when it does things like detect default schema name.
Previously, this would occur in most cases but not unconditionally.
A new example is added to the schema documentation illustrating how to
establish the "default schema name" within an on-connect event
(upcoming as part of I882edd5bbe06ee5b4d0a9c148854a57b2bcd4741)
Addiional changes to support setting default schema name:
The Oracle dialect now uses
``select sys_context( 'userenv', 'current_schema' ) from dual`` to get
the default schema name, rather than ``SELECT USER FROM DUAL``, to
accommodate for changes to the session-local schema name under Oracle.
Added a read/write ``.autocommit`` attribute to the DBAPI-adaptation layer
for the asyncpg dialect. This so that when working with DBAPI-specific
schemes that need to use "autocommit" directly with the DBAPI connection,
the same ``.autocommit`` attribute which works with both psycopg2 as well
as pg8000 is available.
Mike Bayer [Wed, 18 Nov 2020 14:57:30 +0000 (09:57 -0500)]
Allow MetaData as the target for column_reflect event
The :meth:`_event.DDLEvents.column_reflect` event may now be applied to a
:class:`_schema.MetaData` object where it will take effect for the
:class:`_schema.Table` objects local to that collection.
AlonM [Sun, 15 Nov 2020 13:13:25 +0000 (08:13 -0500)]
Add opsclass to exclusion constraint
Added new parameter :paramref:`_postgresql.ExcludeConstraint.ops` to the
:class:`_postgresql.ExcludeConstraint` object, to support operator class
specification with this constraint. Pull request courtesy Alon Menczer.
Mike Bayer [Mon, 16 Nov 2020 15:15:17 +0000 (10:15 -0500)]
Ensure the "orm" plugin is used unconditionally for bundles
This also introduces that the "orm" plugin may be used
when the plugin_subject is None.
Fixed regression where the :paramref:`.Bundle.single_entity` flag would
take effect for a :class:`.Bundle` even though it were not set.
Additionally, this flag is legacy as it only makes sense for the
:class:`_orm.Query` object and not 2.0 style execution. a deprecation
warning is emitted when used with new-style execution.
Mike Bayer [Sat, 14 Nov 2020 00:35:25 +0000 (19:35 -0500)]
don't check standard_conforming_strings less than server version 8.2
Fixed a small regression where the query for "show
standard_conforming_strings" upon initialization would be emitted even if
the server version info were detected as less than version 8.2, previously
it would only occur for server version 8.2 or greater. The query fails on
Amazon Redshift which reports a PG server version older than this value.
I approved this merge before observing that the "ssl_mode"
flag is not actually part of the separate "ssl" dictionary,
so this patch as given is unnecessary as we already pass
all strings into opts. but also I can't get the flag to work
on either mysqlclient or pymysql, it's not recognized. so
I need specific info from the submitter what it is that is
actually needed, and if a change is needed it has to be
appropriate for the drivers in use.
Gord Thompson [Fri, 13 Nov 2020 19:13:32 +0000 (14:13 -0500)]
Added ssl_mode flag to mysqldb
Fixes: #5692
ssl_mode flag is added to mysqldb
### Description
mysqldb driver supports "ssl_mode" flag, which can be one of following values:
"DISABLED", "PREFERRED", "REQUIRED", "VERIFY_CA", "VERIFY_IDENTITY".
Depending on these values MySQL will behave accordingly to them.
So this flag has been added to the sqlaclhemy. So in case if TLS is not supported on the server, we can drop the connection right away.
Oscar Batori [Wed, 16 Sep 2020 17:41:34 +0000 (13:41 -0400)]
Added missing keywords for MySQL dialect
Fixes: #5696
Add missing MySQL keywords.
### Description
Adds some missing keywords from the MySQL dialect. While it's hard to describe this as a documentation/typographical fix, there are no current test for presence of the correct keywords, nor a realistic way to implement them those tests.
### Checklist
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Wed, 11 Nov 2020 16:13:27 +0000 (11:13 -0500)]
Warn / raise for returning() / return_defaults() combinations
A warning is emmitted if a returning() method such as
:meth:`_sql.Insert.returning` is called multiple times, as this does not
yet support additive operation. Version 1.4 will support additive
operation for this. Additionally, any combination of the
:meth:`_sql.Insert.returning` and :meth:`_sql.Insert.return_defaults`
methods now raises an error as these methods are mutually exclusive;
previously the operation would fail silently.
Thomas Grainger [Tue, 10 Nov 2020 10:49:59 +0000 (05:49 -0500)]
s/craete/create/g
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
it's a typo fix
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
