Mike Bayer [Sun, 14 Oct 2018 19:55:46 +0000 (15:55 -0400)]
Don't call rollback on DBAPI connection that's "closed"
Use the existence of ConnectionRecord.connection to estimate
that this connection is likely closed, and if so, don't
try to call "rollback" on it. This rollback is normally harmless
but is causing segfaults in mysqlclient due to
https://github.com/PyMySQL/mysqlclient-python/issues/270.
Jayson Reis [Mon, 1 Oct 2018 16:58:46 +0000 (12:58 -0400)]
selectinload omit join
The "selectin" loader strategy now omits the JOIN in the case of a
simple one-to-many load, where it instead relies upon the foreign key
columns of the related table in order to match up to primary keys in
the parent table. This optimization can be disabled by setting
the :paramref:`.relationship.omit_join` flag to False.
Many thanks to Jayson Reis for the efforts on this.
As part of this change, horizontal shard no longer relies upon
the _mapper_zero() method to get the query-bound mapper, instead
using the more generalized _bind_mapper() (which will use mapper_zero
if no explicit FROM is present). A short check for the particular
recursive condition is added to BundleEntity and it no longer assigns
itself as the "namespace" to its ColumnEntity objects which creates
a reference cycle.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #4340
Change-Id: I649587e1c07b684ecd63f7d10054cd165891baf4
Pull-request: https://bitbucket.org/zzzeek/sqlalchemy/pull-requests/7
Mike Bayer [Wed, 7 Feb 2018 00:30:55 +0000 (19:30 -0500)]
Route bulk update/delete exec through new Query._execute_crud method
Added support for bulk :meth:`.Query.update` and :meth:`.Query.delete`
to the :class:`.ShardedQuery` class within the horiziontal sharding
extension. This also adds an additional expansion hook to the
bulk update/delete methods :meth:`.Query._execute_crud`.
Mike Bayer [Wed, 3 Oct 2018 14:40:38 +0000 (10:40 -0400)]
Support tuples of heterogeneous types for empty expanding IN
Pass a list of all the types for the left side of an
IN expression to the visit_empty_set_expr() method, so that
the "empty expanding IN" can produce clauses for each element.
Mike Bayer [Tue, 2 Oct 2018 21:49:44 +0000 (17:49 -0400)]
Perform additional retrieval of correct column names
Added a workaround for a MySQL bug #88718 introduced in the 8.0 series,
where the reflection of a foreign key constraint is not reporting the
correct case sensitivity for the referred column, leading to errors during
use of the reflected constraint such as when using the automap extension.
The workaround emits an additional query to the information_schema tables in
order to retrieve the correct case sensitive name.
Joe Urciuoli [Wed, 19 Sep 2018 17:40:23 +0000 (13:40 -0400)]
Fix dependency_for final argument
Fixed issue where part of the utility language helper internals was passing
the wrong kind of argument to the Python ``__import__`` builtin as the list
of modules to be imported. The issue produced no symptoms within the core
library but could cause issues with external applications that redefine the
``__import__`` builtin or otherwise instrument it. Pull request courtesy Joe
Urciuoli.
Per the submitter: "The fourth argument provided to `__import__` (which
`import_` feeds in to) is supposed to be a a list of strings, but this code is
passing a single string. This was causing the sqlalchemy `import_` function to
break the string (for example 'interfaces') into an array of single characters
['i', 'n', ...], which causes the actual `__import__` to not find the module
`sqlalchemy.orm.i` (since it's trying to import `sqlalchemy.orm.i` and
`sqlalchemy.orm.n` .. etc)"
No issue could be reproduced locally as it seems you can put anything non-
empty/None into that last argument, even a list like ``['X']``, and all the
sub-modules seem to appear. Omit it, and then the sub-modules aren't present.
Perhaps it just runs the module or not if this attribute is present.
The long-standing behavior of the association proxy collection maintaining
only a weak reference to the parent object is reverted; the proxy will now
maintain a strong reference to the parent for as long as the proxy
collection itself is also in memory, eliminating the "stale association
proxy" error. This change is being made on an experimental basis to see if
any use cases arise where it causes side effects.
Mike Bayer [Mon, 1 Oct 2018 20:23:33 +0000 (16:23 -0400)]
Add reflection support for Postgresql partitioned tables
Added rudimental support for reflection of Postgresql
partitioned tables, e.g. that relkind='p' is added to reflection
queries that return table information.
Mike Bayer [Tue, 27 Feb 2018 18:15:10 +0000 (13:15 -0500)]
Break association proxy into a descriptor + per-class accessor
Reworked :class:`.AssociationProxy` to store state that's specific to a
parent class in a separate object, so that a single
:class:`.AssocationProxy` can serve for multiple parent classes, as is
intrinsic to inheritance, without any ambiguity in the state returned by it.
A new method :meth:`.AssociationProxy.for_class` is added to allow
inspection of class-specific state.
Fixed additional warnings generated by Python 3.7 due to changes in the
organization of the Python ``collections`` and ``collections.abc`` packages.
Previous ``collections`` warnings were fixed in version 1.2.11. Pull request
courtesy xtreak.
See I2d1c0ef97c8ecac7af152cc56263422a40faa6bb for the original collections.abc
fixes.
Removed the collection converter used by the :class:`.MappedCollection`
class. This converter was used only to assert that the incoming dictionary
keys matched that of their corresponding objects, and only during a bulk set
operation. The converter can interfere with a custom validator or
:meth:`.AttributeEvents.bulk_replace` listener that wants to convert
incoming values further. The ``TypeError`` which would be raised by this
converter when an incoming key didn't match the value is removed; incoming
values during a bulk assignment will be keyed to their value-generated key,
and not the key that's explicitly present in the dictionary.
Overall, @converter is superseded by the
:meth:`.AttributeEvents.bulk_replace` event handler added as part of
:ticket:`3896`.
Mike Bayer [Tue, 25 Sep 2018 14:38:40 +0000 (10:38 -0400)]
Copy create_constraint flag for Enum
Fixed bug where the :paramref:`.Enum.create_constraint` flag on the
:class:`.Enum` datatype would not be propagated to copies of the type, which
affects use cases such as declarative mixins and abstract bases.
Mike Bayer [Wed, 13 Jun 2018 21:48:51 +0000 (17:48 -0400)]
Add use_nchar_for_unicode flag; don't use nchar types for generic unicode
The Oracle dialect will no longer use the NCHAR/NCLOB datatypes to
represent generic unicode strings or clob fields in conjunction with
:class:`.Unicode` and :class:`.UnicodeText` unless the flag
``use_nchar_for_unicode=True`` is passed to :func:`.create_engine`.
Additionally, string types under Oracle now coerce to unicode under Python
2 in all cases, however unlike in previous iterations, we use SQLAlchemy's
native unicode handlers which are very performant (when C extensions are
enabled; when they are not, we use cx_Oracle's handlers).
Samuel Chou [Wed, 19 Sep 2018 17:30:24 +0000 (13:30 -0400)]
Allow dialects to customize group by clause compilation
Refactored :class:`.SQLCompiler` to expose a
:meth:`.SQLCompiler.group_by_clause` method similar to the
:meth:`.SQLCompiler.order_by_clause` and :meth:`.SQLCompiler.limit_clause`
methods, which can be overridden by dialects to customize how GROUP BY
renders. Pull request courtesy Samuel Chou.
Taem Park [Tue, 21 Aug 2018 14:54:45 +0000 (10:54 -0400)]
Add LIFO for connection pooling
Added new "lifo" mode to :class:`.QueuePool`, typically enabled by setting
the flag :paramref:`.create_engine.pool_use_lifo` to True. "lifo" mode
means the same connection just checked in will be the first to be checked
out again, allowing excess connections to be cleaned up from the server
side during periods of the pool being only partially utilized. Pull request
courtesy Taem Park.
Mike Bayer [Mon, 17 Sep 2018 15:38:52 +0000 (11:38 -0400)]
Adapt right side in join if lateral detected
Fixed bug where use of :class:`.Lateral` construct in conjunction with
:meth:`.Query.join` as well as :meth:`.Query.select_entity_from` would not
apply clause adaption to the right side of the join. "lateral" introduces
the use case of the right side of a join being correlatable. Previously,
adaptation of this clause wasn't considered.
Added a check within the weakref cleanup for the :class:`.InstanceState`
object to check for the presence of the ``dict`` builtin, in an effort to
reduce error messages generated when these cleanups occur during interpreter
shutdown. Pull request courtesy Romuald Brunet.
Mike Bayer [Mon, 17 Sep 2018 18:05:46 +0000 (14:05 -0400)]
Use cx_Oracle dml_ret_array_val
Fixed issue for cx_Oracle 7.0 where the behavior of Oracle param.getvalue()
now returns a list, rather than a single scalar value, breaking
autoincrement logic throughout the Core and ORM. The dml_ret_array_val
compatibility flag is used for cx_Oracle 6.3 and 6.4 to establish compatible
behavior with 7.0 and forward, for cx_Oracle 6.2.1 and prior a version
number check falls back to the old logic.
Mike Bayer [Thu, 6 Sep 2018 14:44:09 +0000 (10:44 -0400)]
Clarify init_scalar event use case
Since I didn't even realize what this was for when reading the docs,
make it clearer that this is to mirror a Column default and remove
the extra verbiage about the mechanics of INSERTs.
Mike Bayer [Thu, 30 Aug 2018 21:12:58 +0000 (17:12 -0400)]
MariaDB 10.3 updates
MariaDB seems to handle some additional UPDATE/DELETE FROM
syntaxes as well as some forms of INTERSECT and EXCEPT. Open
up tests that expect failure for MySQL to allow success for
MariaDB 10.3.
Alessandro Cucci [Sat, 25 Aug 2018 13:14:22 +0000 (09:14 -0400)]
Add option to sort into inserts/updates to bulk_save_objects
Added new flag :paramref:`.Session.bulk_save_objects.preserve_order` to the
:meth:`.Session.bulk_save_objects` method, which defaults to True. When set
to False, the given mappings will be grouped into inserts and updates per
each object type, to allow for greater opportunities to batch common
operations together. Pull request courtesy Alessandro Cucci.
Mike Bayer [Mon, 27 Aug 2018 15:07:48 +0000 (11:07 -0400)]
Add BakedQuery.to_query() method
Added new feature :meth:`.BakedQuery.to_query`, which allows for a
clean way of using one :class:`.BakedQuery` as a subquery inside of another
:class:`.BakedQuery` without needing to refer explicitly to a
:class:`.Session`.
Mike Bayer [Sun, 26 Aug 2018 16:35:59 +0000 (12:35 -0400)]
Include Session._query_cls as part of the cache key
Fixed issue where :class:`.BakedQuery` did not include the specific query
class used by the :class:`.Session` as part of the cache key, leading to
incompatibilities when using custom query classes, in particular the
:class:`.ShardedQuery` which has some different argument signatures.
Mike Bayer [Thu, 23 Aug 2018 16:40:26 +0000 (12:40 -0400)]
Unwrap Proxy objects when scanning declared_attr
Fixed bug where the declarative scan for attributes would receive the
expression proxy delivered by a hybrid attribute at the class level, and
not the hybrid attribute itself, when receiving the descriptor via the
``@declared_attr`` callable on a subclass of an already-mapped class. This
would lead to an attribute that did not report itself as a hybrid when
viewed within :attr:`.Mapper.all_orm_descriptors`.
Mike Bayer [Thu, 23 Aug 2018 15:55:13 +0000 (11:55 -0400)]
Don't run postfetch_post_update for a DELETE
Fixed 1.2 regression caused by :ticket:`3472` where the handling of an
"updated_at" style column within the context of a post-update operation
would also occur for a row that is to be deleted following the update,
meaning both that a column with a Python-side value generator would show
the now-deleted value that was emitted for the UPDATE before the DELETE
(which was not the previous behavor), as well as that a SQL- emitted value
generator would have the attribute expired, meaning the previous value
would be unreachable due to the row having been deleted and the object
detached from the session.The "postfetch" logic that was added as part of
:ticket:`3472` is now skipped entirely for an object that ultimately is to
be deleted.
Mike Bayer [Wed, 22 Aug 2018 15:13:54 +0000 (11:13 -0400)]
Pass desired array type from pg.array_agg to functions.array_agg
Fixed the :func:`.postgresql.array_agg` function, which is a slightly
altered version of the usual :func:`.functions.array_agg` function, to also
accept an incoming "type" argument without forcing an ARRAY around it,
essentially the same thing that was fixed for the generic function in 1.1
in :ticket:`4107`.
Mike Bayer [Wed, 22 Aug 2018 00:59:04 +0000 (20:59 -0400)]
Strip quotes from format_type in addition to other characters
Fixed bug in PostgreSQL ENUM reflection where a case-sensitive, quoted name
would be reported by the query including quotes, which would not match a
target column during table reflection as the quotes needed to be stripped
off.
Mike Bayer [Mon, 20 Aug 2018 02:19:59 +0000 (22:19 -0400)]
Add missing range_ / rows parameters to additional over() methods
Added missing window function parameters
:paramref:`.WithinGroup.over.range_` and :paramref:`.WithinGroup.over.rows`
parameters to the :meth:`.WithinGroup.over` and
:meth:`.FunctionFilter.over` methods, to correspond to the range/rows
feature added to the "over" method of SQL functions as part of
:ticket:`3049` in version 1.1.
Mike Bayer [Fri, 17 Aug 2018 15:37:30 +0000 (11:37 -0400)]
Accommodate for classically mapped base classes in declarative
Fixed issue in previously untested use case, allowing a declarative mapped
class to inherit from a classically-mapped class outside of the declarative
base, including that it accommodates for unmapped intermediate classes. An
unmapped intermediate class may specify ``__abstract__``, which is now
interpreted correctly, or the intermediate class can remain unmarked, and
the classically mapped base class will be detected within the hierarchy
regardless. In order to anticipate existing scenarios which may be mixing
in classical mappings into existing declarative hierarchies, an error is
now raised if multiple mapped bases are detected for a given class.
Mike Bayer [Wed, 15 Aug 2018 21:11:14 +0000 (17:11 -0400)]
Add concept of "implicit boolean", treat as native
Fixed issue that is closely related to :ticket:`3639` where an expression
rendered in a boolean context on a non-native boolean backend would
be compared to 1/0 even though it is already an implcitly boolean
expression, when :meth:`.ColumnElement.self_group` were used. While this
does not affect the user-friendly backends (MySQL, SQLite) it was not
handled by Oracle (and possibly SQL Server). Whether or not the
expression is implicitly boolean on any database is now determined
up front as an additional check to not generate the integer comparison
within the compliation of the statement.
Add ability to preserve order in MySQL ON DUPLICATE KEY UPDATE.
Added support for the parameters in an ON DUPLICATE KEY UPDATE statement on
MySQL to be ordered, since parameter order in a MySQL UPDATE clause is
significant, in a similar manner as that described at
:ref:`updates_order_parameters`. Pull request courtesy Maxim Bublis.
Mike Bayer [Tue, 7 Aug 2018 22:59:05 +0000 (18:59 -0400)]
Fixed issue with :meth:`.TypeEngine.bind_expression` and
:meth:`.TypeEngine.column_expression` methods where these methods would not
work if the target type were part of a :class:`.Variant`, or other target
type of a :class:`.TypeDecorator`. Additionally, the SQL compiler now
calls upon the dialect-level implementation when it renders these methods
so that dialects can now provide for SQL-level processing for built-in
types.
Nicolas Rolin [Fri, 25 May 2018 17:27:22 +0000 (13:27 -0400)]
Add support of empty list in exanding of bindparam
Added new logic to the "expanding IN" bound parameter feature whereby if
the given list is empty, a special "empty set" expression that is specific
to different backends is generated, thus allowing IN expressions to be
fully dynamic including empty IN expressions.
Mike Bayer [Sat, 4 Aug 2018 17:45:07 +0000 (13:45 -0400)]
Include UPDATE/DELETE extra_froms in correlation
Fixed bug where the multi-table support for UPDATE and DELETE statements
did not consider the additional FROM elements as targets for correlation,
when a correlated SELECT were also combined with the statement. This
change now includes that a SELECT statement in the WHERE clause for such a
statement will try to auto-correlate back to these additional tables in the
parent UPDATE/DELETE or unconditionally correlate if
:meth:`.Select.correlate` is used. Note that auto-correlation raises an
error if the SELECT statement would have no FROM clauses as a result, which
can now occur if the parent UPDATE/DELETE specifies the same tables in its
additional set of tables ; specify :meth:`.Select.correlate` explicitly to
resolve.
Mike Bayer [Wed, 1 Aug 2018 18:12:49 +0000 (14:12 -0400)]
Bind Integers to int for cx_Oracle
For cx_Oracle, Integer datatypes will now be bound to "int", per advice
from the cx_Oracle developers. Previously, using cx_Oracle.NUMBER caused a
loss in precision within the cx_Oracle 6.x series.
Mike Bayer [Wed, 1 Aug 2018 16:01:59 +0000 (12:01 -0400)]
Handle association proxy delete and provide for scalar delete cascade
Fixed multiple issues regarding de-association of scalar objects with the
association proxy. ``del`` now works, and additionally a new flag
:paramref:`.AssociationProxy.cascade_scalar_deletes` is added, which when
set to True indicates that setting a scalar attribute to ``None`` or
deleting via ``del`` will also set the source association to ``None``.
The Python builtin ``dir()`` is now supported for a SQLAlchemy "properties"
object, such as that of a Core columns collection (e.g. ``.c``),
``mapper.attrs``, etc. Allows iPython autocompletion to work as well.
Pull request courtesy Uwe Korn.
Started importing "collections" from "collections.abc" under Python 3.3 and
greater for Python 3.8 compatibility. Pull request courtesy Nathaniel
Knight.
In Python 3.3, the abstract base classes (Iterable, Mapping, etc.)
were moved from the `collections` module and put in the
`collections.abc` module. They remain in the `collections` module for
backwards compatibility, and will until Python 3.8.
This commit adds a variable (`collections_abc`) to the `util/compat`
module, which will be the `collections` module for Python < 3.3 and
before, or the `collections.abc` module for Python >= 3.3. It also
uses the new variable, getting rid of some deprecation warnings that
were seen when running under Python 3.7.
Mike Bayer [Wed, 1 Aug 2018 16:27:22 +0000 (12:27 -0400)]
Correct sharding tests for provisioned follower
The sharding tests created named sqlite databases that were
shared across test suites. It is unknown why these suddenly
started failing and were not failing before.
Fix quoting schemas in _get_table_sql for the SQLite backend
Fixed issue where the "schema" name used for a SQLite database within table
reflection would not quote the schema name correctly. Pull request
courtesy Phillip Cloud.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
