Mike Bayer [Tue, 20 Aug 2019 13:42:36 +0000 (09:42 -0400)]
Document how to work with reflection and custom datatypes
Describe the link between table reflection, datatype lookups,
and what approaches are needed in the case where in-Python datatypes
are needed in the reflected table metadata.
Mike Bayer [Sun, 18 Aug 2019 17:03:49 +0000 (13:03 -0400)]
Reflect PK of referred table if referred columns not present
Fixed bug where a FOREIGN KEY that was set up to refer to the parent table
by table name only without the column names would not correctly be
reflected as far as setting up the "referred columns", since SQLite's
PRAGMA does not report on these columns if they weren't given explicitly.
For some reason this was harcoded to assume the name of the local column,
which might work for some cases but is not correct. The new approach
reflects the primary key of the referred table and uses the constraint
columns list as the referred columns list, if the remote column(s) aren't
present in the reflected pragma directly.
Mike Bayer [Sat, 17 Aug 2019 02:38:51 +0000 (22:38 -0400)]
Revise psycopg2 execute_values approach
Revised the approach for the just added support for the psycopg2
"execute_values()" feature added in 1.3.7 for :ticket:`4623`. The approach
relied upon a regular expression that would fail to match for a more
complex INSERT statement such as one which had subqueries involved. The
new approach matches exactly the string that was rendered as the VALUES
clause.
Mike Bayer [Fri, 16 Aug 2019 22:07:06 +0000 (18:07 -0400)]
Add new "exec_once_unless_exception" system; apply to dialect.initialize
Fixed an issue whereby if the dialect "initialize" process which occurs on
first connect would encounter an unexpected exception, the initialize
process would fail to complete and then no longer attempt on subsequent
connection attempts, leaving the dialect in an un-initialized, or partially
initialized state, within the scope of parameters that need to be
established based on inspection of a live connection. The "invoke once"
logic in the event system has been reworked to accommodate for this
occurrence using new, private API features that establish an "exec once"
hook that will continue to allow the initializer to fire off on subsequent
connections, until it completes without raising an exception. This does not
impact the behavior of the existing ``once=True`` flag within the event
system.
Yuval Dinari [Mon, 12 Aug 2019 14:44:59 +0000 (10:44 -0400)]
Add new executemany_mode, support for psycopg2.extras.execute_values()
Added new dialect flag for the psycopg2 dialect, ``executemany_mode`` which
supersedes the previous experimental ``use_batch_mode`` flag.
``executemany_mode`` supports both the "execute batch" and "execute values"
functions provided by psycopg2, the latter which is used for compiled
:func:`.insert` constructs. Pull request courtesy Yuval Dinari.
Mike Bayer [Sun, 11 Aug 2019 19:24:13 +0000 (15:24 -0400)]
Rewrite pool reset_on_return parsing using a util function
Choosing a util.symbol() based on a user parameter is about to have
another use case added as part of #4623, so add a generalized solution
ahead of it.
Mike Bayer [Sun, 11 Aug 2019 14:54:49 +0000 (10:54 -0400)]
Use ternary when running conditional with Query._offset
Fixed bug where using :meth:`.Query.first` or a slice expression in
conjunction with a query that has an expression based "offset" applied
would raise TypeError, due to an "or" conditional against "offset" that did
not expect it to be a SQL expression as opposed to an integer or None.
Mike Bayer [Sun, 11 Aug 2019 21:42:59 +0000 (17:42 -0400)]
Emit SET NAMES for all MySQL connections w charset
The MySQL dialects will emit "SET NAMES" at the start of a connection when
charset is given to the MySQL driver, to appease an apparent behavior
observed in MySQL 8.0 that raises a collation error when a UNION includes
string columns unioned against columns of the form CAST(NULL AS CHAR(..)),
which is what SQLAlchemy's polymorphic_union function does. The issue
seems to have affected PyMySQL for at least a year, however has recently
appeared as of mysqlclient 1.4.4 based on changes in how this DBAPI creates
a connection. As the presence of this directive impacts three separate
MySQL charset settings which each have intricate effects based on their
presense, SQLAlchemy will now emit the directive on new connections to
ensure correct behavior.
Mike Bayer [Sat, 10 Aug 2019 22:17:59 +0000 (18:17 -0400)]
Clarify INSERT/UPDATE defaults vs. data marshalling
- Add a full introductory paragraph to INSERT/UPDATE defaults
stating what we mean when we talk about this concept. Add a note
differentiating what a default does, vs. a rule that intercepts
data as it moves into the database, providing links
- Add a quick section referring to TypeDecorator in the ORM
section on modifying attribute behavior
- Add an "ORM tip", a new thing that we can use in Core to link
to ORM concepts when useful, in the TypeDecorator section which
mentions that for more open ended conversion of arbitrary user
data based on business rules, @validates might be useful
(although this still does not suit the case of full blown form
validation).
- add glossary entries for DML, data marshalling since we already
use these terms and I'd like to refer to them more often.
Mike Bayer [Fri, 9 Aug 2019 03:34:20 +0000 (23:34 -0400)]
Correct name for json_serializer / json_deserializer, document and test
The dialects that support json are supposed to take arguments
``json_serializer`` and ``json_deserializer`` at the create_engine() level,
however the SQLite dialect calls them ``_json_serilizer`` and
``_json_deserilalizer``. The names have been corrected, the old names are
accepted with a change warning, and these parameters are now documented as
:paramref:`.create_engine.json_serializer` and
:paramref:`.create_engine.json_deserializer`.
Mike Bayer [Tue, 6 Aug 2019 20:10:09 +0000 (16:10 -0400)]
Implement checkfirst for Index.create(), Index.drop()
The :meth:`.Index.create` and :meth:`.Index.drop` methods now have a
parameter :paramref:`.Index.create.checkfirst`, in the same way as that of
:class:`.Table` and :class:`.Sequence`, which when enabled will cause the
operation to detect if the index exists (or not) before performing a create
or drop operation.
Mike Bayer [Fri, 2 Aug 2019 16:34:16 +0000 (12:34 -0400)]
Modernize internal reflection
- Deprecated remaining engine-level introspection and utility methods
including :meth:`.Engine.run_callable`, :meth:`.Engine.transaction`,
:meth:`.Engine.table_names`, :meth:`.Engine.has_table`. The utility
methods are superseded by modern context-manager patterns, and the table
introspection tasks are suited by the :class:`.Inspector` object.
- The internal dialect method ``Dialect.reflecttable`` has been removed. A
review of third party dialects has not found any making use of this method,
as it was already documented as one that should not be used by external
dialects. Additionally, the private ``Engine._run_visitor`` method
is also removed.
- The long-deprecated ``Inspector.get_table_names.order_by`` parameter has
been removed.
- The :paramref:`.Table.autoload_with` parameter now accepts an :class:`.Inspector` object
directly, as well as any :class:`.Engine` or :class:`.Connection` as was the case before.
Mike Bayer [Sat, 3 Aug 2019 03:20:06 +0000 (23:20 -0400)]
Strong reference listen function wrapped by "once"
Fixed issue in event system where using the ``once=True`` flag with
dynamically generated listener functions would cause event registration of
future events to fail if those listener functions were garbage collected
after they were used, due to an assumption that a listened function is
strongly referenced. The "once" wrapped is now modified to strongly
reference the inner function persistently, and documentation is updated
that using "once" does not imply automatic de-registration of listener
functions.
Mike Bayer [Fri, 2 Aug 2019 17:03:29 +0000 (13:03 -0400)]
Always include a schema name in SQLite PRAGMA
Fixed bug where usage of "PRAGMA table_info" in SQLite dialect meant that
reflection features to detect for table existence, list of table columns,
and list of foreign keys, would default to any table in any attached
database, when no schema name was given and the table did not exist in the
base schema. The fix explicitly runs PRAGMA for the 'main' schema and then
the 'temp' schema if the 'main' returned no rows, to maintain the behavior
of tables + temp tables in the "no schema" namespace, attached tables only
in the "schema" namespace.
Mike Bayer [Thu, 1 Aug 2019 17:27:43 +0000 (13:27 -0400)]
Add additional tests to verify _is_clone_of proxy level link
The commit message in 896d47f318c5c27620fd6da is not accurate,
we do in fact still need _is_clone_of when a clone of a column is
created so that we can implement _cloned_set(). Additionally,
the linkage of _is_clone_of within make_proxy() also suits
an additional use case that seems to be related to [ticket:2419].
Adjust one of the tests which likely got changed within 1.4's
refactoring of Select to test this correctly.
Mike Bayer [Thu, 1 Aug 2019 15:45:34 +0000 (11:45 -0400)]
Don't assume key when matching cloned columns in _make_proxy
Fixed issue where internal cloning of SELECT constructs could lead to a key
error if the copy of the SELECT changed its state such that its list of
columns changed. This was observed to be occurring in some ORM scenarios
which may be unique to 1.3 and above, so is partially a regression fix.
For 1.4, the _is_clone_of key will be removed entirely as it seems to
have no purpose. This commit is the initial backport to 1.3 which
includes tests.
Mike Bayer [Mon, 29 Jul 2019 18:49:22 +0000 (14:49 -0400)]
Invoke column_expression() for subsequent SELECTs in CompoundSelect
Fixed bug where :meth:`.TypeEngine.column_expression` method would not be
applied to subsequent SELECT statements inside of a UNION or other
:class:`.CompoundSelect`, even though the SELECT statements are rendered at
the topmost level of the statement. New logic now differentiates between
rendering the column expression, which is needed for all SELECTs in the
list, vs. gathering the returned data type for the result row, which is
needed only for the first SELECT.
Mike Bayer [Mon, 29 Jul 2019 18:07:21 +0000 (14:07 -0400)]
Correct for MySQL 8.0 table and schema names in FK reflection
Added another fix for an upstream MySQL 8 issue where a case sensitive
table name is reported incorrectly in foreign key constraint reflection,
this is an extension of the fix first added for :ticket:`4344` which
affects a case sensitive column name. The new issue occurs through MySQL
8.0.17, so the general logic of the 88718 fix remains in place.
Mike Bayer [Tue, 23 Jul 2019 22:38:48 +0000 (18:38 -0400)]
Ensure all Index arguments are counted when matching expr/columns
Fixed issue where :class:`.Index` object which contained a mixture of
functional expressions which were not resolvable to a particular column,
in combination with string-based column names, would fail to initialize
its internal state correctly leading to failures during DDL compilation.
Mike Bayer [Tue, 23 Jul 2019 13:24:32 +0000 (09:24 -0400)]
Don't assume m2o key is present in the dictionary
Fixed regression caused by new selectinload for many-to-one logic where
a primaryjoin condition not based on real foreign keys would cause
KeyError if a related object did not exist for a given key value on the
parent object.
Mike Bayer [Fri, 19 Jul 2019 17:05:06 +0000 (13:05 -0400)]
Optimize out JOIN for selectinload with many to one
The optimzation applied to selectin loading in :ticket:`4340` where a JOIN
is not needed to eagerly load related items is now applied to many-to-one
relationships as well, so that only the related table is queried for a
simple join condition. In this case, the related items are queried
based on the value of a foreign key column on the parent; if these columns
are deferred or otherwise not loaded on any of the parent objects in
the collection, the loader falls back to the JOIN method.
Mike Bayer [Tue, 16 Jul 2019 16:41:09 +0000 (12:41 -0400)]
Support tuple IN VALUES for SQLite, others
Added support for composite (tuple) IN operators with SQLite, by rendering
the VALUES keyword for this backend. As other backends such as DB2 are
known to use the same syntax, the syntax is enabled in the base compiler
using a dialect-level flag ``tuple_in_values``. The change also includes
support for "empty IN tuple" expressions for SQLite when using "in_()"
between a tuple value and an empty set.
Mike Bayer [Thu, 18 Jul 2019 15:12:49 +0000 (11:12 -0400)]
Fix regression for self-ref join to self error message
Fixed regression caused by :ticket:`4365` where a join from an entity to
itself without using aliases no longer raises an informative error message,
instead failing on an assertion. The informative error condition has been
restored.
Mike Bayer [Thu, 18 Jul 2019 14:58:24 +0000 (10:58 -0400)]
Fixes for uselist=True with m2o relationships
Fixed bug where a many-to-one relationship that specified ``uselist=True``
would fail to update correctly during a primary key change where a related
column needs to change.
Fixed bug where the detection for many-to-one or one-to-one use with a
"dynamic" relationship, which is an invalid configuration, would fail to
raise if the relationship were configured with ``uselist=True``. The
current fix is that it warns, instead of raises, as this would otherwise be
backwards incompatible, however in a future release it will be a raise.
Mike Bayer [Tue, 16 Jul 2019 17:02:16 +0000 (13:02 -0400)]
Intercept unresolveable comparator attrbute error for attr access
Fixed bug where a synonym created against a mapped attribute that does not
exist yet, as is the case when it refers to backref before mappers are
configured, would raise recursion errors when trying to test for attributes
on it which ultimately don't exist (as occurs when the classes are run
through Sphinx autodoc), as the unconfigured state of the synonym would put
it into an attribute not found loop.
Mike Bayer [Thu, 11 Apr 2019 18:21:13 +0000 (14:21 -0400)]
Remove threadlocal engine strategy, engine strategies pool threadlocal
The "threadlocal" execution strategy, deprecated in 1.3, has been
removed for 1.4, as well as the concept of "engine strategies" and the
``Engine.contextual_connect`` method. The "strategy='mock'" keyword
argument is still accepted for now with a deprecation warning; use
:func:`.create_mock_engine` instead for this use case.
Mike Bayer [Sun, 14 Jul 2019 15:43:29 +0000 (11:43 -0400)]
Add profile sort option to test suite
I use the nfl sort a lot to see what calling changes are happening
between two versions in order to identify the offending code, so
add it as a command line option.
Mike Bayer [Sat, 13 Jul 2019 02:43:31 +0000 (22:43 -0400)]
self_group() for FunctionFilter
Fixed issue where the :class:`.array_agg` construct in combination with
:meth:`.FunctionElement.filter` would not produce the correct operator
precedence between the FILTER keyword and the array index operator.
Mike Bayer [Sun, 7 Jul 2019 15:12:31 +0000 (11:12 -0400)]
Allow duplicate columns in from clauses and selectables
The :func:`.select` construct and related constructs now allow for
duplication of column labels and columns themselves in the columns clause,
mirroring exactly how column expressions were passed in. This allows
the tuples returned by an executed result to match what was SELECTed
for in the first place, which is how the ORM :class:`.Query` works, so
this establishes better cross-compatibility between the two constructs.
Additionally, it allows column-positioning-sensitive structures such as
UNIONs (i.e. :class:`.CompoundSelect`) to be more intuitively constructed
in those cases where a particular column might appear in more than one
place. To support this change, the :class:`.ColumnCollection` has been
revised to support duplicate columns as well as to allow integer index
access.
Mike Bayer [Thu, 11 Jul 2019 15:56:54 +0000 (11:56 -0400)]
profiling plugin fixes
support that we have pytest-xdist arguments now,
as well as allow force-write to reset the list of callcounts which
seem to accumulate commas for some reason
Mike Bayer [Wed, 10 Jul 2019 20:12:48 +0000 (16:12 -0400)]
Add performance improvement for Enum w/ Python 2 enum library
Adjusted the initialization for :class:`.Enum` to minimize how often it
invokes the ``.__members__`` attribute of a given PEP-435 enumeration
object, to suit the case where this attribute is expensive to invoke, as is
the case for some popular third party enumeration libraries.
Mike Bayer [Mon, 8 Jul 2019 19:46:35 +0000 (15:46 -0400)]
Support multidimensional array literals in Postgresql
Added support for multidimensional Postgresql array literals via nesting
the :class:`.postgresql.array` object within another one. The
multidimensional array type is detected automatically.
Mike Bayer [Mon, 8 Jul 2019 14:42:20 +0000 (10:42 -0400)]
Ensure .engine is part of Connectable interface, implement as descriptor
Fixed bug where using reflection function such as :meth:`.MetaData.reflect`
with an :class:`.Engine` object that had execution options applied to it
would fail, as the resulting :class:`.OptionEngine` proxy object failed to
include a ``.engine`` attribute used within the reflection routines.
Mike Bayer [Thu, 13 Jun 2019 16:37:22 +0000 (12:37 -0400)]
SelectBase no longer a FromClause
As part of the SQLAlchemy 2.0 migration project, a conceptual change has
been made to the role of the :class:`.SelectBase` class hierarchy,
which is the root of all "SELECT" statement constructs, in that they no
longer serve directly as FROM clauses, that is, they no longer subclass
:class:`.FromClause`. For end users, the change mostly means that any
placement of a :func:`.select` construct in the FROM clause of another
:func:`.select` requires first that it be wrapped in a subquery first,
which historically is through the use of the :meth:`.SelectBase.alias`
method, and is now also available through the use of
:meth:`.SelectBase.subquery`. This was usually a requirement in any
case since several databases don't accept unnamed SELECT subqueries
in their FROM clause in any case.
See the documentation in this change for lots more detail.
Mike Bayer [Fri, 5 Jul 2019 16:19:55 +0000 (12:19 -0400)]
Note DBAPIs and dialects that we don't support
Since we have strong CI for the DBAPIs and dialects that are actively
supported, this indicates that those DBAPIs that aren't in CI are
continuing to fall behind in support, to the point where we can
not address issues that may arise. As such, the Sybase and Firebird
dialects overall are moving into an explicit "not supported" zone
where we would like to eventually remove them. Additionally,
a pass is made through legacy MySQL and PostgreSQL DBAPI dialects
as well as those which we aren't able to include in CI to note
that these DBAPIs aren't actively supported by the project.
Mike Bayer [Tue, 2 Jul 2019 02:33:26 +0000 (22:33 -0400)]
Clear proxy_set cache when creating an annotated column
Fixed an unlikely issue where the "corresponding column" routine for unions
and other :class:`.CompoundSelect` objects could return the wrong column in
some overlapping column situtations, thus potentially impacting some ORM
operations when set operations are in use, if the underlying
:func:`.select` constructs were used previously in other similar kinds of
routines, due to a cached value not being cleared.
Denis Kataev [Fri, 28 Jun 2019 13:57:47 +0000 (09:57 -0400)]
Test for _ORMJoin and fix issue
Fixed an issue where the :meth:`.orm._ORMJoin.join` method, which is a
not-internally-used ORM-level method that exposes what is normally an
internal process of :meth:`.Query.join`, did not propagate the ``full`` and
``outerjoin`` keyword arguments correctly. Pull request courtesy Denis
Kataev.
Mike Bayer [Fri, 28 Jun 2019 14:41:38 +0000 (10:41 -0400)]
CAST bind values against SQL Server sys into NVARCHAR
Ensured that the queries used to reflect indexes and view definitions will
explicitly CAST string parameters into NVARCHAR, as many SQL Server drivers
frequently treat string values, particularly those with non-ascii
characters or larger string values, as TEXT which often don't compare
correctly against VARCHAR characters in SQL Server's information schema
tables for some reason. These CAST operations already take place for
reflection queries against SQL Server ``information_schema.`` tables but
were missing from three additional queries that are against ``sys.``
tables.
Mike Bayer [Thu, 20 Jun 2019 19:37:59 +0000 (15:37 -0400)]
Add Load.options() for hierchical construction of loader options
Added new loader option method :meth:`.Load.options` which allows loader
options to be constructed hierarchically, so that many sub-options can be
applied to a particular path without needing to call :func:`.defaultload`
many times. Thanks to Alessio Bogon for the idea.
Also applies a large pass to the loader option documentation which
needed improvement.
Unfortunately target version py27 makes Black want to break
print functions, even if you put a "from __future__ import print_function"
at the top, so it is being inconsistent in its "we've decided this
is a python 2 file" logic and basically this thing really needs
command line arguments
Mike Bayer [Thu, 27 Jun 2019 17:25:12 +0000 (13:25 -0400)]
Unwrap TIMESTAMP when doing an isinstance()
Fixed bug where the special logic to render "NULL" for the
:class:`.TIMESTAMP` datatype when ``nullable=True`` would not work if the
column's datatype were a :class:`.TypeDecorator` or a :class:`.Variant`.
The logic now ensures that it unwraps down to the original
:class:`.TIMESTAMP` so that this special case NULL keyword is correctly
rendered when requested.
Mike Bayer [Thu, 27 Jun 2019 14:07:04 +0000 (10:07 -0400)]
Remove test-within-a-test structure
These tests in of_type were relying upon generating the
polymorphic selectable from the mapper which is proving to be
more challenging to correct when that mechanism is changing.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
