Mike Bayer [Thu, 26 Sep 2019 22:07:46 +0000 (18:07 -0400)]
Move identity_lookup to session
This performance critical method is on Query needlessly, just to appease
the horizontal sharding API. Have the performance impact of invoking
Query only incur if horizontal sharding is actually used.
Mike Bayer [Wed, 25 Sep 2019 21:42:51 +0000 (17:42 -0400)]
Unify generation between Core and ORM query
generation is to be enhanced to include caching
functionality, so ensure that Query and all generative in Core
(e.g. select, DML etc) are using the same generations system.
Additionally, deprecate Select.append methods and state
Select methods independently of their append versions.
Mutability of expression objects is a special case only when
generating new objects during a visit.
Mike Bayer [Mon, 23 Sep 2019 20:33:04 +0000 (16:33 -0400)]
move pytest assert rewrite to conftest.py
this seems to be the best place to put this as it is guaranteed
before the module is imported. this is for the benefit of
3rd party dialects that also would have this in their conftest.py,
so that they don't have to do the "bootstrap" loading hack.
Mike Bayer [Fri, 20 Sep 2019 13:25:40 +0000 (09:25 -0400)]
Support SQLite URIs
Added support for sqlite "URI" connections, which allow for sqlite-specific
flags to be passed in the query string such as "read only" for Python
sqlite3 drivers that support this.
In 14b1e6fe8e18d139846c1aba6761d4eea3dc25c3 we added
suite-level requirements but did not add them to the
base, causing failures in third party dialect test suites.
Mike Bayer [Wed, 18 Sep 2019 01:04:21 +0000 (21:04 -0400)]
Add missing suite level requirements from 2efd89d02941
In 2efd89d02941ab4267d6e2842963fd38b1539f6c we added
suite-level requirements but did not add them to the
base, causing failures in third party dialect test suites.
Mike Bayer [Tue, 17 Sep 2019 16:49:46 +0000 (12:49 -0400)]
Add missing suite level requirements from #4234
In 9d5e117f6fcc38d8773bc943c615888dc8a3a819 we added
suite-level requirements but did not add them to the
base, causing failures in third party dialect test suites.
Mike Bayer [Thu, 12 Sep 2019 16:37:38 +0000 (12:37 -0400)]
Document visitors module
As we are going to be adding a lot of new features to the visitors
module which may impact end-user custom constructs, start documenting
the package for now.
Steven Loria [Mon, 9 Sep 2019 14:48:29 +0000 (10:48 -0400)]
Remove unnecessary util.callable usage
Fixes: #4850
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Removes usage of `util.callable`.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Fri, 6 Sep 2019 22:40:46 +0000 (18:40 -0400)]
Catch set_parent_w_dispatch missing
Added an explicit error message for the case when objects passed to
:class:`.Table` are not :class:`.SchemaItem` objects, rather than resolving
to an attribute error.
Mike Bayer [Fri, 6 Sep 2019 16:45:53 +0000 (12:45 -0400)]
Adjustments to _copy_internals()
We are looking to build a generalization of copy_internals(),
so move out any special logic from these methods. Re-implement
and clarify rationale for the Alias doesnt copy a TableClause rule as
part of the adaption
traversal, establish that we forgot to build out comparison and cache
key for CTE, remove incomplete _copy_internals() from GenerativeSelect
(it doesn't handle the order_by_clause or group_by_clause, so is incomplete)
add SQL Server 2017 to mssql/base.py - Fixes #4833
### Description
Add version 14 for SQL Server 2017 to mssql/base.py
Fixes #4833
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
We could get dialect.requires_name_normalize rather than use hard code
as "firebird" or "oracle", since we have add `normalize` attribute for
quite a long time.
### Description
Use `dialect.requires_name_normalize` instead `testing.against("firebird", "oracle")`
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Thu, 5 Sep 2019 15:46:44 +0000 (11:46 -0400)]
Document Query deduplication
Users are frequently confused why count() does not return the same
number as the number of objects returned by all(). While we continue
to want to find a better solution to this problem, in the meantime
this has never been clearly documented. Add an FAQ section with
links from .count() , .all(), ORM tutorial.
Mike Bayer [Wed, 4 Sep 2019 22:46:53 +0000 (18:46 -0400)]
Strip special chars in anonymized bind names
Characters that interfere with "pyformat" or "named" formats in bound
parameters, namely ``%, (, )`` and the space character, as well as a few
other typically undesirable characters, are stripped early for a
:func:`.bindparam` that is using an anonymized name, which is typically
generated automatically from a named column which itself includes these
characters in its name and does not use a ``.key``, so that they do not
interfere either with the SQLAlchemy compiler's use of string formatting or
with the driver-level parsing of the parameter, both of which could be
demonstrated before the fix. The change only applies to anonymized
parameter names that are generated and consumed internally, not end-user
defined names, so the change should have no impact on any existing code.
Applies in particular to the psycopg2 driver which does not otherwise quote
special parameter names, but also strips leading underscores to suit Oracle
(but not yet leading numbers, as some anon parameters are currently
entirely numeric/underscore based); Oracle in any case continues to quote
parameter names that include special characters.
Mike Bayer [Tue, 3 Sep 2019 13:56:41 +0000 (09:56 -0400)]
Raise for NULL discriminator and pk is present
An exception is now raised if the ORM loads a row for a polymorphic
instance that has a primary key but the discriminator column is NULL, as
discriminator columns should not be null.
Mike Bayer [Thu, 29 Aug 2019 16:09:17 +0000 (12:09 -0400)]
Annotate session-bind-lookup entity in Query-produced selectables
Added new entity-targeting capabilities to the :class:`.Query` object to
help with the case where the :class:`.Session` is using a bind dictionary
against mapped classes, rather than a single bind, and the :class:`.Query`
is against a Core statement that was ultimately generated from a method
such as :meth:`.Query.subquery`; a deep search is performed to locate
any ORM entity related to the query in order to locate a mapper if
one is not otherwise present.
Mike Bayer [Sun, 18 Aug 2019 14:02:24 +0000 (10:02 -0400)]
Render LIMIT/OFFSET conditions after compile on select dialects
Added new "post compile parameters" feature. This feature allows a
:func:`.bindparam` construct to have its value rendered into the SQL string
before being passed to the DBAPI driver, but after the compilation step,
using the "literal render" feature of the compiler. The immediate
rationale for this feature is to support LIMIT/OFFSET schemes that don't
work or perform well as bound parameters handled by the database driver,
while still allowing for SQLAlchemy SQL constructs to be cacheable in their
compiled form. The immediate targets for the new feature are the "TOP
N" clause used by SQL Server (and Sybase) which does not support a bound
parameter, as well as the "ROWNUM" and optional "FIRST_ROWS()" schemes used
by the Oracle dialect, the former of which has been known to perform better
without bound parameters and the latter of which does not support a bound
parameter. The feature builds upon the mechanisms first developed to
support "expanding" parameters for IN expressions. As part of this
feature, the Oracle ``use_binds_for_limits`` feature is turned on
unconditionally and this flag is now deprecated.
- adds limited support for "unique" bound parameters within
a text() construct.
- adds an additional int() check within the literal render
function of the Integer datatype and tests that non-int values
raise ValueError.
Mike Bayer [Fri, 30 Aug 2019 17:11:05 +0000 (13:11 -0400)]
Don't create enum constraints in enum sortable tests
Fixed unit test regression released in 1.3.8 that would cause failure for
Oracle, SQL Server and other non-native ENUM platforms due to new
enumeration tests added as part of :ticket:`4285` enum sortability in the
unit of work; the enumerations created constraints that were duplicated on
name.
Mike Bayer [Tue, 27 Aug 2019 21:08:11 +0000 (17:08 -0400)]
Label simple column transformations as the column name
Additional logic has been added such that certain SQL expressions which
typically wrap a single database column will use the name of that column as
their "anonymous label" name within a SELECT statement, potentially making
key-based lookups in result tuples more intutive. The primary example of
this is that of a CAST expression, e.g. ``CAST(table.colname AS INTEGER)``,
which will export its default name as "colname", rather than the usual
"anon_1" label, that is, ``CAST(table.colname AS INTEGER) AS colname``.
If the inner expression doesn't have a name, then the previous "anonymous
label" logic is used. When using SELECT statements that make use of
:meth:`.Select.apply_labels`, such as those emitted by the ORM, the
labeling logic will produce ``<tablename>_<inner column name>`` in the same
was as if the column were named alone. The logic applies right now to the
:func:`.cast` and :func:`.type_coerce` constructs as well as some
single-element boolean expressions.
Nicolas CANIART [Thu, 22 Aug 2019 18:16:29 +0000 (14:16 -0400)]
Implement type-level sorting for Enum; apply to ORM primary keys
Added support for the use of an :class:`.Enum` datatype using Python
pep-435 enumeration objects as values for use as a primary key column
mapped by the ORM. As these values are not inherently sortable, as
required by the ORM for primary keys, a new
:attr:`.TypeEngine.sort_key_function` attribute is added to the typing
system which allows any SQL type to implement a sorting for Python objects
of its type which is consulted by the unit of work. The :class:`.Enum`
type then defines this using the database value of a given enumeration.
The sorting scheme can be also be redefined by passing a callable to the
:paramref:`.Enum.sort_key_function` parameter. Pull request courtesy
Nicolas Caniart.
Added support for reflection of CHECK constraints that include the special
PostgreSQL qualifier "NOT VALID", which can be present for CHECK
constraints that were added to an exsiting table with the directive that
they not be applied to existing data in the table. The PostgreSQL
dictionary for CHECK constraints as returned by
:meth:`.Inspector.get_check_constraints` may include an additional entry
``dialect_options`` which within will contain an entry ``"not_valid":
True`` if this symbol is detected. Pull request courtesy Bill Finn.
Mike Bayer [Mon, 26 Aug 2019 15:44:09 +0000 (11:44 -0400)]
Serialize the context dictionary in Load objects
Fixed bug where :class:`.Load` objects were not pickleable due to
mapper/relationship state in the internal context dictionary. These
objects are now converted to picklable using similar techniques as that of
other elements within the loader option system that have long been
serializable.
Mike Bayer [Mon, 26 Aug 2019 13:45:06 +0000 (09:45 -0400)]
Remove erroneous assertion from array._bind_param
Fixed bug where Postgresql operators such as
:meth:`.postgresql.ARRAY.Comparator.contains` and
:meth:`.postgresql.ARRAY.Comparator.contained_by` would fail to function
correctly for non-integer values when used against a
:class:`.postgresql.array` object, due to an erroneous assert statement.
Mike Bayer [Wed, 26 Apr 2017 22:50:05 +0000 (18:50 -0400)]
Run eager loaders on unexpire
Eager loaders, such as joined loading, SELECT IN loading, etc., when
configured on a mapper or via query options will now be invoked during
the refresh on an expired object; in the case of selectinload and
subqueryload, since the additional load is for a single object only,
the "immediateload" scheme is used in these cases which resembles the
single-parent query emitted by lazy loading.
Mike Bayer [Wed, 21 Aug 2019 21:29:55 +0000 (17:29 -0400)]
Add hide_parameters flag to create_engine
Added new parameter :paramref:`.create_engine.hide_parameters` which when
set to True will cause SQL parameters to no longer be logged, nor rendered
in the string representation of a :class:`.StatementError` object.
Mike Bayer [Tue, 20 Aug 2019 13:42:36 +0000 (09:42 -0400)]
Document how to work with reflection and custom datatypes
Describe the link between table reflection, datatype lookups,
and what approaches are needed in the case where in-Python datatypes
are needed in the reflected table metadata.
Mike Bayer [Sun, 18 Aug 2019 17:03:49 +0000 (13:03 -0400)]
Reflect PK of referred table if referred columns not present
Fixed bug where a FOREIGN KEY that was set up to refer to the parent table
by table name only without the column names would not correctly be
reflected as far as setting up the "referred columns", since SQLite's
PRAGMA does not report on these columns if they weren't given explicitly.
For some reason this was harcoded to assume the name of the local column,
which might work for some cases but is not correct. The new approach
reflects the primary key of the referred table and uses the constraint
columns list as the referred columns list, if the remote column(s) aren't
present in the reflected pragma directly.
Mike Bayer [Sat, 17 Aug 2019 02:38:51 +0000 (22:38 -0400)]
Revise psycopg2 execute_values approach
Revised the approach for the just added support for the psycopg2
"execute_values()" feature added in 1.3.7 for :ticket:`4623`. The approach
relied upon a regular expression that would fail to match for a more
complex INSERT statement such as one which had subqueries involved. The
new approach matches exactly the string that was rendered as the VALUES
clause.
Mike Bayer [Fri, 16 Aug 2019 22:07:06 +0000 (18:07 -0400)]
Add new "exec_once_unless_exception" system; apply to dialect.initialize
Fixed an issue whereby if the dialect "initialize" process which occurs on
first connect would encounter an unexpected exception, the initialize
process would fail to complete and then no longer attempt on subsequent
connection attempts, leaving the dialect in an un-initialized, or partially
initialized state, within the scope of parameters that need to be
established based on inspection of a live connection. The "invoke once"
logic in the event system has been reworked to accommodate for this
occurrence using new, private API features that establish an "exec once"
hook that will continue to allow the initializer to fire off on subsequent
connections, until it completes without raising an exception. This does not
impact the behavior of the existing ``once=True`` flag within the event
system.
Yuval Dinari [Mon, 12 Aug 2019 14:44:59 +0000 (10:44 -0400)]
Add new executemany_mode, support for psycopg2.extras.execute_values()
Added new dialect flag for the psycopg2 dialect, ``executemany_mode`` which
supersedes the previous experimental ``use_batch_mode`` flag.
``executemany_mode`` supports both the "execute batch" and "execute values"
functions provided by psycopg2, the latter which is used for compiled
:func:`.insert` constructs. Pull request courtesy Yuval Dinari.
Mike Bayer [Sun, 11 Aug 2019 19:24:13 +0000 (15:24 -0400)]
Rewrite pool reset_on_return parsing using a util function
Choosing a util.symbol() based on a user parameter is about to have
another use case added as part of #4623, so add a generalized solution
ahead of it.
Mike Bayer [Sun, 11 Aug 2019 14:54:49 +0000 (10:54 -0400)]
Use ternary when running conditional with Query._offset
Fixed bug where using :meth:`.Query.first` or a slice expression in
conjunction with a query that has an expression based "offset" applied
would raise TypeError, due to an "or" conditional against "offset" that did
not expect it to be a SQL expression as opposed to an integer or None.
Mike Bayer [Sun, 11 Aug 2019 21:42:59 +0000 (17:42 -0400)]
Emit SET NAMES for all MySQL connections w charset
The MySQL dialects will emit "SET NAMES" at the start of a connection when
charset is given to the MySQL driver, to appease an apparent behavior
observed in MySQL 8.0 that raises a collation error when a UNION includes
string columns unioned against columns of the form CAST(NULL AS CHAR(..)),
which is what SQLAlchemy's polymorphic_union function does. The issue
seems to have affected PyMySQL for at least a year, however has recently
appeared as of mysqlclient 1.4.4 based on changes in how this DBAPI creates
a connection. As the presence of this directive impacts three separate
MySQL charset settings which each have intricate effects based on their
presense, SQLAlchemy will now emit the directive on new connections to
ensure correct behavior.
Mike Bayer [Sat, 10 Aug 2019 22:17:59 +0000 (18:17 -0400)]
Clarify INSERT/UPDATE defaults vs. data marshalling
- Add a full introductory paragraph to INSERT/UPDATE defaults
stating what we mean when we talk about this concept. Add a note
differentiating what a default does, vs. a rule that intercepts
data as it moves into the database, providing links
- Add a quick section referring to TypeDecorator in the ORM
section on modifying attribute behavior
- Add an "ORM tip", a new thing that we can use in Core to link
to ORM concepts when useful, in the TypeDecorator section which
mentions that for more open ended conversion of arbitrary user
data based on business rules, @validates might be useful
(although this still does not suit the case of full blown form
validation).
- add glossary entries for DML, data marshalling since we already
use these terms and I'd like to refer to them more often.
Mike Bayer [Fri, 9 Aug 2019 03:34:20 +0000 (23:34 -0400)]
Correct name for json_serializer / json_deserializer, document and test
The dialects that support json are supposed to take arguments
``json_serializer`` and ``json_deserializer`` at the create_engine() level,
however the SQLite dialect calls them ``_json_serilizer`` and
``_json_deserilalizer``. The names have been corrected, the old names are
accepted with a change warning, and these parameters are now documented as
:paramref:`.create_engine.json_serializer` and
:paramref:`.create_engine.json_deserializer`.
Mike Bayer [Tue, 6 Aug 2019 20:10:09 +0000 (16:10 -0400)]
Implement checkfirst for Index.create(), Index.drop()
The :meth:`.Index.create` and :meth:`.Index.drop` methods now have a
parameter :paramref:`.Index.create.checkfirst`, in the same way as that of
:class:`.Table` and :class:`.Sequence`, which when enabled will cause the
operation to detect if the index exists (or not) before performing a create
or drop operation.
Mike Bayer [Fri, 2 Aug 2019 16:34:16 +0000 (12:34 -0400)]
Modernize internal reflection
- Deprecated remaining engine-level introspection and utility methods
including :meth:`.Engine.run_callable`, :meth:`.Engine.transaction`,
:meth:`.Engine.table_names`, :meth:`.Engine.has_table`. The utility
methods are superseded by modern context-manager patterns, and the table
introspection tasks are suited by the :class:`.Inspector` object.
- The internal dialect method ``Dialect.reflecttable`` has been removed. A
review of third party dialects has not found any making use of this method,
as it was already documented as one that should not be used by external
dialects. Additionally, the private ``Engine._run_visitor`` method
is also removed.
- The long-deprecated ``Inspector.get_table_names.order_by`` parameter has
been removed.
- The :paramref:`.Table.autoload_with` parameter now accepts an :class:`.Inspector` object
directly, as well as any :class:`.Engine` or :class:`.Connection` as was the case before.
Mike Bayer [Sat, 3 Aug 2019 03:20:06 +0000 (23:20 -0400)]
Strong reference listen function wrapped by "once"
Fixed issue in event system where using the ``once=True`` flag with
dynamically generated listener functions would cause event registration of
future events to fail if those listener functions were garbage collected
after they were used, due to an assumption that a listened function is
strongly referenced. The "once" wrapped is now modified to strongly
reference the inner function persistently, and documentation is updated
that using "once" does not imply automatic de-registration of listener
functions.
Mike Bayer [Fri, 2 Aug 2019 17:03:29 +0000 (13:03 -0400)]
Always include a schema name in SQLite PRAGMA
Fixed bug where usage of "PRAGMA table_info" in SQLite dialect meant that
reflection features to detect for table existence, list of table columns,
and list of foreign keys, would default to any table in any attached
database, when no schema name was given and the table did not exist in the
base schema. The fix explicitly runs PRAGMA for the 'main' schema and then
the 'temp' schema if the 'main' returned no rows, to maintain the behavior
of tables + temp tables in the "no schema" namespace, attached tables only
in the "schema" namespace.
Mike Bayer [Thu, 1 Aug 2019 17:27:43 +0000 (13:27 -0400)]
Add additional tests to verify _is_clone_of proxy level link
The commit message in 896d47f318c5c27620fd6da is not accurate,
we do in fact still need _is_clone_of when a clone of a column is
created so that we can implement _cloned_set(). Additionally,
the linkage of _is_clone_of within make_proxy() also suits
an additional use case that seems to be related to [ticket:2419].
Adjust one of the tests which likely got changed within 1.4's
refactoring of Select to test this correctly.
Mike Bayer [Thu, 1 Aug 2019 15:45:34 +0000 (11:45 -0400)]
Don't assume key when matching cloned columns in _make_proxy
Fixed issue where internal cloning of SELECT constructs could lead to a key
error if the copy of the SELECT changed its state such that its list of
columns changed. This was observed to be occurring in some ORM scenarios
which may be unique to 1.3 and above, so is partially a regression fix.
For 1.4, the _is_clone_of key will be removed entirely as it seems to
have no purpose. This commit is the initial backport to 1.3 which
includes tests.
Mike Bayer [Mon, 29 Jul 2019 18:49:22 +0000 (14:49 -0400)]
Invoke column_expression() for subsequent SELECTs in CompoundSelect
Fixed bug where :meth:`.TypeEngine.column_expression` method would not be
applied to subsequent SELECT statements inside of a UNION or other
:class:`.CompoundSelect`, even though the SELECT statements are rendered at
the topmost level of the statement. New logic now differentiates between
rendering the column expression, which is needed for all SELECTs in the
list, vs. gathering the returned data type for the result row, which is
needed only for the first SELECT.
Mike Bayer [Mon, 29 Jul 2019 18:07:21 +0000 (14:07 -0400)]
Correct for MySQL 8.0 table and schema names in FK reflection
Added another fix for an upstream MySQL 8 issue where a case sensitive
table name is reported incorrectly in foreign key constraint reflection,
this is an extension of the fix first added for :ticket:`4344` which
affects a case sensitive column name. The new issue occurs through MySQL
8.0.17, so the general logic of the 88718 fix remains in place.
Mike Bayer [Tue, 23 Jul 2019 22:38:48 +0000 (18:38 -0400)]
Ensure all Index arguments are counted when matching expr/columns
Fixed issue where :class:`.Index` object which contained a mixture of
functional expressions which were not resolvable to a particular column,
in combination with string-based column names, would fail to initialize
its internal state correctly leading to failures during DDL compilation.
Mike Bayer [Tue, 23 Jul 2019 13:24:32 +0000 (09:24 -0400)]
Don't assume m2o key is present in the dictionary
Fixed regression caused by new selectinload for many-to-one logic where
a primaryjoin condition not based on real foreign keys would cause
KeyError if a related object did not exist for a given key value on the
parent object.
Mike Bayer [Fri, 19 Jul 2019 17:05:06 +0000 (13:05 -0400)]
Optimize out JOIN for selectinload with many to one
The optimzation applied to selectin loading in :ticket:`4340` where a JOIN
is not needed to eagerly load related items is now applied to many-to-one
relationships as well, so that only the related table is queried for a
simple join condition. In this case, the related items are queried
based on the value of a foreign key column on the parent; if these columns
are deferred or otherwise not loaded on any of the parent objects in
the collection, the loader falls back to the JOIN method.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
