Mike Bayer [Sat, 4 Aug 2018 17:45:07 +0000 (13:45 -0400)]
Include UPDATE/DELETE extra_froms in correlation
Fixed bug where the multi-table support for UPDATE and DELETE statements
did not consider the additional FROM elements as targets for correlation,
when a correlated SELECT were also combined with the statement. This
change now includes that a SELECT statement in the WHERE clause for such a
statement will try to auto-correlate back to these additional tables in the
parent UPDATE/DELETE or unconditionally correlate if
:meth:`.Select.correlate` is used. Note that auto-correlation raises an
error if the SELECT statement would have no FROM clauses as a result, which
can now occur if the parent UPDATE/DELETE specifies the same tables in its
additional set of tables ; specify :meth:`.Select.correlate` explicitly to
resolve.
Mike Bayer [Wed, 1 Aug 2018 18:12:49 +0000 (14:12 -0400)]
Bind Integers to int for cx_Oracle
For cx_Oracle, Integer datatypes will now be bound to "int", per advice
from the cx_Oracle developers. Previously, using cx_Oracle.NUMBER caused a
loss in precision within the cx_Oracle 6.x series.
Started importing "collections" from "collections.abc" under Python 3.3 and
greater for Python 3.8 compatibility. Pull request courtesy Nathaniel
Knight.
In Python 3.3, the abstract base classes (Iterable, Mapping, etc.)
were moved from the `collections` module and put in the
`collections.abc` module. They remain in the `collections` module for
backwards compatibility, and will until Python 3.8.
This commit adds a variable (`collections_abc`) to the `util/compat`
module, which will be the `collections` module for Python < 3.3 and
before, or the `collections.abc` module for Python >= 3.3. It also
uses the new variable, getting rid of some deprecation warnings that
were seen when running under Python 3.7.
Change-Id: I2d1c0ef97c8ecac7af152cc56263422a40faa6bb
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/464
(cherry picked from commit a59c3b8f302a34ab037ec445b7452b1f353b91af)
Mike Bayer [Wed, 1 Aug 2018 16:27:22 +0000 (12:27 -0400)]
Correct sharding tests for provisioned follower
The sharding tests created named sqlite databases that were
shared across test suites. It is unknown why these suddenly
started failing and were not failing before.
Fix quoting schemas in _get_table_sql for the SQLite backend
Fixed issue where the "schema" name used for a SQLite database within table
reflection would not quote the schema name correctly. Pull request
courtesy Phillip Cloud.
Change-Id: I2770788c1f094a7743209250ec26b5ef5fb2d9e8
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/463
(cherry picked from commit 893eac06e511f3765c0c89bab76d7933d83ffccc)
Mike Bayer [Fri, 13 Jul 2018 16:58:21 +0000 (12:58 -0400)]
Use exprs for bundle __clause_element__
Fixed bug in :class:`.Bundle` construct where placing two columns of the
same name would be de-duplicated, when the :class:`.Bundle` were used as
part of the rendered SQL, such as in the ORDER BY or GROUP BY of the statement.
Mike Bayer [Tue, 10 Jul 2018 13:41:21 +0000 (09:41 -0400)]
Drop default-related structures after the Table is dropped.
Fixed bug where a :class:`.Sequence` would be dropped explicitly before any
:class:`.Table` that refers to it, which breaks in the case when the
sequence is also involved in a server-side default for that table, when
using :meth:`.MetaData.drop_all`. The step which processes sequences
to be dropped via non server-side column default functions is now invoked
after the table itself is dropped.
Mike Bayer [Sun, 8 Jul 2018 23:10:36 +0000 (19:10 -0400)]
Check tokens in chop path for inspectionattr before calling is_mapper
Fixed regression in 1.2.9 due to :ticket:`4287` where using a
:class::`.Load` option in conjunction with a string wildcard would result
in a TypeError.
Sean Dunn [Fri, 29 Jun 2018 14:26:57 +0000 (10:26 -0400)]
Add unique_constraint_name to MSSQL FK reflection
Fixed bug in MSSQL reflection where when two same-named tables in different
schemas had same-named primary key constraints, foreign key constraints
referring to one of the tables would have their columns doubled, causing
errors. Pull request courtesy Sean Dunn.
Mike Bayer [Thu, 28 Jun 2018 18:33:14 +0000 (14:33 -0400)]
Reflect ASC/DESC in MySQL index columns
Fixed bug in index reflection where on MySQL 8.0 an index that includes
ASC or DESC in an indexed column specfication would not be correctly
reflected, as MySQL 8.0 introduces support for returning this information
in a table definition string.
Mike Bayer [Thu, 28 Jun 2018 15:49:02 +0000 (11:49 -0400)]
Vendor python3 formatargspec
Replaced the usage of inspect.formatargspec() with a vendored version
copied from the Python standard library, as inspect.formatargspec()
is deprecated and as of Python 3.7.0 is emitting a warning.
Mike Bayer [Wed, 27 Jun 2018 20:08:23 +0000 (16:08 -0400)]
Try to get mysqlconnector somewhat working
Add CI support for MySQL connector and try to fix some of the
more obvious issues. CI tests will run against MySQL 5.7
only for starters as there appear to be issues with
MySQL 8.0
Mike Bayer [Tue, 26 Jun 2018 20:53:51 +0000 (16:53 -0400)]
Add do_setinputsizes event for cx_Oracle
Added a new event currently used only by the cx_Oracle dialect,
:meth:`.DialectEvents.setiputsizes`. The event passes a dictionary of
:class:`.BindParameter` objects to DBAPI-specific type objects that will be
passed, after conversion to parameter names, to the cx_Oracle
``cursor.setinputsizes()`` method. This allows both visibility into the
setinputsizes process as well as the ability to alter the behavior of what
datatypes are passed to this method.
Mike Bayer [Mon, 25 Jun 2018 02:50:06 +0000 (22:50 -0400)]
Ensure BakedQuery is cloned before we add options to it
Fixed bug in new polymorphic selectin loading where the BakedQuery used
internally would be mutated by the given loader options, which would both
inappropriately mutate the subclass query as well as carry over the effect
to subsequent queries.
Mike Bayer [Wed, 13 Jun 2018 22:13:21 +0000 (18:13 -0400)]
Look up adapter info for previous left side in chained query.join()
Fixed issue where chaining multiple join elements inside of
:meth:`.Query.join` might not correctly adapt to the previous left-hand
side, when chaining joined inheritance classes that share the same base
class.
Nils Philippsen [Sun, 24 Jun 2018 15:47:05 +0000 (11:47 -0400)]
fix TypeReflectionTest for sqlite 3.24
Fixed issue in test suite where SQLite 3.24 added a new reserved word that
conflicted with a usage in TypeReflectionTest. Pull request courtesy Nils
Philippsen.
Change-Id: I396562cecb5ca774f29e9234845bcc6a399fc5cb
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/452
(cherry picked from commit 40a5d0a4b006780167976b296984eb9790f3df7f)
Mike Bayer [Mon, 18 Jun 2018 14:12:56 +0000 (17:12 +0300)]
Fix UnboundLocalError in mssql during isolation level grab
Fixed issue within the SQL Server dialect under Python 3 where when running
against a non-standard SQL server database that does not contain either the
"sys.dm_exec_sessions" or "sys.dm_pdw_nodes_exec_sessions" views, leading
to a failure to fetch the isolation level, the error raise would fail due
to an UnboundLocalError.
Mike Bayer [Sun, 24 Jun 2018 17:06:38 +0000 (13:06 -0400)]
Use utf8mb4 (or utf8mb3) for all things MySQL
Fixed bug in MySQLdb dialect and variants such as PyMySQL where an
additional "unicode returns" check upon connection makes explicit use of
the "utf8" character set, which in MySQL 8.0 emits a warning that utf8mb4
should be used. This is now replaced with a utf8mb4 equivalent.
Documentation is also updated for the MySQL dialect to specify utf8mb4 in
all examples. Additional changes have been made to the test suite to use
utf8mb3 charsets and databases (there seem to be collation issues in some
edge cases with utf8mb4), and to support configuration default changes made
in MySQL 8.0 such as explicit_defaults_for_timestamp as well as new errors
raised for invalid MyISAM indexes.
Mike Bayer [Mon, 25 Jun 2018 04:23:54 +0000 (00:23 -0400)]
Compare mappers more accurately in Load._chop_path
Fixed bug in cache key generation for baked queries which could cause a
too-short cache key to be generated for the case of eager loads across
subclasses. This could in turn cause the eagerload query to be cached in
place of a non-eagerload query, or vice versa, for a polymorhic "selectin"
load, or possibly for lazy loads or selectin loads as well.
Mike Bayer [Wed, 13 Jun 2018 19:59:35 +0000 (15:59 -0400)]
Support JOIN in UPDATE..FROM
The :class:`.Update` construct now accommodates a :class:`.Join` object
as supported by MySQL for UPDATE..FROM. As the construct already
accepted an alias object for a similar purpose, the feature of UPDATE
against a non-table was already implied so this has been added.
Mike Bayer [Fri, 15 Jun 2018 02:17:00 +0000 (22:17 -0400)]
render WITH clause after INSERT for INSERT..SELECT on Oracle, MySQL
Fixed INSERT FROM SELECT with CTEs for the Oracle and MySQL dialects, where
the CTE was being placed above the entire statement as is typical with
other databases, however Oracle and MariaDB 10.2 wants the CTE underneath
the "INSERT" segment. Note that the Oracle and MySQL dialects don't yet
work when a CTE is applied to a subquery inside of an UPDATE or DELETE
statement, as the CTE is still applied to the top rather than inside the
subquery.
Also adds test suite support CTEs against backends.
Mike Bayer [Fri, 15 Jun 2018 02:56:21 +0000 (22:56 -0400)]
Lookup index columns in parent table by key for copy
Fixed regression in 1.2 due to :ticket:`4147` where a :class:`.Table` that
has had some of its indexed columns redefined with new ones, as would occur
when overriding columns during reflection or when using
:paramref:`.Table.extend_existing`, such that the :meth:`.Table.tometadata`
method would fail when attempting to copy those indexes as they still
referred to the replaced column. The copy logic now accommodates for this
condition.
Mike Bayer [Wed, 6 Jun 2018 20:35:34 +0000 (16:35 -0400)]
Iterate options per path for baked cache key
Fixed an issue that was both a performance regression in 1.2 as well as an
incorrect result regarding the "baked" lazy loader, involving the
generation of cache keys from the original :class:`.Query` object's loader
options. If the loader options were built up in a "branched" style using
common base elements for multiple options, the same options would be
rendered into the cache key repeatedly, causing both a performance issue as
well as generating the wrong cache key. This is fixed, along with a
performance improvement when such "branched" options are applied via
:meth:`.Query.options` to prevent the same option objects from being
applied repeatedly.
Mike Bayer [Mon, 28 May 2018 17:03:14 +0000 (13:03 -0400)]
Add Query.lazy_load_from attribute for sharding
Added new attribute :attr:`.Query.lazy_loaded_from` which is populated
with an :class:`.InstanceState` that is using this :class:`.Query` in
order to lazy load a relationship. The rationale for this is that
it serves as a hint for the horizontal sharding feature to use, such that
the identity token of the state can be used as the default identity token
to use for the query within id_chooser().
Also repaired an issue in the :meth:`.Result.with_post_criteria`
method added in I899808734458e25a023142c2c5bb37cbed869479
for :ticket:`4128` where the "unbake subquery loaders" version was calling
the post crtieria functions given the :class:`.Result` as the argument
rather than applying them to the :class:`.Query`.
Mike Bayer [Fri, 1 Jun 2018 21:54:11 +0000 (16:54 -0500)]
Support undocumented non-entity sequence Query argument
Fixed regression caused by :ticket:`4256` (itself a regression fix for
:ticket:`4228`) which breaks an undocumented behavior which converted for a
non-sequence of entities passed directly to the :class:`.Query` constructor
into a single-element sequence. While this behavior was never supported or
documented, it's already in use so has been added as a behavioral contract
to :class:`.Query`.
Mike Bayer [Sun, 27 May 2018 17:45:20 +0000 (13:45 -0400)]
Mutex on _CONFIGURE_MUTEX in automap.prepare()
Fixed a race condition which could occur if automap
:meth:`.AutomapBase.prepare` were used within a multi-threaded context
against other threads which may call :func:`.configure_mappers` as a
result of use of other mappers. The unfinished mapping work of automap
is particularly sensitive to being pulled in by a
:func:`.configure_mappers` step leading to errors.
Mike Bayer [Wed, 23 May 2018 20:22:48 +0000 (16:22 -0400)]
Turn oracle BINARY_DOUBLE, BINARY_FLOAT, DOUBLE_PRECISION into floats
The Oracle BINARY_FLOAT and BINARY_DOUBLE datatypes now participate within
cx_Oracle.setinputsizes(), passing along NATIVE_FLOAT, so as to support the
NaN value. Additionally, :class:`.oracle.BINARY_FLOAT`,
:class:`.oracle.BINARY_DOUBLE` and :class:`.oracle.DOUBLE_PRECISION` now
subclass :class:`.Float`, since these are floating point datatypes, not
decimal. These datatypes were already defaulting the
:paramref:`.Float.asdecimal` flag to False in line with what
:class:`.Float` already does.
Added reflection capabilities for the :class:`.oracle.BINARY_FLOAT`,
:class:`.oracle.BINARY_DOUBLE` datatypes.
Mike Bayer [Fri, 18 May 2018 16:51:40 +0000 (12:51 -0400)]
call setinputsizes() for integer types
Altered the Oracle dialect such that when an :class:`.Integer` type is in
use, the cx_Oracle.NUMERIC type is set up for setinputsizes(). In
SQLAlchemy 1.1 and earlier, cx_Oracle.NUMERIC was passed for all numeric
types unconditionally, and in 1.2 this was removed to allow for better
numeric precision. However, for integers, some database/client setups
will fail to coerce boolean values True/False into integers which introduces
regressive behavior when using SQLAlchemy 1.2. Overall, the setinputsizes
logic seems like it will need a lot more flexibility going forward so this
is a start for that.
Mike Bayer [Thu, 17 May 2018 23:57:01 +0000 (19:57 -0400)]
Skip for SQL Server on non-native boolean unconstrained
As SQL Server is now non-native boolean as of
I4765d2a2a00b0d14f50282603cc4d48d4739dac1 but uses the BIT
type, we need to constrain this test to continue to not
run against SQL Server.
Mike Bayer [Wed, 16 May 2018 20:31:40 +0000 (16:31 -0400)]
Update dogpile.cache example to be compatible with baked query.
Updated the dogpile.caching example to include new structures that
accommodate for the "baked" query system, which is used by default within
lazy loaders and some eager relationship loaders. The dogpile.caching
"relationship_caching" and "advanced" examples were also broken due to
:ticket:`4256`. The issue here is also worked-around by the fix in
:ticket:`4128`.
Note that this recipe requires
I3f86fcb12a6a9a89aa308b335e75c25969bcc30e in order for the
"advanced" example to work.
Miguel Ventura [Mon, 14 May 2018 20:56:58 +0000 (16:56 -0400)]
Fix string formatting TypeError if tuple is passed
Fixed issue where the "ambiguous literal" error message used when
interpreting literal values as SQL expression values would encounter a
tuple value, and fail to format the message properly. Pull request courtesy
Miguel Ventura.
Change-Id: I50d5d32d5f80ec79703a42d4b19b42c2f9701f24
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/442
(cherry picked from commit 0b0b58c938c6a38fccc3e0ba59876b3b6b4f8009)
Mike Bayer [Wed, 16 May 2018 15:16:57 +0000 (11:16 -0400)]
Change query._identity_lookup into a normal instance method
Fixed regression in 1.2.7 caused by :ticket:`4228`, which itself was fixing
a 1.2-level regression, where the ``query_cls`` callable passed to a
:class:`.Session` was assumed to be a subclass of :class:`.Query` with
class method availability, as opposed to an arbitrary callable. In
particular, the dogpile caching example illustrates ``query_cls`` as a
function and not a :class:`.Query` subclass.
Mike Bayer [Mon, 26 Feb 2018 00:54:37 +0000 (19:54 -0500)]
Merge existing query params in baked lazy load
Corrected backport, was supposed to be in 1.2.5 however
never got backported.
Fixed a long-standing regression that occurred in version
1.0, which prevented the use of a custom :class:`.MapperOption`
that alters the _params of a :class:`.Query` object for a
lazy load, since the lazy loader itself would overwrite those
parameters. This applies to the "temporal range" example
on the wiki. Note however that the
:meth:`.Query.populate_existing` method is now required in
order to rewrite the mapper options associated with an object
already loaded in the identity map.
As part of this change, a custom defined
:class:`.MapperOption` will now cause lazy loaders related to
the target object to use a non-baked query by default unless
the :meth:`.MapperOption._generate_cache_key` method is implemented.
In particular, this repairs one regression which occured when
using the dogpile.cache "advanced" example, which was not
returning cached results and instead emitting SQL due to an
incompatibility with the baked query loader; with the change,
the ``RelationshipCache`` option included for many releases
in the dogpile example will disable the "baked" query altogether.
Note that the dogpile example is also modernized to avoid both
of these issues as part of issue :ticket:`4258`.
Mike Bayer [Tue, 15 May 2018 20:15:51 +0000 (16:15 -0400)]
Prevent double-checkins and guard during reset-on-return invalidations
Fixed connection pool issue whereby if a disconnection error were raised
during the connection pool's "reset on return" sequence in conjunction with
an explicit transaction opened against the enclosing :class:`.Connection`
object (such as from calling :meth:`.Session.close` without a rollback or
commit, or calling :meth:`.Connection.close` without first closing a
transaction declared with :meth:`.Connection.begin`), a double-checkin would
result, which could then lead towards concurrent checkouts of the same
connection. The double-checkin condition is now prevented overall by an
assertion, as well as the specific double-checkin scenario has been
fixed.
Mike Bayer [Thu, 3 May 2018 16:35:23 +0000 (12:35 -0400)]
Use identity_token for refresh(), unexpire, undefer
The horizontal sharding extension now makes use of the identity token
added to ORM identity keys as part of :ticket:`4137`, when an object
refresh or column-based deferred load or unexpiration operation occurs.
Since we know the "shard" that the object originated from, we make
use of this value when refreshing, thereby avoiding queries against
other shards that don't match this object's identity in any case.
Mike Bayer [Fri, 11 May 2018 13:19:10 +0000 (09:19 -0400)]
Default server_version_info to (0, )
Fixed a bug in the test suite where if an external dialect returned
``None`` for ``server_version_info``, the exclusion logic would raise an
``AttributeError``.
Mike Bayer [Thu, 10 May 2018 15:39:06 +0000 (11:39 -0400)]
SQL Server is not native boolean; add new flag for CHECK constraint
Fixed a 1.2 regression caused by :ticket:`4061` where the SQL Server
"BIT" type would be considered to be "native boolean". The goal here
was to avoid creating a CHECK constraint on the column, however the bigger
issue is that the BIT value does not behave like a true/false constant
and cannot be interpreted as a standalone expression, e.g.
"WHERE <column>". The SQL Server dialect now goes back to being
non-native boolean, but with an extra flag that still avoids creating
the CHECK constraint.
Fixed a reference leak issue where the values of the parameter dictionary
used in a statement execution would remain referenced by the "compiled
cache", as a result of storing the key view used by Python 3 dictionary
keys(). Pull request courtesy Olivier Grisel.
Change-Id: Icfb0f38111a165780f6dd3e4e3382a03df79ce26
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/441
(cherry picked from commit 30b02003a70f37aa83e20de6229afe2a3600b648)
Mike Bayer [Fri, 20 Apr 2018 15:44:09 +0000 (11:44 -0400)]
Refactor "get" to allow for pluggable identity token schemes
Fixed regression in 1.2 within sharded query feature where the
new "identity_token" element was not being correctly considered within
the scope of a lazy load operation, when searching the identity map
for a related many-to-one element. The new behavior will allow for
making use of the "id_chooser" in order to determine the best identity
key to retrieve from the identity map. In order to achieve this, some
refactoring of 1.2's "identity_token" approach has made some slight changes
to the implementation of ``ShardedQuery`` which should be noted for other
derivations of this class.
Mike Bayer [Fri, 20 Apr 2018 17:31:45 +0000 (13:31 -0400)]
Document how to opt-out of NCHAR for cx_Oracle
Unfortunately, we need to bind Python unicode values as
NCHAR as in the case where non-ascii characters are present,
it's necessary. We can't know in all cases how this value is being
used, so in those cases where Oracle will not accept NCHAR the
user should explicitly cast a value down to String.
Mike Bayer [Thu, 19 Apr 2018 21:07:32 +0000 (17:07 -0400)]
Ensure select_from_entity adapter is used in adjust_for_single_inheritance
Fixed issue in single-inheritance loading where the use of an aliased
entity against a single-inheritance subclass in conjunction with the
:meth:`.Query.select_from` method would cause the SQL to be rendered with
the unaliased table mixed in to the query, causing a cartesian product. In
particular this was affecting the new "selectin" loader when used against a
single-inheritance subclass.
Kent Bower [Wed, 11 Apr 2018 21:21:26 +0000 (17:21 -0400)]
Reflect Oracle NUMBER(NULL, 0) as INTEGER
The Oracle NUMBER datatype is reflected as INTEGER if the precision is NULL
and the scale is zero, as this is how INTEGER values come back when
reflected from Oracle's tables. Pull request courtesy Kent Bower.
Change-Id: I4627febd46cab7085299c0a5700ee0f0bdca513c
Pull-request: https://github.com/zzzeek/sqlalchemy/pull/437
(cherry picked from commit a3473c08d35e2cce32b014519df5f774c0166cf1)
Mike Bayer [Wed, 11 Apr 2018 13:29:00 +0000 (09:29 -0400)]
Correct join for FKs with schema in SQL Server
Fixed 1.2 regression caused by :ticket:`4060` where the query used to
reflect SQL Server cross-schema foreign keys was limiting the criteria
incorrectly.
Additionally, added some rework of the inter-schema reflection tests
so that MySQL, MSSQL can be included, breaking out some of the
Postgresql-specific behaviors into separate requirements.
Mike Bayer [Wed, 4 Apr 2018 17:36:28 +0000 (13:36 -0400)]
Ensure all visit_sequence accepts **kw args
Fixed issue where the compilation of an INSERT statement with the
"literal_binds" option that also uses an explicit sequence and "inline"
generation, as on Postgresql and Oracle, would fail to accommodate the
extra keyword argument within the sequence processing routine.
Mike Bayer [Tue, 3 Apr 2018 19:35:00 +0000 (15:35 -0400)]
Use base __ne__ implementation for range types w/ None
Fixed bug where the special "not equals" operator for the Postgresql
"range" datatypes such as DATERANGE would fail to render "IS NOT NULL" when
compared to the Python ``None`` value.
Also break up range tests into backend round trip and straight
compilation suites.
Mike Bayer [Fri, 30 Mar 2018 15:22:30 +0000 (11:22 -0400)]
Filter non-integer characters from pyodbc SQL Server version
Adjusted the SQL Server version detection for pyodbc to only allow for
numeric tokens, filtering out non-integers, since the dialect does tuple-
numeric comparisons with this value. This is normally true for all known
SQL Server / pyodbc drivers in any case.
Mike Bayer [Wed, 28 Mar 2018 15:58:47 +0000 (11:58 -0400)]
Invalidate on failed connect handler
Fixed bug in connection pool where a connection could be present in the
pool without all of its "connect" event handlers called, if a previous
"connect" handler threw an exception; note that the dialects themselves
have connect handlers that emit SQL, such as those which set transaction
isolation, which can fail if the database is in a non-available state, but
still allows a connection. The connection is now invalidated first if any
of the connect handlers fail.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
