Mike Bayer [Fri, 27 Dec 2019 19:10:36 +0000 (14:10 -0500)]
Note time passage requirement for pool.invalidate()
For Windows, time.time() may only have 16 millisecond
accuracy, so invalidation routines which compare
the time.time() of invalidate() to the time.time() when
the ConnectionRecord last connected may fail in a unit test
environment that does not pause at least this much time
since the ConnectionRecord startup. Using >= for comparison
instead of > was considered but this only leads to more confusing
results as the ConnecitonRecord goes into a re-connect loop
as time continues to not pass.
Overall, while using routines such as Python 3.7's time_ns()
might be helpful, for now make sure tests which rely on this
are marked under timing intensive and add small sleeps.
Mike Bayer [Mon, 9 Dec 2019 23:05:00 +0000 (18:05 -0500)]
Use expanding IN for all literal value IN expressions
The "expanding IN" feature, which generates IN expressions at query
execution time which are based on the particular parameters associated with
the statement execution, is now used for all IN expressions made against
lists of literal values. This allows IN expressions to be fully cacheable
independently of the list of values being passed, and also includes support
for empty lists. For any scenario where the IN expression contains
non-literal SQL expressions, the old behavior of pre-rendering for each
position in the IN is maintained. The change also completes support for
expanding IN with tuples, where previously type-specific bind processors
weren't taking effect.
As part of this change, a more explicit separation between
"literal execute" and "post compile" bound parameters is being made;
as the "ansi bind rules" feature is rendering bound parameters
inline, as we now support "postcompile" generically, these should
be used here, however we have to render literal values at
execution time even for "expanding" parameters. new test fixtures
etc. are added to assert everything goes to the right place.
Mike Bayer [Fri, 20 Dec 2019 16:40:10 +0000 (11:40 -0500)]
Copy bind_processors when altering for expanding IN
Fixed issue where the collection of value processors on a
:class:`.Compiled` object would be mutated when "expanding IN" parameters
were used with a datatype that has bind value processors; in particular,
this would mean that when using statement caching and/or baked queries, the
same compiled._bind_processors collection would be mutated concurrently.
Since these processors are the same function for a given bind parameter
namespace every time, there was no actual negative effect of this issue,
however, the execution of a :class:`.Compiled` object should never be
causing any changes in its state, especially given that they are intended
to be thread-safe and reusable once fully constructed.
Mike Bayer [Fri, 20 Dec 2019 15:17:17 +0000 (10:17 -0500)]
Ensure comparison includes "don't compare values" feature
upcoming changes for "expanding IN in all cases" and
"lambda elements" both rely upon comparisons that work
across changing bound values, so commit the testing fixture
ahead of time. Additionally, repair the feature itself
within traversals.
Mike Bayer [Thu, 19 Dec 2019 02:50:24 +0000 (21:50 -0500)]
Don't apply aliasing + adaption for simple relationship joins
Identified a performance issue in the system by which a join is constructed
based on a mapped relationship. The clause adaption system would be used
for the majority of join expressions including in the common case where no
adaptation is needed. The conditions under which this adaptation occur
have been refined so that average non-aliased joins along a simple
relationship without a "secondary" table use about 70% less function calls.
Mike Bayer [Thu, 19 Dec 2019 14:19:53 +0000 (09:19 -0500)]
Improve documentation for refresh_flush event
This event does not include INSERTed primary key columns
and additionally will always be limited to default/onupdate
columns. Note better choices for interception of INSERT
events.
Added support for prefixes to the :class:`.CTE` construct, to allow
support for Postgresql 12 "MATERIALIZED" and "NOT MATERIALIZED" phrases.
Pull request courtesy Marat Sharafutdinov.
Mike Bayer [Mon, 16 Dec 2019 21:38:06 +0000 (16:38 -0500)]
Do the CompoundSelect check for number of columns in the compile phase
Starting to go forward with the general idea of moving more
of Core / ORM construction into the compile phase. Bigger
initiatives like the refactor of Query will follow onto this.
Mike Bayer [Fri, 13 Dec 2019 17:44:23 +0000 (12:44 -0500)]
Close connection if begin fails
Fixed issue where by if the "begin" of a transaction failed at the Core
engine/connection level, such as due to network error or database is locked
for some transactional recipes, within the context of the :class:`.Session`
procuring that connection from the connection pool and then immediately
returning it, the ORM :class:`.Session` would not close the connection
despite this connection not being stored within the state of that
:class:`.Session`. This would lead to the connection being cleaned out by
the connection pool weakref handler within garbage collection which is an
unpreferred codepath that in some special configurations can emit errors in
standard error.
Mike Bayer [Fri, 6 Dec 2019 16:38:55 +0000 (11:38 -0500)]
Use label reference coercion for select() distinct keyword argument
Fixed bug where "distinct" keyword passed to :func:`.select` would not
treat a string value as a "label reference" in the same way that the
:meth:`.select.distinct` does; it would instead raise unconditionally. This
keyword argument and the others passed to :func:`.select` will ultimately
be deprecated for SQLAlchemy 2.0.
Mike Bayer [Fri, 6 Dec 2019 15:24:25 +0000 (10:24 -0500)]
Include DISTINCT in error message for label reference
Needed to add tests to ensure this label reference is handled
correctly, so also modified the exception message to
be more clear if someone has this error within distinct().
Mike Bayer [Thu, 5 Dec 2019 00:18:57 +0000 (19:18 -0500)]
Introduce lambda combinations
As the ORM's combinatoric tests mostly use entities and
table metadata that's defined in fixtures, we can't use
@testing.combinations directly as it takes place at the
module level. Instead we use lambdas, but to reduce
verbosity we use a code replacement so that the namespace
of the lambda can be provided at runtime rather than
module import time.
Mike Bayer [Mon, 2 Dec 2019 16:47:08 +0000 (11:47 -0500)]
Skip sub-minute timezone tests for Python 3 < 3.7
The datetime.timezone class as of Python 3.7, or SQLAlchemy's port for Python
2, supports seconds and microseconds. For Python 3.6 and earlier, it is not
supported.
Gord Thompson [Tue, 26 Nov 2019 14:57:48 +0000 (09:57 -0500)]
Add DATETIMEOFFSET support for mssql+pyodbc
Repaired support for the :class:`.mssql.DATETIMEOFFSET` datatype on PyODBC,
by adding PyODBC-level result handlers as it does not include native
support for this datatype. This includes usage of the Python 3 "timezone"
tzinfo subclass in order to set up a timezone, which on Python 2 makes
use of a minimal backport of "timezone" in sqlalchemy.util.
Mike Bayer [Fri, 29 Nov 2019 15:50:44 +0000 (10:50 -0500)]
Gracefully degrade for SQLite JSON receiving direct numeric value
Fixed issue to workaround SQLite's behavior of assigning "numeric" affinity
to JSON datatypes, first described at :ref:`change_3850`, which returns
scalar numeric JSON values as a number and not as a string that can be JSON
deserialized. The SQLite-specific JSON deserializer now gracefully
degrades for this case as an exception and bypasses deserialization for
single numeric values, as from a JSON perspective they are already
deserialized.
Also adds a combinatoric fixture for JSON single values within
the dialect-general test suite.
Mike Bayer [Tue, 26 Nov 2019 16:09:00 +0000 (11:09 -0500)]
Don't clear metadata in TablesTest setup
Some upcoming commits will make use of @metadata_fixture
within a TablesTest class. As the fixture takes place before
setup, remove anything in setup that would interfere with it.
Gord Thompson [Thu, 21 Nov 2019 14:43:40 +0000 (09:43 -0500)]
Add sequence support for MariaDB 10.3+.
Added support for use of the :class:`.Sequence` construct with MariaDB 10.3
and greater, as this is now supported by this database. The construct
integrates with the :class:`.Table` object in the same way that it does for
other databases like PostrgreSQL and Oracle; if is present on the integer
primary key "autoincrement" column, it is used to generate defaults. For
backwards compatibility, to support a :class:`.Table` that has a
:class:`.Sequence` on it to support sequence only databases like Oracle,
while still not having the sequence fire off for MariaDB, the optional=True
flag should be set, which indicates the sequence should only be used to
generate the primary key if the target database offers no other option.
Mike Bayer [Mon, 25 Nov 2019 20:09:47 +0000 (15:09 -0500)]
Remove ORM elements from annotations at the schema level.
Fixed issue where when constructing constraints from ORM-bound columns,
primarily :class:`.ForeignKey` objects but also :class:`.UniqueConstraint`,
:class:`.CheckConstraint` and others, the ORM-level
:class:`.InstrumentedAttribute` is discarded entirely, and all ORM-level
annotations from the columns are removed; this is so that the constraints
are still fully pickleable without the ORM-level entities being pulled in.
These annotations are not necessary to be present at the schema/metadata
level.
Fully implemented coercions for constraint columns within
schema.py, including for FK referenced columns.
Mike Bayer [Sat, 23 Nov 2019 17:25:20 +0000 (12:25 -0500)]
Generalize DescriptorProps.uses_objects
Previously, uses_objects was specific to the SynonymAttribute;
generalize it so that it defaults to False for other DescriptorProps.
Immediate fix is against CompositeProperty.
Fixed regression introduced in 1.3.0 related to the association proxy
refactor in :ticket:`4351` that prevented :func:`.composite` attributes
from working in terms of an association proxy that references them.
Add test coverage for association proxies that refer to Composite
attributes as endpoints.
Mike Bayer [Fri, 22 Nov 2019 19:28:21 +0000 (14:28 -0500)]
Raise for persistence casades set with viewonly=True
An error is raised if any persistence-related "cascade" settings are made
on a :func:`.relationship` that also sets up viewonly=True. The "cascade"
settings now default to non-persistence related settings only when viewonly
is also set. This is the continuation from :ticket:`4993` where this
setting was changed to emit a warning in 1.3.
Mike Bayer [Wed, 20 Nov 2019 17:15:57 +0000 (12:15 -0500)]
Warn for settings that don't work with viewonly=True
Setting persistence-related flags on :func:`.relationship` while also
setting viewonly=True will now emit a regular warning, as these flags do
not make sense for a viewonly=True relationship. In particular, the
"cascade" settings have their own warning that is generated based on the
individual values, such as "delete, delete-orphan", that should not apply
to a viewonly relationship. Note however that in the case of "cascade",
these settings are still erroneously taking effect even though the
relationship is set up as "viewonly". In 1.4, all persistence-related
cascade settings will be disallowed on a viewonly=True relationship in
order to resolve this issue.
Mike Bayer [Fri, 22 Nov 2019 15:27:02 +0000 (10:27 -0500)]
Don't raise w/ raiseload strategy for many-to-one history in flush
Fixed issue involving ``lazy="raise"`` strategy where an ORM delete of an
object would raise for a simple "use-get" style many-to-one relationship
that had lazy="raise" configured. This is inconsistent vs. the change
introduced in 1.3 as part of :ticket:`4353`, where it was established that
a history operation that does not expect emit SQL should bypass the
``lazy="raise"`` check, and instead effectively treat it as
``lazy="raise_on_sql"`` for this case. The fix adjusts the lazy loader
strategy to not raise for the case where the lazy load was instructed that
it should not emit SQL if the object were not present.
Mike Bayer [Tue, 19 Nov 2019 14:30:31 +0000 (09:30 -0500)]
Skip on slice assignment to self
Fixed issue where when assigning a collection to itself as a slice, the
mutation operation would fail as it would first erase the assigned
collection inadvertently. As an assignment that does not change the
contents should not generate events, the operation is now a no-op. Note
that the fix only applies to Python 3; in Python 2, the ``__setitem__``
hook isn't called in this case; ``__setslice__`` is used instead which
recreates the list item-by-item in all cases.
Mike Bayer [Wed, 13 Nov 2019 15:09:54 +0000 (10:09 -0500)]
Base mssql.DATETIMEOFFSET on DateTime
Fixed the base class of the :class:`.mssql.DATETIMEOFFSET` datatype to
be based on the :class:`.DateTime` class hierarchy, as this is a
datetime-holding datatype.
Note however that at the moment the pyodbc driver has only limited
support for this datatype and will not work with all ODBC drivers.
Mike Bayer [Mon, 11 Nov 2019 21:51:46 +0000 (16:51 -0500)]
Repair Oracle Interval
The :class:`.oracle.INTERVAL` class of the Oracle dialect is now correctly
a subclass of the abstract version of :class:`.Interval` as well as the
correct "emulated" base class, which allows for correct behavior under both
native and non-native modes; previously it was only based on
:class:`.TypeEngine`.
Mike Bayer [Mon, 11 Nov 2019 21:17:50 +0000 (16:17 -0500)]
Detect PyMySQL connection was killed
Added "Connection was killed" message interpreted from the base
pymysql.Error class in order to detect closed connection, based on reports
that this message is arriving via a pymysql.InternalError() object which
indicates pymysql is not handling it correctly.
Mike Bayer [Thu, 7 Nov 2019 20:31:48 +0000 (15:31 -0500)]
Add type accessors for JSON indexed/pathed element access
Added new accessors to expressions of type :class:`.JSON` to allow for
specific datatype access and comparison, covering strings, integers,
numeric, boolean elements. This revises the documented approach of
CASTing to string when comparing values, instead adding specific
functionality into the PostgreSQL, SQlite, MySQL dialects to reliably
deliver these basic types in all cases.
The change also delivers a new feature to the test exclusions
system so that combinations and exclusions can be used together.
Mike Bayer [Sun, 10 Nov 2019 20:42:40 +0000 (15:42 -0500)]
Exclude local columns when adapting secondary in a join condition
Fixed ORM bug where a "secondary" table that referred to a selectable which
in some way would refer to the local primary table would apply aliasing to
both sides of the join condition when a relationship-related join, either
via :meth:`.Query.join` or by :func:`.joinedload`, were generated. The
"local" side is now excluded.
Mike Bayer [Sat, 9 Nov 2019 17:33:16 +0000 (12:33 -0500)]
Support exclusion rules in combinations
Like py.test we need to be able to mark certain combination
elements with exclusion rules. Add additional logic
to pytestlplugin and exclusions so that the exclusion decorators
can be added to the combination tuples, where they will be applied
to the decorated function along with a qualifier that the test
arguments need to match what's given.
Mike Bayer [Sat, 9 Nov 2019 21:12:30 +0000 (16:12 -0500)]
Interpret empty LIMIT, expression LIMIT correctly
Fixed issue in MSSQL dialect where an expression-based OFFSET value in a
SELECT would be rejected, even though the dialect can render this
expression inside of a ROW NUMBER-oriented LIMIT/OFFSET construct.
CaselIT [Mon, 4 Nov 2019 22:11:21 +0000 (17:11 -0500)]
Support for generated columns
Added DDL support for "computed columns"; these are DDL column
specifications for columns that have a server-computed value, either upon
SELECT (known as "virtual") or at the point of which they are INSERTed or
UPDATEd (known as "stored"). Support is established for Postgresql, MySQL,
Oracle SQL Server and Firebird. Thanks to Federico Caselli for lots of work
on this one.
ORM round trip tests included. The ORM makes use of existing
FetchedValue support and no additional ORM logic is present for
the basic feature.
It has been observed that Oracle RETURNING does not return the
new value of a computed column upon UPDATE; it returns the
prior value. As this is very dangerous, a warning is emitted
if a computed column is rendered into the RETURNING clause
of an UPDATE statement.
Mike Bayer [Fri, 8 Nov 2019 14:48:27 +0000 (09:48 -0500)]
Work around setuptools issue #1902
Added a workaround for a setuptools-related failure that has been observed
as occurring on Windows installations, where setuptools is not correctly
reporting a build error when the MSVC build dependencies are not installed
and therefore not allowing graceful degradation into non C extensions
builds.
Mike Bayer [Thu, 29 Aug 2019 18:45:23 +0000 (14:45 -0400)]
Add anonymizing context to cache keys, comparison; convert traversal
Created new visitor system called "internal traversal" that
applies a data driven approach to the concept of a class that
defines its own traversal steps, in contrast to the existing
style of traversal now known as "external traversal" where
the visitor class defines the traversal, i.e. the SQLCompiler.
The internal traversal system now implements get_children(),
_copy_internals(), compare() and _cache_key() for most Core elements.
Core elements with special needs like Select still implement
some of these methods directly however most of these methods
are no longer explicitly implemented.
The data-driven system is also applied to ORM elements that
take part in SQL expressions so that these objects, like mappers,
aliasedclass, query options, etc. can all participate in the
cache key process.
Still not considered is that this approach to defining traversibility
will be used to create some kind of generic introspection system
that works across Core / ORM. It's also not clear if
real statement caching using the _cache_key() method is feasible,
if it is shown that running _cache_key() is nearly as expensive as
compiling in any case. Because it is data driven, it is more
straightforward to optimize using inlined code, as is the case now,
as well as potentially using C code to speed it up.
In addition, the caching sytem now accommodates for anonymous
name labels, which is essential so that constructs which have
anonymous labels can be cacheable, that is, their position
within a statement in relation to other anonymous names causes
them to generate an integer counter relative to that construct
which will be the same every time. Gathering of bound parameters
from any cache key generation is also now required as there is
no use case for a cache key that does not extract bound parameter
values.
Nils Philippsen [Sat, 2 Nov 2019 21:36:50 +0000 (17:36 -0400)]
Expect ordering NULLs to work and skip nested aggregate tests on sqlite >= 3.30.
Fixed test failures which would occur with newer SQLite as of version 3.30
or greater, due to their addition of nulls ordering syntax as well as new
restrictions on aggregate functions. Pull request courtesy Nils Philippsen.
Mike Bayer [Thu, 31 Oct 2019 14:30:46 +0000 (10:30 -0400)]
Improve SQL Server pyodbc documentation
While we were told years ago that ODBC is intended to be used with
DSNs only, however this use does not correspond well with how most
other database connectivity systems work in that modern systems
already have their own registries of connection information in any
case, meaning this is usually the best place to add details such
as hostnames and driver names, rather than having them locked away
in a server-specific ODBC registry. So here we dial back the
language that one style or another of connecting is "preferred";
both styles are supported equally, and the critical advantage of
hostname mapping in that the target database name is both explicit
as well as modifyable is also added.
Add additional background for how DSNs work and refine other
sentences. "URL encoding" is the correct terminology for
adding spaces and special characters to a URL.
Mike Bayer [Thu, 31 Oct 2019 13:30:12 +0000 (09:30 -0400)]
Cache every key in reflection_cache
Fixed bug in :class:`.Inspector` where the cache key generation did not
take into account arguments passed in the form of tuples, such as the tuple
of view name styles to return for the PostgreSQL dialect. This would lead
the inspector to cache too generally for a more specific set of criteria.
The logic has been adjusted to include every keyword element in the cache,
as every argument is expected to be appropriate for a cache else the
caching decorator should be bypassed by the dialect.
Mike Bayer [Wed, 30 Oct 2019 18:42:10 +0000 (14:42 -0400)]
omit_join=True is not supported
The :paramref:`.relationship.omit_join` flag was not intended to be
manually set to True, and will now emit a warning when this occurs. The
omit_join optimization is detected automatically, and the ``omit_join``
flag was only intended to disable the optimization in the hypothetical case
that the optimization may have interfered with correct results, which has
not been observed with the modern version of this feature. Setting the
flag to True when it is not automatically detected may cause the selectin
load feature to not work correctly when a non-default primary join
condition is in use.
sumau [Mon, 28 Oct 2019 19:22:08 +0000 (15:22 -0400)]
Use simple growth scale with any max size for BufferedRowResultProxy
The maximum buffer size for the :class:`.BufferedRowResultProxy`, which
is used by dialects such as PostgreSQL when ``stream_results=True``, can
now be set to a number greater than 1000 and the buffer will grow to
that size. Previously, the buffer would not go beyond 1000 even if the
value were set larger. The growth of the buffer is also now based
on a simple multiplying factor currently set to 5. Pull request courtesy
Soumaya Mauthoor.
lizraeli [Mon, 28 Oct 2019 19:33:41 +0000 (15:33 -0400)]
Correctly interpret None passed to query.get(); warn for empty PK values
A warning is emitted if a primary key value is passed to :meth:`.Query.get`
that consists of None for all primary key column positions. Previously,
passing a single None outside of a tuple would raise a ``TypeError`` and
passing a composite None (tuple of None values) would silently pass
through. The fix now coerces the single None into a tuple where it is
handled consistently with the other None conditions. Thanks to Lev
Izraelit for the help with this.
Pedro Cunial [Mon, 28 Oct 2019 19:27:44 +0000 (15:27 -0400)]
Remove redundant assignment in .../engine/create.py
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Remove a redundant assignment in the engine creation file.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Patrick Hayes [Thu, 24 Oct 2019 14:11:52 +0000 (10:11 -0400)]
Add public accessor `is_single_entity` to Query
Added accessor :attr:`.Query.is_single_entity` to :class:`.Query`, which
will indicate if the results returned by this :class:`.Query` will be a
list of ORM entities, or a tuple of entities or column expressions.
SQLAlchemy hopes to improve upon the behavior of single entity / tuples in
future releases such that the behavior would be explicit up front, however
this attribute should be helpful with the current behavior. Pull request
courtesy Patrick Hayes.
Mike Bayer [Fri, 25 Oct 2019 15:34:37 +0000 (11:34 -0400)]
Don't cache a query that has before_compile modifications
The :class:`.BakedQuery` will not cache a query that was modified by a
:meth:`.QueryEvents.before_compile` event, so that compilation hooks that
may be applying ad-hoc modifications to queries will take effect on each
run. In particular this is helpful for events that modify queries used in
lazy loading as well as eager loading such as "select in" loading. In
order to re-enable caching for a query modified by this event, a new
flag ``bake_ok`` is added; see :ref:`baked_with_before_compile` for
details.
A longer term plan to provide a new form of SQL caching should solve this
kind of issue more comprehensively.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
