Mike Bayer [Fri, 6 Dec 2019 16:38:55 +0000 (11:38 -0500)]
Use label reference coercion for select() distinct keyword argument
Fixed bug where "distinct" keyword passed to :func:`.select` would not
treat a string value as a "label reference" in the same way that the
:meth:`.select.distinct` does; it would instead raise unconditionally. This
keyword argument and the others passed to :func:`.select` will ultimately
be deprecated for SQLAlchemy 2.0.
Mike Bayer [Fri, 6 Dec 2019 15:24:25 +0000 (10:24 -0500)]
Include DISTINCT in error message for label reference
Needed to add tests to ensure this label reference is handled
correctly, so also modified the exception message to
be more clear if someone has this error within distinct().
Mike Bayer [Thu, 5 Dec 2019 00:18:57 +0000 (19:18 -0500)]
Introduce lambda combinations
As the ORM's combinatoric tests mostly use entities and
table metadata that's defined in fixtures, we can't use
@testing.combinations directly as it takes place at the
module level. Instead we use lambdas, but to reduce
verbosity we use a code replacement so that the namespace
of the lambda can be provided at runtime rather than
module import time.
Mike Bayer [Mon, 2 Dec 2019 16:47:08 +0000 (11:47 -0500)]
Skip sub-minute timezone tests for Python 3 < 3.7
The datetime.timezone class as of Python 3.7, or SQLAlchemy's port for Python
2, supports seconds and microseconds. For Python 3.6 and earlier, it is not
supported.
Gord Thompson [Tue, 26 Nov 2019 14:57:48 +0000 (09:57 -0500)]
Add DATETIMEOFFSET support for mssql+pyodbc
Repaired support for the :class:`.mssql.DATETIMEOFFSET` datatype on PyODBC,
by adding PyODBC-level result handlers as it does not include native
support for this datatype. This includes usage of the Python 3 "timezone"
tzinfo subclass in order to set up a timezone, which on Python 2 makes
use of a minimal backport of "timezone" in sqlalchemy.util.
Mike Bayer [Fri, 29 Nov 2019 15:50:44 +0000 (10:50 -0500)]
Gracefully degrade for SQLite JSON receiving direct numeric value
Fixed issue to workaround SQLite's behavior of assigning "numeric" affinity
to JSON datatypes, first described at :ref:`change_3850`, which returns
scalar numeric JSON values as a number and not as a string that can be JSON
deserialized. The SQLite-specific JSON deserializer now gracefully
degrades for this case as an exception and bypasses deserialization for
single numeric values, as from a JSON perspective they are already
deserialized.
Also adds a combinatoric fixture for JSON single values within
the dialect-general test suite.
Mike Bayer [Tue, 26 Nov 2019 16:09:00 +0000 (11:09 -0500)]
Don't clear metadata in TablesTest setup
Some upcoming commits will make use of @metadata_fixture
within a TablesTest class. As the fixture takes place before
setup, remove anything in setup that would interfere with it.
Gord Thompson [Thu, 21 Nov 2019 14:43:40 +0000 (09:43 -0500)]
Add sequence support for MariaDB 10.3+.
Added support for use of the :class:`.Sequence` construct with MariaDB 10.3
and greater, as this is now supported by this database. The construct
integrates with the :class:`.Table` object in the same way that it does for
other databases like PostrgreSQL and Oracle; if is present on the integer
primary key "autoincrement" column, it is used to generate defaults. For
backwards compatibility, to support a :class:`.Table` that has a
:class:`.Sequence` on it to support sequence only databases like Oracle,
while still not having the sequence fire off for MariaDB, the optional=True
flag should be set, which indicates the sequence should only be used to
generate the primary key if the target database offers no other option.
Mike Bayer [Mon, 25 Nov 2019 20:09:47 +0000 (15:09 -0500)]
Remove ORM elements from annotations at the schema level.
Fixed issue where when constructing constraints from ORM-bound columns,
primarily :class:`.ForeignKey` objects but also :class:`.UniqueConstraint`,
:class:`.CheckConstraint` and others, the ORM-level
:class:`.InstrumentedAttribute` is discarded entirely, and all ORM-level
annotations from the columns are removed; this is so that the constraints
are still fully pickleable without the ORM-level entities being pulled in.
These annotations are not necessary to be present at the schema/metadata
level.
Fully implemented coercions for constraint columns within
schema.py, including for FK referenced columns.
Mike Bayer [Sat, 23 Nov 2019 17:25:20 +0000 (12:25 -0500)]
Generalize DescriptorProps.uses_objects
Previously, uses_objects was specific to the SynonymAttribute;
generalize it so that it defaults to False for other DescriptorProps.
Immediate fix is against CompositeProperty.
Fixed regression introduced in 1.3.0 related to the association proxy
refactor in :ticket:`4351` that prevented :func:`.composite` attributes
from working in terms of an association proxy that references them.
Add test coverage for association proxies that refer to Composite
attributes as endpoints.
Mike Bayer [Fri, 22 Nov 2019 19:28:21 +0000 (14:28 -0500)]
Raise for persistence casades set with viewonly=True
An error is raised if any persistence-related "cascade" settings are made
on a :func:`.relationship` that also sets up viewonly=True. The "cascade"
settings now default to non-persistence related settings only when viewonly
is also set. This is the continuation from :ticket:`4993` where this
setting was changed to emit a warning in 1.3.
Mike Bayer [Wed, 20 Nov 2019 17:15:57 +0000 (12:15 -0500)]
Warn for settings that don't work with viewonly=True
Setting persistence-related flags on :func:`.relationship` while also
setting viewonly=True will now emit a regular warning, as these flags do
not make sense for a viewonly=True relationship. In particular, the
"cascade" settings have their own warning that is generated based on the
individual values, such as "delete, delete-orphan", that should not apply
to a viewonly relationship. Note however that in the case of "cascade",
these settings are still erroneously taking effect even though the
relationship is set up as "viewonly". In 1.4, all persistence-related
cascade settings will be disallowed on a viewonly=True relationship in
order to resolve this issue.
Mike Bayer [Fri, 22 Nov 2019 15:27:02 +0000 (10:27 -0500)]
Don't raise w/ raiseload strategy for many-to-one history in flush
Fixed issue involving ``lazy="raise"`` strategy where an ORM delete of an
object would raise for a simple "use-get" style many-to-one relationship
that had lazy="raise" configured. This is inconsistent vs. the change
introduced in 1.3 as part of :ticket:`4353`, where it was established that
a history operation that does not expect emit SQL should bypass the
``lazy="raise"`` check, and instead effectively treat it as
``lazy="raise_on_sql"`` for this case. The fix adjusts the lazy loader
strategy to not raise for the case where the lazy load was instructed that
it should not emit SQL if the object were not present.
Mike Bayer [Tue, 19 Nov 2019 14:30:31 +0000 (09:30 -0500)]
Skip on slice assignment to self
Fixed issue where when assigning a collection to itself as a slice, the
mutation operation would fail as it would first erase the assigned
collection inadvertently. As an assignment that does not change the
contents should not generate events, the operation is now a no-op. Note
that the fix only applies to Python 3; in Python 2, the ``__setitem__``
hook isn't called in this case; ``__setslice__`` is used instead which
recreates the list item-by-item in all cases.
Mike Bayer [Wed, 13 Nov 2019 15:09:54 +0000 (10:09 -0500)]
Base mssql.DATETIMEOFFSET on DateTime
Fixed the base class of the :class:`.mssql.DATETIMEOFFSET` datatype to
be based on the :class:`.DateTime` class hierarchy, as this is a
datetime-holding datatype.
Note however that at the moment the pyodbc driver has only limited
support for this datatype and will not work with all ODBC drivers.
Mike Bayer [Mon, 11 Nov 2019 21:51:46 +0000 (16:51 -0500)]
Repair Oracle Interval
The :class:`.oracle.INTERVAL` class of the Oracle dialect is now correctly
a subclass of the abstract version of :class:`.Interval` as well as the
correct "emulated" base class, which allows for correct behavior under both
native and non-native modes; previously it was only based on
:class:`.TypeEngine`.
Mike Bayer [Mon, 11 Nov 2019 21:17:50 +0000 (16:17 -0500)]
Detect PyMySQL connection was killed
Added "Connection was killed" message interpreted from the base
pymysql.Error class in order to detect closed connection, based on reports
that this message is arriving via a pymysql.InternalError() object which
indicates pymysql is not handling it correctly.
Mike Bayer [Thu, 7 Nov 2019 20:31:48 +0000 (15:31 -0500)]
Add type accessors for JSON indexed/pathed element access
Added new accessors to expressions of type :class:`.JSON` to allow for
specific datatype access and comparison, covering strings, integers,
numeric, boolean elements. This revises the documented approach of
CASTing to string when comparing values, instead adding specific
functionality into the PostgreSQL, SQlite, MySQL dialects to reliably
deliver these basic types in all cases.
The change also delivers a new feature to the test exclusions
system so that combinations and exclusions can be used together.
Mike Bayer [Sun, 10 Nov 2019 20:42:40 +0000 (15:42 -0500)]
Exclude local columns when adapting secondary in a join condition
Fixed ORM bug where a "secondary" table that referred to a selectable which
in some way would refer to the local primary table would apply aliasing to
both sides of the join condition when a relationship-related join, either
via :meth:`.Query.join` or by :func:`.joinedload`, were generated. The
"local" side is now excluded.
Mike Bayer [Sat, 9 Nov 2019 17:33:16 +0000 (12:33 -0500)]
Support exclusion rules in combinations
Like py.test we need to be able to mark certain combination
elements with exclusion rules. Add additional logic
to pytestlplugin and exclusions so that the exclusion decorators
can be added to the combination tuples, where they will be applied
to the decorated function along with a qualifier that the test
arguments need to match what's given.
Mike Bayer [Sat, 9 Nov 2019 21:12:30 +0000 (16:12 -0500)]
Interpret empty LIMIT, expression LIMIT correctly
Fixed issue in MSSQL dialect where an expression-based OFFSET value in a
SELECT would be rejected, even though the dialect can render this
expression inside of a ROW NUMBER-oriented LIMIT/OFFSET construct.
CaselIT [Mon, 4 Nov 2019 22:11:21 +0000 (17:11 -0500)]
Support for generated columns
Added DDL support for "computed columns"; these are DDL column
specifications for columns that have a server-computed value, either upon
SELECT (known as "virtual") or at the point of which they are INSERTed or
UPDATEd (known as "stored"). Support is established for Postgresql, MySQL,
Oracle SQL Server and Firebird. Thanks to Federico Caselli for lots of work
on this one.
ORM round trip tests included. The ORM makes use of existing
FetchedValue support and no additional ORM logic is present for
the basic feature.
It has been observed that Oracle RETURNING does not return the
new value of a computed column upon UPDATE; it returns the
prior value. As this is very dangerous, a warning is emitted
if a computed column is rendered into the RETURNING clause
of an UPDATE statement.
Mike Bayer [Fri, 8 Nov 2019 14:48:27 +0000 (09:48 -0500)]
Work around setuptools issue #1902
Added a workaround for a setuptools-related failure that has been observed
as occurring on Windows installations, where setuptools is not correctly
reporting a build error when the MSVC build dependencies are not installed
and therefore not allowing graceful degradation into non C extensions
builds.
Mike Bayer [Thu, 29 Aug 2019 18:45:23 +0000 (14:45 -0400)]
Add anonymizing context to cache keys, comparison; convert traversal
Created new visitor system called "internal traversal" that
applies a data driven approach to the concept of a class that
defines its own traversal steps, in contrast to the existing
style of traversal now known as "external traversal" where
the visitor class defines the traversal, i.e. the SQLCompiler.
The internal traversal system now implements get_children(),
_copy_internals(), compare() and _cache_key() for most Core elements.
Core elements with special needs like Select still implement
some of these methods directly however most of these methods
are no longer explicitly implemented.
The data-driven system is also applied to ORM elements that
take part in SQL expressions so that these objects, like mappers,
aliasedclass, query options, etc. can all participate in the
cache key process.
Still not considered is that this approach to defining traversibility
will be used to create some kind of generic introspection system
that works across Core / ORM. It's also not clear if
real statement caching using the _cache_key() method is feasible,
if it is shown that running _cache_key() is nearly as expensive as
compiling in any case. Because it is data driven, it is more
straightforward to optimize using inlined code, as is the case now,
as well as potentially using C code to speed it up.
In addition, the caching sytem now accommodates for anonymous
name labels, which is essential so that constructs which have
anonymous labels can be cacheable, that is, their position
within a statement in relation to other anonymous names causes
them to generate an integer counter relative to that construct
which will be the same every time. Gathering of bound parameters
from any cache key generation is also now required as there is
no use case for a cache key that does not extract bound parameter
values.
Nils Philippsen [Sat, 2 Nov 2019 21:36:50 +0000 (17:36 -0400)]
Expect ordering NULLs to work and skip nested aggregate tests on sqlite >= 3.30.
Fixed test failures which would occur with newer SQLite as of version 3.30
or greater, due to their addition of nulls ordering syntax as well as new
restrictions on aggregate functions. Pull request courtesy Nils Philippsen.
Mike Bayer [Thu, 31 Oct 2019 14:30:46 +0000 (10:30 -0400)]
Improve SQL Server pyodbc documentation
While we were told years ago that ODBC is intended to be used with
DSNs only, however this use does not correspond well with how most
other database connectivity systems work in that modern systems
already have their own registries of connection information in any
case, meaning this is usually the best place to add details such
as hostnames and driver names, rather than having them locked away
in a server-specific ODBC registry. So here we dial back the
language that one style or another of connecting is "preferred";
both styles are supported equally, and the critical advantage of
hostname mapping in that the target database name is both explicit
as well as modifyable is also added.
Add additional background for how DSNs work and refine other
sentences. "URL encoding" is the correct terminology for
adding spaces and special characters to a URL.
Mike Bayer [Thu, 31 Oct 2019 13:30:12 +0000 (09:30 -0400)]
Cache every key in reflection_cache
Fixed bug in :class:`.Inspector` where the cache key generation did not
take into account arguments passed in the form of tuples, such as the tuple
of view name styles to return for the PostgreSQL dialect. This would lead
the inspector to cache too generally for a more specific set of criteria.
The logic has been adjusted to include every keyword element in the cache,
as every argument is expected to be appropriate for a cache else the
caching decorator should be bypassed by the dialect.
Mike Bayer [Wed, 30 Oct 2019 18:42:10 +0000 (14:42 -0400)]
omit_join=True is not supported
The :paramref:`.relationship.omit_join` flag was not intended to be
manually set to True, and will now emit a warning when this occurs. The
omit_join optimization is detected automatically, and the ``omit_join``
flag was only intended to disable the optimization in the hypothetical case
that the optimization may have interfered with correct results, which has
not been observed with the modern version of this feature. Setting the
flag to True when it is not automatically detected may cause the selectin
load feature to not work correctly when a non-default primary join
condition is in use.
sumau [Mon, 28 Oct 2019 19:22:08 +0000 (15:22 -0400)]
Use simple growth scale with any max size for BufferedRowResultProxy
The maximum buffer size for the :class:`.BufferedRowResultProxy`, which
is used by dialects such as PostgreSQL when ``stream_results=True``, can
now be set to a number greater than 1000 and the buffer will grow to
that size. Previously, the buffer would not go beyond 1000 even if the
value were set larger. The growth of the buffer is also now based
on a simple multiplying factor currently set to 5. Pull request courtesy
Soumaya Mauthoor.
lizraeli [Mon, 28 Oct 2019 19:33:41 +0000 (15:33 -0400)]
Correctly interpret None passed to query.get(); warn for empty PK values
A warning is emitted if a primary key value is passed to :meth:`.Query.get`
that consists of None for all primary key column positions. Previously,
passing a single None outside of a tuple would raise a ``TypeError`` and
passing a composite None (tuple of None values) would silently pass
through. The fix now coerces the single None into a tuple where it is
handled consistently with the other None conditions. Thanks to Lev
Izraelit for the help with this.
Pedro Cunial [Mon, 28 Oct 2019 19:27:44 +0000 (15:27 -0400)]
Remove redundant assignment in .../engine/create.py
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Remove a redundant assignment in the engine creation file.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Patrick Hayes [Thu, 24 Oct 2019 14:11:52 +0000 (10:11 -0400)]
Add public accessor `is_single_entity` to Query
Added accessor :attr:`.Query.is_single_entity` to :class:`.Query`, which
will indicate if the results returned by this :class:`.Query` will be a
list of ORM entities, or a tuple of entities or column expressions.
SQLAlchemy hopes to improve upon the behavior of single entity / tuples in
future releases such that the behavior would be explicit up front, however
this attribute should be helpful with the current behavior. Pull request
courtesy Patrick Hayes.
Mike Bayer [Fri, 25 Oct 2019 15:34:37 +0000 (11:34 -0400)]
Don't cache a query that has before_compile modifications
The :class:`.BakedQuery` will not cache a query that was modified by a
:meth:`.QueryEvents.before_compile` event, so that compilation hooks that
may be applying ad-hoc modifications to queries will take effect on each
run. In particular this is helpful for events that modify queries used in
lazy loading as well as eager loading such as "select in" loading. In
order to re-enable caching for a query modified by this event, a new
flag ``bake_ok`` is added; see :ref:`baked_with_before_compile` for
details.
A longer term plan to provide a new form of SQL caching should solve this
kind of issue more comprehensively.
Mike Bayer [Fri, 25 Oct 2019 14:08:18 +0000 (10:08 -0400)]
Warn that before_compile for lazyload needs bake_queries=False
The longer term future plan for ORM queries is that there
will be a new hook that receives queries before invocation
rather than "compilation", which will make use of a new
caching system.
Mike Bayer [Fri, 25 Oct 2019 02:29:28 +0000 (22:29 -0400)]
Disallow memory tests from running on Windows
These tests fail with multiprocess errors involving pickling
of the profile file. The memory tests are not critical
for windows development nor are the profiling tests overall
as they are against platform independent measurements.
Anders Kaseorg [Thu, 24 Oct 2019 18:27:57 +0000 (14:27 -0400)]
Fix Python 3.8 SyntaxWarning: "is not" with a literal
### Description
Fixes this warning from Python 3.8 in `Query._set_entities`:
```
lib/sqlalchemy/orm/query.py:179: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if entities is not ():
```
Fixes #4938.
### Checklist
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Wed, 23 Oct 2019 14:53:04 +0000 (10:53 -0400)]
Use default repr() for quoted_name under python 3
Changed the ``repr()`` of the :class:`.quoted_name` construct to use
regular string repr() under Python 3, rather than running it through
"backslashreplace" escaping, which can be misleading.
Modified the approach of "name normalization" for the Oracle and Firebird
dialects, which converts from the UPPERCASE-as-case-insensitive convention
of these dialects into lowercase-as-case-insensitive for SQLAlchemy, to not
automatically apply the :class:`.quoted_name` construct to a name that
matches itself under upper or lower case conversion, as is the case for
many non-european characters. All names used within metadata structures
are converted to :class:`.quoted_name` objects in any case; the change
here would only affect the output of some inspection functions.
Moved name normalize to be under default dialect, added test coverage
in test/sql/test_quote.py
Mike Bayer [Wed, 23 Oct 2019 15:18:56 +0000 (11:18 -0400)]
Add doc note for multiple table mapping
When mapping to a construct like OUTER JOIN, an UPDATE from the
ORM expects that all involved tables have a row already present;
document this as well as a potential workaround.
Mike Bayer [Mon, 12 Aug 2019 19:09:37 +0000 (15:09 -0400)]
Implment encoding_errors for cx_oracle
Added dialect-level flag ``encoding_errors`` to the cx_Oracle dialect,
which can be specified as part of :func:`.create_engine`. This is passed
to SQLAlchemy's unicode decoding converter under Python 2, and to
cx_Oracle's ``cursor.var()`` object as the ``encodingErrors`` parameter
under Python 3, for the very unusual case that broken encodings are present
in the target database which cannot be fetched unless error handling is
relaxed. The value is ultimately one of the Python "encoding errors"
parameters passed to ``decode()``.
Mike Bayer [Thu, 17 Oct 2019 17:09:24 +0000 (13:09 -0400)]
Implement facade for pytest parametrize, fixtures, classlevel
Add factilities to implement pytest.mark.parametrize and
pytest.fixtures patterns, which largely resemble things we are
already doing.
Ensure a facade is used, so that the test suite remains independent
of py.test, but also tailors the functions to the more limited
scope in which we are using them.
Additionally, create a class-based version that works from the
same facade.
Several old polymorphic tests as well as two of the sql test
are refactored to use the new features.
Gord Thompson [Sun, 20 Oct 2019 15:04:51 +0000 (11:04 -0400)]
add on_connect to MySQLDialect_pyodbc
Fixes: #4876
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
add on_connect to MySQLDialect_pyodbc to specify Unicode encoding/decoding settings for the pyodbc connection
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Fri, 18 Oct 2019 01:29:51 +0000 (21:29 -0400)]
Ensure SQL Server default schema name not interpreted as dot-separated tokens
Fixed an issue in the :meth:`.Engine.table_names` method where it would
feed the dialect's default schema name back into the dialect level table
function, which in the case of SQL Server would interpret it as a
dot-tokenized schema name as viewed by the mssql dialect, which would
cause the method to fail in the case where the database username actually
had a dot inside of it. In 1.3, this method is still used by the
:meth:`.MetaData.reflect` function so is a prominent codepath. In 1.4,
which is the current master development branch, this issue doesn't exist,
both because :meth:`.MetaData.reflect` isn't using this method nor does the
method pass the default schema name explicitly. The fix nonetheless
guards against the default server name value returned by the dialect from
being interpreted as dot-tokenized name under any circumstances by
wrapping it in quoted_name().
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
