CaselIT [Mon, 4 Nov 2019 22:11:21 +0000 (17:11 -0500)]
Support for generated columns
Added DDL support for "computed columns"; these are DDL column
specifications for columns that have a server-computed value, either upon
SELECT (known as "virtual") or at the point of which they are INSERTed or
UPDATEd (known as "stored"). Support is established for Postgresql, MySQL,
Oracle SQL Server and Firebird. Thanks to Federico Caselli for lots of work
on this one.
ORM round trip tests included. The ORM makes use of existing
FetchedValue support and no additional ORM logic is present for
the basic feature.
It has been observed that Oracle RETURNING does not return the
new value of a computed column upon UPDATE; it returns the
prior value. As this is very dangerous, a warning is emitted
if a computed column is rendered into the RETURNING clause
of an UPDATE statement.
Mike Bayer [Fri, 8 Nov 2019 14:48:27 +0000 (09:48 -0500)]
Work around setuptools issue #1902
Added a workaround for a setuptools-related failure that has been observed
as occurring on Windows installations, where setuptools is not correctly
reporting a build error when the MSVC build dependencies are not installed
and therefore not allowing graceful degradation into non C extensions
builds.
Mike Bayer [Thu, 29 Aug 2019 18:45:23 +0000 (14:45 -0400)]
Add anonymizing context to cache keys, comparison; convert traversal
Created new visitor system called "internal traversal" that
applies a data driven approach to the concept of a class that
defines its own traversal steps, in contrast to the existing
style of traversal now known as "external traversal" where
the visitor class defines the traversal, i.e. the SQLCompiler.
The internal traversal system now implements get_children(),
_copy_internals(), compare() and _cache_key() for most Core elements.
Core elements with special needs like Select still implement
some of these methods directly however most of these methods
are no longer explicitly implemented.
The data-driven system is also applied to ORM elements that
take part in SQL expressions so that these objects, like mappers,
aliasedclass, query options, etc. can all participate in the
cache key process.
Still not considered is that this approach to defining traversibility
will be used to create some kind of generic introspection system
that works across Core / ORM. It's also not clear if
real statement caching using the _cache_key() method is feasible,
if it is shown that running _cache_key() is nearly as expensive as
compiling in any case. Because it is data driven, it is more
straightforward to optimize using inlined code, as is the case now,
as well as potentially using C code to speed it up.
In addition, the caching sytem now accommodates for anonymous
name labels, which is essential so that constructs which have
anonymous labels can be cacheable, that is, their position
within a statement in relation to other anonymous names causes
them to generate an integer counter relative to that construct
which will be the same every time. Gathering of bound parameters
from any cache key generation is also now required as there is
no use case for a cache key that does not extract bound parameter
values.
Nils Philippsen [Sat, 2 Nov 2019 21:36:50 +0000 (17:36 -0400)]
Expect ordering NULLs to work and skip nested aggregate tests on sqlite >= 3.30.
Fixed test failures which would occur with newer SQLite as of version 3.30
or greater, due to their addition of nulls ordering syntax as well as new
restrictions on aggregate functions. Pull request courtesy Nils Philippsen.
Mike Bayer [Thu, 31 Oct 2019 14:30:46 +0000 (10:30 -0400)]
Improve SQL Server pyodbc documentation
While we were told years ago that ODBC is intended to be used with
DSNs only, however this use does not correspond well with how most
other database connectivity systems work in that modern systems
already have their own registries of connection information in any
case, meaning this is usually the best place to add details such
as hostnames and driver names, rather than having them locked away
in a server-specific ODBC registry. So here we dial back the
language that one style or another of connecting is "preferred";
both styles are supported equally, and the critical advantage of
hostname mapping in that the target database name is both explicit
as well as modifyable is also added.
Add additional background for how DSNs work and refine other
sentences. "URL encoding" is the correct terminology for
adding spaces and special characters to a URL.
Mike Bayer [Thu, 31 Oct 2019 13:30:12 +0000 (09:30 -0400)]
Cache every key in reflection_cache
Fixed bug in :class:`.Inspector` where the cache key generation did not
take into account arguments passed in the form of tuples, such as the tuple
of view name styles to return for the PostgreSQL dialect. This would lead
the inspector to cache too generally for a more specific set of criteria.
The logic has been adjusted to include every keyword element in the cache,
as every argument is expected to be appropriate for a cache else the
caching decorator should be bypassed by the dialect.
Mike Bayer [Wed, 30 Oct 2019 18:42:10 +0000 (14:42 -0400)]
omit_join=True is not supported
The :paramref:`.relationship.omit_join` flag was not intended to be
manually set to True, and will now emit a warning when this occurs. The
omit_join optimization is detected automatically, and the ``omit_join``
flag was only intended to disable the optimization in the hypothetical case
that the optimization may have interfered with correct results, which has
not been observed with the modern version of this feature. Setting the
flag to True when it is not automatically detected may cause the selectin
load feature to not work correctly when a non-default primary join
condition is in use.
sumau [Mon, 28 Oct 2019 19:22:08 +0000 (15:22 -0400)]
Use simple growth scale with any max size for BufferedRowResultProxy
The maximum buffer size for the :class:`.BufferedRowResultProxy`, which
is used by dialects such as PostgreSQL when ``stream_results=True``, can
now be set to a number greater than 1000 and the buffer will grow to
that size. Previously, the buffer would not go beyond 1000 even if the
value were set larger. The growth of the buffer is also now based
on a simple multiplying factor currently set to 5. Pull request courtesy
Soumaya Mauthoor.
lizraeli [Mon, 28 Oct 2019 19:33:41 +0000 (15:33 -0400)]
Correctly interpret None passed to query.get(); warn for empty PK values
A warning is emitted if a primary key value is passed to :meth:`.Query.get`
that consists of None for all primary key column positions. Previously,
passing a single None outside of a tuple would raise a ``TypeError`` and
passing a composite None (tuple of None values) would silently pass
through. The fix now coerces the single None into a tuple where it is
handled consistently with the other None conditions. Thanks to Lev
Izraelit for the help with this.
Pedro Cunial [Mon, 28 Oct 2019 19:27:44 +0000 (15:27 -0400)]
Remove redundant assignment in .../engine/create.py
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Remove a redundant assignment in the engine creation file.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Patrick Hayes [Thu, 24 Oct 2019 14:11:52 +0000 (10:11 -0400)]
Add public accessor `is_single_entity` to Query
Added accessor :attr:`.Query.is_single_entity` to :class:`.Query`, which
will indicate if the results returned by this :class:`.Query` will be a
list of ORM entities, or a tuple of entities or column expressions.
SQLAlchemy hopes to improve upon the behavior of single entity / tuples in
future releases such that the behavior would be explicit up front, however
this attribute should be helpful with the current behavior. Pull request
courtesy Patrick Hayes.
Mike Bayer [Fri, 25 Oct 2019 15:34:37 +0000 (11:34 -0400)]
Don't cache a query that has before_compile modifications
The :class:`.BakedQuery` will not cache a query that was modified by a
:meth:`.QueryEvents.before_compile` event, so that compilation hooks that
may be applying ad-hoc modifications to queries will take effect on each
run. In particular this is helpful for events that modify queries used in
lazy loading as well as eager loading such as "select in" loading. In
order to re-enable caching for a query modified by this event, a new
flag ``bake_ok`` is added; see :ref:`baked_with_before_compile` for
details.
A longer term plan to provide a new form of SQL caching should solve this
kind of issue more comprehensively.
Mike Bayer [Fri, 25 Oct 2019 14:08:18 +0000 (10:08 -0400)]
Warn that before_compile for lazyload needs bake_queries=False
The longer term future plan for ORM queries is that there
will be a new hook that receives queries before invocation
rather than "compilation", which will make use of a new
caching system.
Mike Bayer [Fri, 25 Oct 2019 02:29:28 +0000 (22:29 -0400)]
Disallow memory tests from running on Windows
These tests fail with multiprocess errors involving pickling
of the profile file. The memory tests are not critical
for windows development nor are the profiling tests overall
as they are against platform independent measurements.
Anders Kaseorg [Thu, 24 Oct 2019 18:27:57 +0000 (14:27 -0400)]
Fix Python 3.8 SyntaxWarning: "is not" with a literal
### Description
Fixes this warning from Python 3.8 in `Query._set_entities`:
```
lib/sqlalchemy/orm/query.py:179: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if entities is not ():
```
Fixes #4938.
### Checklist
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Wed, 23 Oct 2019 14:53:04 +0000 (10:53 -0400)]
Use default repr() for quoted_name under python 3
Changed the ``repr()`` of the :class:`.quoted_name` construct to use
regular string repr() under Python 3, rather than running it through
"backslashreplace" escaping, which can be misleading.
Modified the approach of "name normalization" for the Oracle and Firebird
dialects, which converts from the UPPERCASE-as-case-insensitive convention
of these dialects into lowercase-as-case-insensitive for SQLAlchemy, to not
automatically apply the :class:`.quoted_name` construct to a name that
matches itself under upper or lower case conversion, as is the case for
many non-european characters. All names used within metadata structures
are converted to :class:`.quoted_name` objects in any case; the change
here would only affect the output of some inspection functions.
Moved name normalize to be under default dialect, added test coverage
in test/sql/test_quote.py
Mike Bayer [Wed, 23 Oct 2019 15:18:56 +0000 (11:18 -0400)]
Add doc note for multiple table mapping
When mapping to a construct like OUTER JOIN, an UPDATE from the
ORM expects that all involved tables have a row already present;
document this as well as a potential workaround.
Mike Bayer [Mon, 12 Aug 2019 19:09:37 +0000 (15:09 -0400)]
Implment encoding_errors for cx_oracle
Added dialect-level flag ``encoding_errors`` to the cx_Oracle dialect,
which can be specified as part of :func:`.create_engine`. This is passed
to SQLAlchemy's unicode decoding converter under Python 2, and to
cx_Oracle's ``cursor.var()`` object as the ``encodingErrors`` parameter
under Python 3, for the very unusual case that broken encodings are present
in the target database which cannot be fetched unless error handling is
relaxed. The value is ultimately one of the Python "encoding errors"
parameters passed to ``decode()``.
Mike Bayer [Thu, 17 Oct 2019 17:09:24 +0000 (13:09 -0400)]
Implement facade for pytest parametrize, fixtures, classlevel
Add factilities to implement pytest.mark.parametrize and
pytest.fixtures patterns, which largely resemble things we are
already doing.
Ensure a facade is used, so that the test suite remains independent
of py.test, but also tailors the functions to the more limited
scope in which we are using them.
Additionally, create a class-based version that works from the
same facade.
Several old polymorphic tests as well as two of the sql test
are refactored to use the new features.
Gord Thompson [Sun, 20 Oct 2019 15:04:51 +0000 (11:04 -0400)]
add on_connect to MySQLDialect_pyodbc
Fixes: #4876
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
add on_connect to MySQLDialect_pyodbc to specify Unicode encoding/decoding settings for the pyodbc connection
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Fri, 18 Oct 2019 01:29:51 +0000 (21:29 -0400)]
Ensure SQL Server default schema name not interpreted as dot-separated tokens
Fixed an issue in the :meth:`.Engine.table_names` method where it would
feed the dialect's default schema name back into the dialect level table
function, which in the case of SQL Server would interpret it as a
dot-tokenized schema name as viewed by the mssql dialect, which would
cause the method to fail in the case where the database username actually
had a dot inside of it. In 1.3, this method is still used by the
:meth:`.MetaData.reflect` function so is a prominent codepath. In 1.4,
which is the current master development branch, this issue doesn't exist,
both because :meth:`.MetaData.reflect` isn't using this method nor does the
method pass the default schema name explicitly. The fix nonetheless
guards against the default server name value returned by the dialect from
being interpreted as dot-tokenized name under any circumstances by
wrapping it in quoted_name().
Mike Bayer [Fri, 11 Oct 2019 18:45:24 +0000 (14:45 -0400)]
Implement raiseload for deferred columns
Added "raiseload" feature for ORM mapped columns.
As part of this change, the behavior of "deferred" is now more strict;
an attribute that is set up as "deferred" at the mapper level no longer
participates in an "unexpire" operation; that is, when an unexpire loads
all the expired columns of an object which are not themselves in a deferred
group, those which are mapper-level deferred will never be loaded.
Deferral options set at query time should always be reset by an expiration
operation.
Renames deferred_scalar_loader to expired_attribute_loader
Unfortunately we can't have raiseload() do this because it would break
existing wildcard behavior.
Mike Bayer [Tue, 15 Oct 2019 01:53:44 +0000 (21:53 -0400)]
Use cx_Oracle.FIXED_NCHAR for sqltypes.NCHAR
The :class:`.sqltypes.NCHAR` datatype will now bind to the
``cx_Oracle.FIXED_NCHAR`` DBAPI data bindings when used in a bound
parameter, which supplies proper comparison behavior against a
variable-length string. Previously, the :class:`.sqltypes.NCHAR` datatype
would bind to ``cx_oracle.NCHAR`` which is not fixed length; the
:class:`.sqltypes.CHAR` datatype already binds to ``cx_Oracle.FIXED_CHAR``
so it is now consistent that :class:`.sqltypes.NCHAR` binds to
``cx_Oracle.FIXED_NCHAR``.
Mike Bayer [Mon, 14 Oct 2019 00:33:24 +0000 (20:33 -0400)]
Use separate label generator for column_label naming convention
Fixed bug where a table that would have a column label overlap with a plain
column name, such as "foo.id AS foo_id" vs. "foo.foo_id", would prematurely
generate the ``._label`` attribute for a column before this overlap could
be detected due to the use of the ``index=True`` or ``unique=True`` flag on
the column in conjunction with the default naming convention of
``"column_0_label"``. This would then lead to failures when ``._label``
were used later to generate a bound parameter name, in particular those
used by the ORM when generating the WHERE clause for an UPDATE statement.
The issue has been fixed by using an alternate ``._label`` accessor for DDL
generation that does not affect the state of the :class:`.Column`. The
accessor also bypasses the key-deduplication step as it is not necessary
for DDL, the naming is now consistently ``"<tablename>_<columnname>"``
without any subsequent numeric symbols when used in DDL.
Mike Bayer [Mon, 14 Oct 2019 03:02:22 +0000 (23:02 -0400)]
Add _alembic_quote method to format_constraint()
Alembic needs a portable way of getting at the name of an
index without quoting being applied. As we would like the
indexes created by the Column index=True flag to support
deferred index names, supply a function that delivers this
for Alembic without it having to dig too deeply into the
internals. the _alembic_quote flag may be made public
at a later time, however as we've been through many quoting
flags that are difficult to get rid of, try to be conservative
to start.
Mike Bayer [Wed, 9 Oct 2019 17:55:19 +0000 (13:55 -0400)]
pass executemany context to _repr_params
Fixed bug where parameter repr as used in logging and error reporting needs
additional context in order to distinguish between a list of parameters for
a single statement and a list of parameter lists, as the "list of lists"
structure could also indicate a single parameter list where the first
parameter itself is a list, such as for an array parameter. The
engine/connection now passes in an additional boolean indicating how the
parameters should be considered. The only SQLAlchemy backend that expects
arrays as parameters is that of psycopg2 which uses pyformat parameters,
so this issue has not been too apparent, however as other drivers that use
positional gain more features it is important that this be supported. It
also eliminates the need for the parameter repr function to guess based on
the parameter structure passed.
Mike Bayer [Wed, 9 Oct 2019 20:05:34 +0000 (16:05 -0400)]
Repair Oracle compat version check; dont warn if failed
Fixed regression in Oracle dialect that was inadvertently using max
identifier length of 128 characters on Oracle server 12.2 and greater even
though the stated contract for the remainder of the 1.3 series is that
this value stays at 30 until version SQLAlchemy 1.4. Also repaired issues
with the retrieval of the "compatibility" version, and removed the warning
emitted when the "v$parameter" view was not accessible as this was causing
user confusion.
Mike Bayer [Tue, 8 Oct 2019 20:42:21 +0000 (16:42 -0400)]
Omit onclause as source of FROMs from a Join
The :class:`.Join` construct no longer considers the "onclause" as a source
of additional FROM objects to be omitted from the FROM list of an enclosing
:class:`.Select` object as standalone FROM objects. This applies to an ON
clause that includes a reference to another FROM object outside the JOIN;
while this is usually not correct from a SQL perspective, it's also
incorrect for it to be omitted, and the behavioral change makes the
:class:`.Select` / :class:`.Join` behave a bit more intuitively.
Mike Bayer [Thu, 3 Oct 2019 21:36:27 +0000 (17:36 -0400)]
Add result map targeting for custom compiled, text objects
In order for text(), custom compiled objects, etc. to be usable
by Query(), they are all targeted by object key in the result map.
As we no longer want Query to implicitly label these, as well as that
text() has no label feature, support adding entries to the result
map that have no name, key, or type, only the object itself, and
then ensure that the compiler sets up for positional targeting
when this condition is detected.
Allows for more flexible ORM query usage with custom expressions
and text() while having less special logic in query itself.
Mike Bayer [Mon, 7 Oct 2019 20:12:30 +0000 (16:12 -0400)]
Fix max_identifier_length for SQL server
Fixed bug in SQL Server dialect with new "max_identifier_length" feature
where the mssql dialect already featured this flag, and the implementation
did not accommodate for the new initialization hook correctly.
Mike Bayer [Mon, 7 Oct 2019 19:25:27 +0000 (15:25 -0400)]
Drop right-nested join rewriting
Dropped support for right-nested join rewriting to support old SQLite
versions prior to 3.7.16, released in 2013. It is expected that
all modern Python versions among those now supported should all include
much newer versions of SQLite.
Mike Bayer [Sat, 5 Oct 2019 22:27:44 +0000 (18:27 -0400)]
create second level deduping when use_labels is turned on
As of #4753 we allow duplicate columns. This creates some new
problems that there can be duplicate columns in a subquery
which are then not addressible on the outside because they
are ambiguous (Postgresql has this behavior at least). Additionally
it creates situations where we are making an anon label of an
anon label which is leaking into the query.
New logic for generating anon labels handles this situation and
also alters the .c collection
of a subquery such that we are only getting the first column
from the derived selectable that has that name, the subsequent ones
have a new deduping label with two underscores and are not exposed
in .c. The dedupe logic when rendering the columns will handle
duplicate label names for different columns, vs. the same column
repeated, as separate cases.
Mike Bayer [Sun, 6 Oct 2019 01:28:48 +0000 (21:28 -0400)]
Remove deprecated extension and similar classes
All long-deprecated "extension" classes have been removed, including
MapperExtension, SessionExtension, PoolListener, ConnectionProxy,
AttributExtension. These classes have been deprecated since version 0.7
long superseded by the event listener system.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
