Mike Bayer [Thu, 31 Oct 2019 14:30:46 +0000 (10:30 -0400)]
Improve SQL Server pyodbc documentation
While we were told years ago that ODBC is intended to be used with
DSNs only, however this use does not correspond well with how most
other database connectivity systems work in that modern systems
already have their own registries of connection information in any
case, meaning this is usually the best place to add details such
as hostnames and driver names, rather than having them locked away
in a server-specific ODBC registry. So here we dial back the
language that one style or another of connecting is "preferred";
both styles are supported equally, and the critical advantage of
hostname mapping in that the target database name is both explicit
as well as modifyable is also added.
Add additional background for how DSNs work and refine other
sentences. "URL encoding" is the correct terminology for
adding spaces and special characters to a URL.
Mike Bayer [Thu, 31 Oct 2019 13:30:12 +0000 (09:30 -0400)]
Cache every key in reflection_cache
Fixed bug in :class:`.Inspector` where the cache key generation did not
take into account arguments passed in the form of tuples, such as the tuple
of view name styles to return for the PostgreSQL dialect. This would lead
the inspector to cache too generally for a more specific set of criteria.
The logic has been adjusted to include every keyword element in the cache,
as every argument is expected to be appropriate for a cache else the
caching decorator should be bypassed by the dialect.
Mike Bayer [Wed, 30 Oct 2019 18:42:10 +0000 (14:42 -0400)]
omit_join=True is not supported
The :paramref:`.relationship.omit_join` flag was not intended to be
manually set to True, and will now emit a warning when this occurs. The
omit_join optimization is detected automatically, and the ``omit_join``
flag was only intended to disable the optimization in the hypothetical case
that the optimization may have interfered with correct results, which has
not been observed with the modern version of this feature. Setting the
flag to True when it is not automatically detected may cause the selectin
load feature to not work correctly when a non-default primary join
condition is in use.
sumau [Mon, 28 Oct 2019 19:22:08 +0000 (15:22 -0400)]
Use simple growth scale with any max size for BufferedRowResultProxy
The maximum buffer size for the :class:`.BufferedRowResultProxy`, which
is used by dialects such as PostgreSQL when ``stream_results=True``, can
now be set to a number greater than 1000 and the buffer will grow to
that size. Previously, the buffer would not go beyond 1000 even if the
value were set larger. The growth of the buffer is also now based
on a simple multiplying factor currently set to 5. Pull request courtesy
Soumaya Mauthoor.
lizraeli [Mon, 28 Oct 2019 19:33:41 +0000 (15:33 -0400)]
Correctly interpret None passed to query.get(); warn for empty PK values
A warning is emitted if a primary key value is passed to :meth:`.Query.get`
that consists of None for all primary key column positions. Previously,
passing a single None outside of a tuple would raise a ``TypeError`` and
passing a composite None (tuple of None values) would silently pass
through. The fix now coerces the single None into a tuple where it is
handled consistently with the other None conditions. Thanks to Lev
Izraelit for the help with this.
Pedro Cunial [Mon, 28 Oct 2019 19:27:44 +0000 (15:27 -0400)]
Remove redundant assignment in .../engine/create.py
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Remove a redundant assignment in the engine creation file.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Patrick Hayes [Thu, 24 Oct 2019 14:11:52 +0000 (10:11 -0400)]
Add public accessor `is_single_entity` to Query
Added accessor :attr:`.Query.is_single_entity` to :class:`.Query`, which
will indicate if the results returned by this :class:`.Query` will be a
list of ORM entities, or a tuple of entities or column expressions.
SQLAlchemy hopes to improve upon the behavior of single entity / tuples in
future releases such that the behavior would be explicit up front, however
this attribute should be helpful with the current behavior. Pull request
courtesy Patrick Hayes.
Mike Bayer [Fri, 25 Oct 2019 15:34:37 +0000 (11:34 -0400)]
Don't cache a query that has before_compile modifications
The :class:`.BakedQuery` will not cache a query that was modified by a
:meth:`.QueryEvents.before_compile` event, so that compilation hooks that
may be applying ad-hoc modifications to queries will take effect on each
run. In particular this is helpful for events that modify queries used in
lazy loading as well as eager loading such as "select in" loading. In
order to re-enable caching for a query modified by this event, a new
flag ``bake_ok`` is added; see :ref:`baked_with_before_compile` for
details.
A longer term plan to provide a new form of SQL caching should solve this
kind of issue more comprehensively.
Mike Bayer [Fri, 25 Oct 2019 14:08:18 +0000 (10:08 -0400)]
Warn that before_compile for lazyload needs bake_queries=False
The longer term future plan for ORM queries is that there
will be a new hook that receives queries before invocation
rather than "compilation", which will make use of a new
caching system.
Mike Bayer [Fri, 25 Oct 2019 02:29:28 +0000 (22:29 -0400)]
Disallow memory tests from running on Windows
These tests fail with multiprocess errors involving pickling
of the profile file. The memory tests are not critical
for windows development nor are the profiling tests overall
as they are against platform independent measurements.
Anders Kaseorg [Thu, 24 Oct 2019 18:27:57 +0000 (14:27 -0400)]
Fix Python 3.8 SyntaxWarning: "is not" with a literal
### Description
Fixes this warning from Python 3.8 in `Query._set_entities`:
```
lib/sqlalchemy/orm/query.py:179: SyntaxWarning: "is not" with a literal. Did you mean "!="?
if entities is not ():
```
Fixes #4938.
### Checklist
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Wed, 23 Oct 2019 14:53:04 +0000 (10:53 -0400)]
Use default repr() for quoted_name under python 3
Changed the ``repr()`` of the :class:`.quoted_name` construct to use
regular string repr() under Python 3, rather than running it through
"backslashreplace" escaping, which can be misleading.
Modified the approach of "name normalization" for the Oracle and Firebird
dialects, which converts from the UPPERCASE-as-case-insensitive convention
of these dialects into lowercase-as-case-insensitive for SQLAlchemy, to not
automatically apply the :class:`.quoted_name` construct to a name that
matches itself under upper or lower case conversion, as is the case for
many non-european characters. All names used within metadata structures
are converted to :class:`.quoted_name` objects in any case; the change
here would only affect the output of some inspection functions.
Moved name normalize to be under default dialect, added test coverage
in test/sql/test_quote.py
Mike Bayer [Wed, 23 Oct 2019 15:18:56 +0000 (11:18 -0400)]
Add doc note for multiple table mapping
When mapping to a construct like OUTER JOIN, an UPDATE from the
ORM expects that all involved tables have a row already present;
document this as well as a potential workaround.
Mike Bayer [Mon, 12 Aug 2019 19:09:37 +0000 (15:09 -0400)]
Implment encoding_errors for cx_oracle
Added dialect-level flag ``encoding_errors`` to the cx_Oracle dialect,
which can be specified as part of :func:`.create_engine`. This is passed
to SQLAlchemy's unicode decoding converter under Python 2, and to
cx_Oracle's ``cursor.var()`` object as the ``encodingErrors`` parameter
under Python 3, for the very unusual case that broken encodings are present
in the target database which cannot be fetched unless error handling is
relaxed. The value is ultimately one of the Python "encoding errors"
parameters passed to ``decode()``.
Mike Bayer [Thu, 17 Oct 2019 17:09:24 +0000 (13:09 -0400)]
Implement facade for pytest parametrize, fixtures, classlevel
Add factilities to implement pytest.mark.parametrize and
pytest.fixtures patterns, which largely resemble things we are
already doing.
Ensure a facade is used, so that the test suite remains independent
of py.test, but also tailors the functions to the more limited
scope in which we are using them.
Additionally, create a class-based version that works from the
same facade.
Several old polymorphic tests as well as two of the sql test
are refactored to use the new features.
Gord Thompson [Sun, 20 Oct 2019 15:04:51 +0000 (11:04 -0400)]
add on_connect to MySQLDialect_pyodbc
Fixes: #4876
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
add on_connect to MySQLDialect_pyodbc to specify Unicode encoding/decoding settings for the pyodbc connection
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Fri, 18 Oct 2019 01:29:51 +0000 (21:29 -0400)]
Ensure SQL Server default schema name not interpreted as dot-separated tokens
Fixed an issue in the :meth:`.Engine.table_names` method where it would
feed the dialect's default schema name back into the dialect level table
function, which in the case of SQL Server would interpret it as a
dot-tokenized schema name as viewed by the mssql dialect, which would
cause the method to fail in the case where the database username actually
had a dot inside of it. In 1.3, this method is still used by the
:meth:`.MetaData.reflect` function so is a prominent codepath. In 1.4,
which is the current master development branch, this issue doesn't exist,
both because :meth:`.MetaData.reflect` isn't using this method nor does the
method pass the default schema name explicitly. The fix nonetheless
guards against the default server name value returned by the dialect from
being interpreted as dot-tokenized name under any circumstances by
wrapping it in quoted_name().
Mike Bayer [Fri, 11 Oct 2019 18:45:24 +0000 (14:45 -0400)]
Implement raiseload for deferred columns
Added "raiseload" feature for ORM mapped columns.
As part of this change, the behavior of "deferred" is now more strict;
an attribute that is set up as "deferred" at the mapper level no longer
participates in an "unexpire" operation; that is, when an unexpire loads
all the expired columns of an object which are not themselves in a deferred
group, those which are mapper-level deferred will never be loaded.
Deferral options set at query time should always be reset by an expiration
operation.
Renames deferred_scalar_loader to expired_attribute_loader
Unfortunately we can't have raiseload() do this because it would break
existing wildcard behavior.
Mike Bayer [Tue, 15 Oct 2019 01:53:44 +0000 (21:53 -0400)]
Use cx_Oracle.FIXED_NCHAR for sqltypes.NCHAR
The :class:`.sqltypes.NCHAR` datatype will now bind to the
``cx_Oracle.FIXED_NCHAR`` DBAPI data bindings when used in a bound
parameter, which supplies proper comparison behavior against a
variable-length string. Previously, the :class:`.sqltypes.NCHAR` datatype
would bind to ``cx_oracle.NCHAR`` which is not fixed length; the
:class:`.sqltypes.CHAR` datatype already binds to ``cx_Oracle.FIXED_CHAR``
so it is now consistent that :class:`.sqltypes.NCHAR` binds to
``cx_Oracle.FIXED_NCHAR``.
Mike Bayer [Mon, 14 Oct 2019 00:33:24 +0000 (20:33 -0400)]
Use separate label generator for column_label naming convention
Fixed bug where a table that would have a column label overlap with a plain
column name, such as "foo.id AS foo_id" vs. "foo.foo_id", would prematurely
generate the ``._label`` attribute for a column before this overlap could
be detected due to the use of the ``index=True`` or ``unique=True`` flag on
the column in conjunction with the default naming convention of
``"column_0_label"``. This would then lead to failures when ``._label``
were used later to generate a bound parameter name, in particular those
used by the ORM when generating the WHERE clause for an UPDATE statement.
The issue has been fixed by using an alternate ``._label`` accessor for DDL
generation that does not affect the state of the :class:`.Column`. The
accessor also bypasses the key-deduplication step as it is not necessary
for DDL, the naming is now consistently ``"<tablename>_<columnname>"``
without any subsequent numeric symbols when used in DDL.
Mike Bayer [Mon, 14 Oct 2019 03:02:22 +0000 (23:02 -0400)]
Add _alembic_quote method to format_constraint()
Alembic needs a portable way of getting at the name of an
index without quoting being applied. As we would like the
indexes created by the Column index=True flag to support
deferred index names, supply a function that delivers this
for Alembic without it having to dig too deeply into the
internals. the _alembic_quote flag may be made public
at a later time, however as we've been through many quoting
flags that are difficult to get rid of, try to be conservative
to start.
Mike Bayer [Wed, 9 Oct 2019 17:55:19 +0000 (13:55 -0400)]
pass executemany context to _repr_params
Fixed bug where parameter repr as used in logging and error reporting needs
additional context in order to distinguish between a list of parameters for
a single statement and a list of parameter lists, as the "list of lists"
structure could also indicate a single parameter list where the first
parameter itself is a list, such as for an array parameter. The
engine/connection now passes in an additional boolean indicating how the
parameters should be considered. The only SQLAlchemy backend that expects
arrays as parameters is that of psycopg2 which uses pyformat parameters,
so this issue has not been too apparent, however as other drivers that use
positional gain more features it is important that this be supported. It
also eliminates the need for the parameter repr function to guess based on
the parameter structure passed.
Mike Bayer [Wed, 9 Oct 2019 20:05:34 +0000 (16:05 -0400)]
Repair Oracle compat version check; dont warn if failed
Fixed regression in Oracle dialect that was inadvertently using max
identifier length of 128 characters on Oracle server 12.2 and greater even
though the stated contract for the remainder of the 1.3 series is that
this value stays at 30 until version SQLAlchemy 1.4. Also repaired issues
with the retrieval of the "compatibility" version, and removed the warning
emitted when the "v$parameter" view was not accessible as this was causing
user confusion.
Mike Bayer [Tue, 8 Oct 2019 20:42:21 +0000 (16:42 -0400)]
Omit onclause as source of FROMs from a Join
The :class:`.Join` construct no longer considers the "onclause" as a source
of additional FROM objects to be omitted from the FROM list of an enclosing
:class:`.Select` object as standalone FROM objects. This applies to an ON
clause that includes a reference to another FROM object outside the JOIN;
while this is usually not correct from a SQL perspective, it's also
incorrect for it to be omitted, and the behavioral change makes the
:class:`.Select` / :class:`.Join` behave a bit more intuitively.
Mike Bayer [Thu, 3 Oct 2019 21:36:27 +0000 (17:36 -0400)]
Add result map targeting for custom compiled, text objects
In order for text(), custom compiled objects, etc. to be usable
by Query(), they are all targeted by object key in the result map.
As we no longer want Query to implicitly label these, as well as that
text() has no label feature, support adding entries to the result
map that have no name, key, or type, only the object itself, and
then ensure that the compiler sets up for positional targeting
when this condition is detected.
Allows for more flexible ORM query usage with custom expressions
and text() while having less special logic in query itself.
Mike Bayer [Mon, 7 Oct 2019 20:12:30 +0000 (16:12 -0400)]
Fix max_identifier_length for SQL server
Fixed bug in SQL Server dialect with new "max_identifier_length" feature
where the mssql dialect already featured this flag, and the implementation
did not accommodate for the new initialization hook correctly.
Mike Bayer [Mon, 7 Oct 2019 19:25:27 +0000 (15:25 -0400)]
Drop right-nested join rewriting
Dropped support for right-nested join rewriting to support old SQLite
versions prior to 3.7.16, released in 2013. It is expected that
all modern Python versions among those now supported should all include
much newer versions of SQLite.
Mike Bayer [Sat, 5 Oct 2019 22:27:44 +0000 (18:27 -0400)]
create second level deduping when use_labels is turned on
As of #4753 we allow duplicate columns. This creates some new
problems that there can be duplicate columns in a subquery
which are then not addressible on the outside because they
are ambiguous (Postgresql has this behavior at least). Additionally
it creates situations where we are making an anon label of an
anon label which is leaking into the query.
New logic for generating anon labels handles this situation and
also alters the .c collection
of a subquery such that we are only getting the first column
from the derived selectable that has that name, the subsequent ones
have a new deduping label with two underscores and are not exposed
in .c. The dedupe logic when rendering the columns will handle
duplicate label names for different columns, vs. the same column
repeated, as separate cases.
Mike Bayer [Sun, 6 Oct 2019 01:28:48 +0000 (21:28 -0400)]
Remove deprecated extension and similar classes
All long-deprecated "extension" classes have been removed, including
MapperExtension, SessionExtension, PoolListener, ConnectionProxy,
AttributExtension. These classes have been deprecated since version 0.7
long superseded by the event listener system.
Mike Bayer [Tue, 1 Oct 2019 21:38:41 +0000 (17:38 -0400)]
Deprecate textual column matching in Row
Deprecate query.instances() without a context
Deprecate string alias with contains_eager()
Deprecated the behavior by which a :class:`.Column` can be used as the key
in a result set row lookup, when that :class:`.Column` is not part of the
SQL selectable that is being selected; that is, it is only matched on name.
A deprecation warning is now emitted for this case. Various ORM use
cases, such as those involving :func:`.text` constructs, have been improved
so that this fallback logic is avoided in most cases.
Calling the :meth:`.Query.instances` method without passing a
:class:`.QueryContext` is deprecated. The original use case for this was
that a :class:`.Query` could yield ORM objects when given only the entities
to be selected as well as a DBAPI cursor object. However, for this to work
correctly there is essential metadata that is passed from a SQLAlchemy
:class:`.ResultProxy` that is derived from the mapped column expressions,
which comes originally from the :class:`.QueryContext`. To retrieve ORM
results from arbitrary SELECT statements, the :meth:`.Query.from_statement`
method should be used.
Note there is a small bump in test_zoomark because the
column._label is being calculated for each of those columns within
baseline_3_properties, as it is now part of the result map.
This label can't be calculated when the column is attached
to the table because it needs to have all the columns present
to do this correctly. Another approach here would be to
pre-load the _label before the test runs however the zoomark
tests don't have an easy place for this to happen and it's
not really worth it.
Mike Bayer [Fri, 4 Oct 2019 15:12:27 +0000 (11:12 -0400)]
Warn for object replaced in identity map during flush
A warning is emitted for a condition in which the :class:`.Session` may
implicitly swap an object out of the identity map for another one with the
same primary key, detaching the old one, which can be an observed result of
load operations which occur within the :meth:`.SessionEvents.after_flush`
hook. The warning is intended to notify the user that some special
condition has caused this to happen and that the previous object may not be
in the expected state.
Gord Thompson [Thu, 3 Oct 2019 22:18:54 +0000 (18:18 -0400)]
fix error in test_round_trip for TimeTest with mysql+pyodbc
Fixes: #4879
### Description
create mysql+pyodbc-specific `_pyodbcTIME` class to avoid error thrown by `result_processor` in the (more) generic mysql/types.py.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Thu, 3 Oct 2019 18:09:54 +0000 (14:09 -0400)]
call setinputsizes for cx_Oracle.DATETIME
Restored adding cx_Oracle.DATETIME to the setinputsizes() call when a
SQLAlchemy :class:`.Date`, :class:`.DateTime` or :class:`.Time` datatype is
used, so that in the case where a bound parameter is passed as NULL
in some complex queries (in particular this happens with some lazy load
situations), the type is still present. This was removed
in the 1.2 series for arbitrary reasons.
Also adds a suite test for this generic situation.
What's not clear is that do we really need setinputsizes() for all
datatypes if we are supporting NULL in bound parameters.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
