Mike Bayer [Fri, 22 May 2020 17:14:58 +0000 (13:14 -0400)]
Don't emit pyodbc "no driver" warning for empty URL
Fixed an issue in the pyodbc connector such that a warning about pyodbc
"drivername" would be emitted when using a totally empty URL. Empty URLs
are normal when producing a non-connected dialect object or when using the
"creator" argument to create_engine(). The warning now only emits if the
driver name is missing but other parameters are still present.
Mike Bayer [Fri, 22 May 2020 14:13:02 +0000 (10:13 -0400)]
Don't incref on new reference key_style
in 4550983e0ce2f35b3585e53894c941c23693e71d we
added a new attribute key_style. remove an erroneous
Py_INCREF when we acquire it from PyLong_FromLong
as we already own the reference. since this
is a new reference we actualy need to Py_DECREF
it because we aren't returning it.
Miguel Grinberg [Thu, 21 May 2020 09:54:47 +0000 (05:54 -0400)]
Fix query string escaping in engine URLs
Fixed issue in :class:`.URL` object where stringifying the object
would not URL encode special characters, preventing the URL from being
re-consumable as a real URL. Pull request courtesy Miguel Grinberg.
Mike Bayer [Wed, 20 May 2020 17:41:44 +0000 (13:41 -0400)]
Performance fixes for new result set
A few small mistakes led to huge callcounts. Additionally,
the warn-on-get behavior which is attempting to warn for
deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear
if its feasible to have this warning or to somehow alter how it
works.
Mike Bayer [Thu, 7 May 2020 14:53:15 +0000 (10:53 -0400)]
Disable "check unicode returns" under Python 3
Disabled the "unicode returns" check that runs on dialect startup when
running under Python 3, which for many years has occurred in order to test
the current DBAPI's behavior for whether or not it returns Python Unicode
or Py2K strings for the VARCHAR and NVARCHAR datatypes. The check still
occurs by default under Python 2, however the mechanism to test the
behavior will be removed in SQLAlchemy 2.0 when Python 2 support is also
removed.
This logic was very effective when it was needed, however now that Python 3
is standard, all DBAPIs are expected to return Python 3 strings for
character datatypes. In the unlikely case that a third party DBAPI does
not support this, the conversion logic within :class:`.String` is still
available and the third party dialect may specify this in its upfront
dialect flags by setting the dialect level flag ``returns_unicode_strings``
to one of :attr:`.String.RETURNS_CONDITIONAL` or
:attr:`.String.RETURNS_BYTES`, both of which will enable Unicode conversion
even under Python 3.
As part of this change, disabling testing of the doctest tutorials under
Python 2.
Mike Bayer [Mon, 18 May 2020 20:08:33 +0000 (16:08 -0400)]
Streamline visitors.iterate
This method might be used more significantly in the
ORM refactor, so further refine it.
* all get_children() methods now work entirely based on iterators.
Basically only select() was sensitive to this anymore and it now
chains the iterators together
* remove all kinds of flags like column_collections, schema_visitor
that apparently aren't used anymore.
* remove the "depthfirst" visitors as these don't seem to be
used either.
* make sure select() yields its columns first as these will be used
to determine the current mapper.
Mike Bayer [Thu, 14 May 2020 16:50:11 +0000 (12:50 -0400)]
Update transaction / connection handling
step one, do away with __connection attribute and using
awkward AttributeError logic
step two, move all management of "connection._transaction"
into the transaction objects themselves where it's easier
to follow.
build MarkerTransaction that takes the role of
"do-nothing block"
new connection datamodel is: connection._transaction, always
a root, connection._nested_transaction, always a nested.
nested transactions still chain to each other as this
is still sort of necessary but they consider the root
transaction separately, and the marker transactions
not at all.
introduce new InvalidRequestError subclass
PendingRollbackError. Apply to connection and session
for all cases where a transaction needs to be rolled
back before continuing. Within Connection,
both PendingRollbackError as well as ResourceClosedError
are now raised directly without being handled by
handle_dbapi_error(); this removes these two exception
cases from the handle_error event handler as well as
from StatementError wrapping, as these two exceptions are
not statement oriented and are instead programmatic
issues, that the application is failing to handle database
errors properly.
Revise savepoints so that when a release fails, they set
themselves as inactive so that their rollback() method
does not throw another exception.
Give savepoints another go on MySQL, can't get release working
however get support for basic round trip going
Mike Bayer [Sat, 16 May 2020 17:05:00 +0000 (13:05 -0400)]
Reword delete-orphan on many error message and document
For many years we have encountered users making use of the
"single_parent" flag in response to the error message for
"delete-orphan" expressing this as a means to cancel the current
error. However, the actual issue here is usually a misuse
of the delete-orphan cascade setting. Reword the error message to
be much more descriptive about what this means and add new
error link sections describing the situation in as much detail
as possible.
Mike Bayer [Thu, 14 May 2020 14:51:29 +0000 (10:51 -0400)]
Actively unset reset agent in discard transaction
The assumptions in _discard_transaction from 916e1fea25afcd07fa1d1d2f72043b372cd02223 were too narrow,
assuming that if the given transaction were not our
"current" one, that this would not be the reset agent. however
as the legacy behvaior is that even a "nested" transaction gets
set as "self._transaction", this did not accommodate for the nested
transaction being thrown away. We will attempt to refine all of this
logic in #5327 for 1.4 /master assuming this is feasible for the
full suite of current use cases.
Mike Bayer [Wed, 13 May 2020 16:42:08 +0000 (12:42 -0400)]
Assert reset agent always set correctly and is active
Fixed fairly critical issue where the DBAPI connection could be returned to
the connection pool while still in an un-rolled-back state. The reset agent
responsible for rolling back the connection could be corrupted in the case
that the transaction was "closed" without being rolled back or committed,
which can occur in some scenarios when using ORM sessions and emitting
.close() in a certain pattern involving savepoints. The fix ensures that
the reset agent is always active.
note that the reset agent will go away in 2.0 and the only real
purpose of it is for logging of ROLLBACK. Apparently with the
SQLite singleton engine in the test suite, there are some strucutral
mismatches in the test fixtures where the reset agent is getting
set differently than the transaction likely due to the same connection
being shared in multiple context, though it's unclear.
Dylan Modesitt [Wed, 6 May 2020 18:17:23 +0000 (14:17 -0400)]
Add 'schema' parameter to table
Added a "schema" parameter to the :func:`_expression.table` construct,
allowing ad-hoc table expressions to also include a schema name.
Pull request courtesy Dylan Modesitt.
Mike Bayer [Thu, 7 May 2020 17:56:38 +0000 (13:56 -0400)]
Use cx_oracle.LONG_STRING /LONG_BINARY for CLOB/BLOB
Changed the implementation of fetching CLOB and BLOB objects to use
cx_Oracle's native implementation which fetches CLOB/BLOB objects inline
with other result columns, rather than performing a separate fetch. As
always, this can be disabled by setting auto_convert_lobs to False.
As part of this change, the behavior of a CLOB that was given a blank
string on INSERT now returns None on SELECT, which is now consistent with
that of VARCHAR on Oracle.
Mike Bayer [Fri, 8 May 2020 15:25:14 +0000 (11:25 -0400)]
Expand CTE testing and begin to make this more prominent.
SQLite supports CTEs as do both the MariaDB and MySQL
databases now. Expand test support and begin adding
new tutorial sections to make this concept more prominent.
Also fix some formatting issues in the tutorial, and start
moving more towards {opensql} - the "SQL" button thing
is cool but the tutorials mainly need to illustrate how
the SQL looks for things. Screens are bigger now and
we likely should move away from the SQL button almost
entirely.
CTE for the tutorial needed zzzeeksphinx fix to recognize
"WITH" so bumping that to latest release number in git.
Mike Bayer [Thu, 7 May 2020 21:13:35 +0000 (17:13 -0400)]
Warn when sorted_tables is not actually sorting
A warning is emitted when making use of the :attr:`.MetaData.sorted_tables`
attribute as well as the :func:`_schema.sort_tables` function, and the
given tables cannot be correctly sorted due to a cyclic dependency between
foreign key constraints. In this case, the functions will no longer sort
the involved tables by foreign key, and a warning will be emitted. Other
tables that are not part of the cycle will still be returned in dependency
order. Previously, the sorted_table routines would return a collection that
would unconditionally omit all foreign keys when a cycle was detected, and
no warning was emitted.
Mike Bayer [Wed, 6 May 2020 13:47:26 +0000 (09:47 -0400)]
Clarify create_engine encoding; update cx_Oracle
The create_engine()->encoding parameter is mostly irrelevant
under Python 3. make it clear this parameter is not generally
useful anymore and refer readers to the dialect documenation.
Update cx_Oracle documentation to feature many examples of
the encoding / nencoding parameters, remove extra detail that
is not generally useful and reorganize information into
more specific sections, de-emphasizing legacy / Python 2
specific sections.
Introduce :paramref:`_orm.relationship.sync_backref` flag in a relationship
to control if the synchronization events that mutate the in-Python
attributes are added.
This flag is implied in a ``viewonly=True`` relationship.
This supersedes the previous change #5149, that warned that ``viewonly=True``
relationship target of a back_populates or backref configuration would be
disallowed.
Mike Bayer [Mon, 4 May 2020 19:48:48 +0000 (15:48 -0400)]
Consider aliased=True, from_joinpoint as legacy
For a 1.4 / 1.3 merge, rewrite the documentation for
Query.join() to indicate calling forms that are now considered
legacy, including the use of strings in join(), sending a
series of join paths in one call, and using the aliased=True
flag. update the elementtree examples as well to use aliased()
(they are much simpler to understand this way too) and update
other links.
Also improve docs for aliased() and some other ORM targets
such as PropComparator.
Mike Bayer [Sun, 3 May 2020 23:35:54 +0000 (19:35 -0400)]
Baked query needs to spoil fully on uncachable option
Fixed issue in the area of where loader options such as selectinload()
interact with the baked query system, such that the caching of a query is
not supposed to occur if the loader options themselves have elements such
as with_polymorphic() objects in them that currently are not
cache-compatible. The baked loader could sometimes not fully invalidate
itself in these some of these scenarios leading to missed eager loads.
Mike Bayer [Mon, 4 May 2020 00:27:24 +0000 (20:27 -0400)]
Don't apply sets or similar to objects in IdentitySet
Modified the internal "identity set" implementation, which is a set that
hashes objects on their id() rather than their hash values, to not actually
call the ``__hash__()`` method of the objects, which are typically
user-mapped objects. Some methods were calling this method as a side
effect of the implementation.
Mike Bayer [Fri, 24 Apr 2020 19:34:19 +0000 (15:34 -0400)]
Integrate new Result into ORM query
The next step in the 2.0 ORM changes is to have the
ORM integrate with the new Result object fully.
this patch uses Result to represent ORM objects rather
than lists. public API to get at this Result is not
added yet. dogpile.cache and horizontal sharding
recipe/extensions have small adjustments to accommodate
this change.
Callcounts have fluctuated, some slightly better and
some slightly worse. A few have gone up by a bit,
however as the codebase is still in flux it is anticipated
there will be some performance gains later on as
ORM fetching is refined to no longer need to accommodate
for extensive aliasing. The addition of caching
will then change the entire story.
Mike Bayer [Sun, 26 Apr 2020 14:59:34 +0000 (10:59 -0400)]
Documentation updates for ResultProxy -> Result
This is based off of
I8091919d45421e3f53029b8660427f844fee0228 and includes
all documentation-only changes as a separate merge,
once the parent is merged.
Mike Bayer [Sat, 2 May 2020 00:30:18 +0000 (20:30 -0400)]
Handle non-string, non column expression in index for deferred attach
Fixed issue where an :class:`.Index` that is deferred in being associated
with a table, such as as when it contains a :class:`.Column` that is not
associated with any :class:`.Table` yet, would fail to attach correctly if
it also contained a non table-oriented expession.
Mike Bayer [Tue, 21 Apr 2020 16:51:13 +0000 (12:51 -0400)]
Propose Result as immediate replacement for ResultProxy
As progress is made on the _future.Result, including breaking
it out such that DBAPI behaviors are local to specific
implementations, it becomes apparent that the Result object
is a functional superset of ResultProxy and that basic
operations like fetchone(), fetchall(), and fetchmany()
behave pretty much exactly the same way on the new object.
Reorganize things so that ResultProxy is now referred to
as LegacyCursorResult, which subclasses CursorResult
that represents the DBAPI-cursor version of Result,
making use of a multiple inheritance pattern so that
the functionality of Result is also available in non-DBAPI
contexts, as will be necessary for some ORM
patterns.
Additionally propose the composition system for Result
that will form the basis for ORM-alternative result
systems such as horizontal sharding and dogpile cache.
As ORM results will soon be coming directly from
instances of Result, these extensions will instead
build their own ResultFetchStrategies that perform
the special steps to create composed or cached
result sets.
Also considering at the moment not emitting deprecation
warnings for fetchXYZ() methods; the immediate issue
is Keystone tests are calling upon it, but as the
implementations here are proving to be not in any
kind of conflict with how Result works, there's
not too much issue leaving them around and deprecating
at some later point.
fixes #4212 raise an error if selectable and flat/aliased are sent to with_polymorphic simultaenously
<!-- Provide a general summary of your proposed changes in the Title field above -->
Hi,
This is a proposed fix for Issue #4212
When the selectable argument is passed to with_polymorphic it is already aliased. Therefore an argument error should be raised if flat is also True. I have added an if statement to the with_polymorphic function, that raises an Argument Error if selectable is none/False AND if flat=True. I have also included a test.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [X ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Mon, 27 Apr 2020 20:51:43 +0000 (16:51 -0400)]
Alias the onclause if ORM join is to same polymorphic selectable
Fixed bug where using :func:`.with_polymorphic` as the target of a join via
:meth:`.RelationshipComparator.of_type` on a mapper that already has a
subquery-based with_polymorphic setting that's equivalent to the one
requested would not correctly alias the ON clause in the join.
Fix a regression introduced by the reflection of computed column in
MSSQL when using SQL server versions before 2012, which does not support
the ``concat`` function and when using the legacy TDS version 4.2.
The dialect will try to detect the protocol version of first connect
and run in compatibility mode if it cannot detect it.
Mike Bayer [Mon, 20 Apr 2020 16:24:40 +0000 (12:24 -0400)]
Add ColumnProperty.Comparator.expressions
Added an accessor :attr:`.ColumnProperty.Comparator.expressions` which
provides access to the group of columns mapped under a multi-column
:class:`.ColumnProperty` attribute.
Added support for columns or type :class:`.ARRAY` of :class:`.Enum`,
:class:`.JSON` or :class:`_postgresql.JSONB` in PostgreSQL.
Previously a workaround was required in these use cases.
Raise an explicit :class:`.exc.CompileError` when adding a table with a
column of type :class:`.ARRAY` of :class:`.Enum` configured with
:paramref:`.Enum.native_enum` set to ``False`` when
:paramref:`.Enum.create_constraint` is not set to ``False``
Deprecate ``DISTINCT ON`` when not targeting PostgreSQL
Deprecate usage of ``DISTINCT ON`` in dialect other than PostgreSQL.
Previously this was silently ignored.
Deprecate old usage of string distinct in MySQL dialect
Mike Bayer [Sun, 19 Apr 2020 20:52:54 +0000 (16:52 -0400)]
Raise informative error when non-object m2o comparison used
An informative error message is raised when an ORM many-to-one comparison
is attempted against an object that is not an actual mapped instance.
Comparisons such as those to scalar subqueries aren't supported;
generalized comparison with subqueries is better achieved using
:meth:`~.RelationshipProperty.Comparator.has`.
Mike Bayer [Tue, 7 Apr 2020 18:15:43 +0000 (14:15 -0400)]
Create initial 2.0 engine implementation
Implemented the SQLAlchemy 2 :func:`.future.create_engine` function which
is used for forwards compatibility with SQLAlchemy 2. This engine
features always-transactional behavior with autobegin.
Allow execution options per statement execution. This includes
that the before_execute() and after_execute() events now accept
an additional dictionary with these options, empty if not
passed; a legacy event decorator is added for backwards compatibility
which now also emits a deprecation warning.
Add some basic tests for execution, transactions, and
the new result object. Build out on a new testing fixture
that swaps in the future engine completely to start with.
Mike Bayer [Tue, 14 Apr 2020 19:30:28 +0000 (15:30 -0400)]
Pass connection to TablesTest.insert_data()
towards the goal of reducing verbosity and repetition
in test fixtures as well as that we are moving to
connection only for execution, move the insert_data()
classmethod to accept a connection and adjust all
fixtures to use it.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
