Mike Bayer [Tue, 26 May 2020 02:36:44 +0000 (22:36 -0400)]
Small callcount reductions and refinement for cached queries
baked wasn't using the new one()/first()/one_or_none() methods,
fixed that.
loading._instance_processor() can skip setting up the
quick populators every time because it can cache the getters.
Callcounts have gone below what 1.3 does for the
test_baked_query performance suite, however runtime for
continued inexplicable reasons has not :(. still suspecting
the result tuples but this seems so hard to believe.
Mike Bayer [Mon, 27 Apr 2020 16:58:12 +0000 (12:58 -0400)]
Convert execution to move through Session
This patch replaces the ORM execution flow with a
single pathway through Session.execute() for all queries,
including Core and ORM.
Currently included is full support for ORM Query,
Query.from_statement(), select(), as well as the
baked query and horizontal shard systems. Initial
changes have also been made to the dogpile caching
example, which like baked query makes use of a
new ORM-specific execution hook that replaces the
use of both QueryEvents.before_compile() as well
as Query._execute_and_instances() as the central
ORM interception hooks.
select() and Query() constructs alike can be passed to
Session.execute() where they will return ORM
results in a Results object. This API is currently
used internally by Query. Full support for
Session.execute()->results to behave in a fully
2.0 fashion will be in later changesets.
bulk update/delete with ORM support will also
be delivered via the update() and delete()
constructs, however these have not yet been adapted
to the new system and may follow in a subsequent
update.
Performance is also beginning to lag as of this
commit and some previous ones. It is hoped that
a few central functions such as the coercions
functions can be rewritten in C to re-gain
performance. Additionally, query caching
is now available and some subsequent patches
will attempt to cache more of the per-execution
work from the ORM layer, e.g. column getters
and adapters.
This patch also contains initial "turn on" of the
caching system enginewide via the query_cache_size
parameter to create_engine(). Still defaulting at
zero for "no caching". The caching system still
needs adjustments in order to gain adequate performance.
Mike Bayer [Sun, 1 Dec 2019 22:24:27 +0000 (17:24 -0500)]
Unify Query and select() , move all processing to compile phase
Convert Query to do virtually all compile state computation
in the _compile_context() phase, and organize it all
such that a plain select() construct may also be used as the
source of information in order to generate ORM query state.
This makes it such that Query is not needed except for
its additional methods like from_self() which are all to
be deprecated.
The construction of ORM state will occur beyond the
caching boundary when the new execution model is integrated.
future select() gains a working join() and filter_by() method.
as we continue to rebase and merge each commit in the steps,
callcounts continue to bump around. will have to look at
the final result when it's all in.
Federico Caselli [Thu, 21 May 2020 19:50:49 +0000 (21:50 +0200)]
Avoid proxy functions in row functions
This streamlines a bit for non-C implementations, however
also adds and tests behavioral contracts that mappings should
not allow integer or slice access and should behave like a
Python mapping in that it raises KeyError for an integer
and TypeError for a slice. Py3/Py2/C/noC :)
Federico Caselli [Fri, 22 May 2020 20:48:16 +0000 (22:48 +0200)]
Add python 3.8 profiles; remove zoomark tests
The zoomark tests have served us well for many years.
At this point, they have been using a very antiquated
calling style for many years and are no longer where we catch
performance issues.
The performance suite now has a large number of individual
tests that catch issues very specifically and additionally
record just one performance count per test. This also
allows us to remove the "replay" fixtures that were not
used for anything else.
Mike Bayer [Fri, 22 May 2020 04:06:06 +0000 (00:06 -0400)]
Add immutabledict C code
Start trying to convert fundamental objects to
C as we now rely on a fairly small core of things,
and 1.4 is having problems with complexity added being
slower than the performance gains we are trying to build in.
immutabledict here does seem to bench as twice as fast as the
Python one, see below. However, it does not appear to be
used prominently enough to make any dent in the performance
tests.
at the very least it may provide us some more lift-and-copy
code for more C extensions.
import timeit
from sqlalchemy.util._collections import not_immutabledict, immutabledict
def run(dict_cls):
for i in range(1000000):
d1 = dict_cls({"x": 5, "y": 4})
d2 = d1.union({"x": 17, "new key": "some other value"}, None)
Mike Bayer [Fri, 22 May 2020 17:14:58 +0000 (13:14 -0400)]
Don't emit pyodbc "no driver" warning for empty URL
Fixed an issue in the pyodbc connector such that a warning about pyodbc
"drivername" would be emitted when using a totally empty URL. Empty URLs
are normal when producing a non-connected dialect object or when using the
"creator" argument to create_engine(). The warning now only emits if the
driver name is missing but other parameters are still present.
Mike Bayer [Fri, 22 May 2020 14:13:02 +0000 (10:13 -0400)]
Don't incref on new reference key_style
in 4550983e0ce2f35b3585e53894c941c23693e71d we
added a new attribute key_style. remove an erroneous
Py_INCREF when we acquire it from PyLong_FromLong
as we already own the reference. since this
is a new reference we actualy need to Py_DECREF
it because we aren't returning it.
Miguel Grinberg [Thu, 21 May 2020 09:54:47 +0000 (05:54 -0400)]
Fix query string escaping in engine URLs
Fixed issue in :class:`.URL` object where stringifying the object
would not URL encode special characters, preventing the URL from being
re-consumable as a real URL. Pull request courtesy Miguel Grinberg.
Mike Bayer [Wed, 20 May 2020 17:41:44 +0000 (13:41 -0400)]
Performance fixes for new result set
A few small mistakes led to huge callcounts. Additionally,
the warn-on-get behavior which is attempting to warn for
deprecated access in SQLAlchemy 2.0 is very expensive; it's not clear
if its feasible to have this warning or to somehow alter how it
works.
Mike Bayer [Thu, 7 May 2020 14:53:15 +0000 (10:53 -0400)]
Disable "check unicode returns" under Python 3
Disabled the "unicode returns" check that runs on dialect startup when
running under Python 3, which for many years has occurred in order to test
the current DBAPI's behavior for whether or not it returns Python Unicode
or Py2K strings for the VARCHAR and NVARCHAR datatypes. The check still
occurs by default under Python 2, however the mechanism to test the
behavior will be removed in SQLAlchemy 2.0 when Python 2 support is also
removed.
This logic was very effective when it was needed, however now that Python 3
is standard, all DBAPIs are expected to return Python 3 strings for
character datatypes. In the unlikely case that a third party DBAPI does
not support this, the conversion logic within :class:`.String` is still
available and the third party dialect may specify this in its upfront
dialect flags by setting the dialect level flag ``returns_unicode_strings``
to one of :attr:`.String.RETURNS_CONDITIONAL` or
:attr:`.String.RETURNS_BYTES`, both of which will enable Unicode conversion
even under Python 3.
As part of this change, disabling testing of the doctest tutorials under
Python 2.
Mike Bayer [Mon, 18 May 2020 20:08:33 +0000 (16:08 -0400)]
Streamline visitors.iterate
This method might be used more significantly in the
ORM refactor, so further refine it.
* all get_children() methods now work entirely based on iterators.
Basically only select() was sensitive to this anymore and it now
chains the iterators together
* remove all kinds of flags like column_collections, schema_visitor
that apparently aren't used anymore.
* remove the "depthfirst" visitors as these don't seem to be
used either.
* make sure select() yields its columns first as these will be used
to determine the current mapper.
Mike Bayer [Thu, 14 May 2020 16:50:11 +0000 (12:50 -0400)]
Update transaction / connection handling
step one, do away with __connection attribute and using
awkward AttributeError logic
step two, move all management of "connection._transaction"
into the transaction objects themselves where it's easier
to follow.
build MarkerTransaction that takes the role of
"do-nothing block"
new connection datamodel is: connection._transaction, always
a root, connection._nested_transaction, always a nested.
nested transactions still chain to each other as this
is still sort of necessary but they consider the root
transaction separately, and the marker transactions
not at all.
introduce new InvalidRequestError subclass
PendingRollbackError. Apply to connection and session
for all cases where a transaction needs to be rolled
back before continuing. Within Connection,
both PendingRollbackError as well as ResourceClosedError
are now raised directly without being handled by
handle_dbapi_error(); this removes these two exception
cases from the handle_error event handler as well as
from StatementError wrapping, as these two exceptions are
not statement oriented and are instead programmatic
issues, that the application is failing to handle database
errors properly.
Revise savepoints so that when a release fails, they set
themselves as inactive so that their rollback() method
does not throw another exception.
Give savepoints another go on MySQL, can't get release working
however get support for basic round trip going
Mike Bayer [Sat, 16 May 2020 17:05:00 +0000 (13:05 -0400)]
Reword delete-orphan on many error message and document
For many years we have encountered users making use of the
"single_parent" flag in response to the error message for
"delete-orphan" expressing this as a means to cancel the current
error. However, the actual issue here is usually a misuse
of the delete-orphan cascade setting. Reword the error message to
be much more descriptive about what this means and add new
error link sections describing the situation in as much detail
as possible.
Fixes: #4860
# Description
Add nowait, skip_lock, of arguments to for_update_clause for mysql
### Checklist
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [x] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Thu, 14 May 2020 14:51:29 +0000 (10:51 -0400)]
Actively unset reset agent in discard transaction
The assumptions in _discard_transaction from 916e1fea25afcd07fa1d1d2f72043b372cd02223 were too narrow,
assuming that if the given transaction were not our
"current" one, that this would not be the reset agent. however
as the legacy behvaior is that even a "nested" transaction gets
set as "self._transaction", this did not accommodate for the nested
transaction being thrown away. We will attempt to refine all of this
logic in #5327 for 1.4 /master assuming this is feasible for the
full suite of current use cases.
Mike Bayer [Wed, 13 May 2020 16:42:08 +0000 (12:42 -0400)]
Assert reset agent always set correctly and is active
Fixed fairly critical issue where the DBAPI connection could be returned to
the connection pool while still in an un-rolled-back state. The reset agent
responsible for rolling back the connection could be corrupted in the case
that the transaction was "closed" without being rolled back or committed,
which can occur in some scenarios when using ORM sessions and emitting
.close() in a certain pattern involving savepoints. The fix ensures that
the reset agent is always active.
note that the reset agent will go away in 2.0 and the only real
purpose of it is for logging of ROLLBACK. Apparently with the
SQLite singleton engine in the test suite, there are some strucutral
mismatches in the test fixtures where the reset agent is getting
set differently than the transaction likely due to the same connection
being shared in multiple context, though it's unclear.
Dylan Modesitt [Wed, 6 May 2020 18:17:23 +0000 (14:17 -0400)]
Add 'schema' parameter to table
Added a "schema" parameter to the :func:`_expression.table` construct,
allowing ad-hoc table expressions to also include a schema name.
Pull request courtesy Dylan Modesitt.
Mike Bayer [Thu, 7 May 2020 17:56:38 +0000 (13:56 -0400)]
Use cx_oracle.LONG_STRING /LONG_BINARY for CLOB/BLOB
Changed the implementation of fetching CLOB and BLOB objects to use
cx_Oracle's native implementation which fetches CLOB/BLOB objects inline
with other result columns, rather than performing a separate fetch. As
always, this can be disabled by setting auto_convert_lobs to False.
As part of this change, the behavior of a CLOB that was given a blank
string on INSERT now returns None on SELECT, which is now consistent with
that of VARCHAR on Oracle.
Mike Bayer [Fri, 8 May 2020 15:25:14 +0000 (11:25 -0400)]
Expand CTE testing and begin to make this more prominent.
SQLite supports CTEs as do both the MariaDB and MySQL
databases now. Expand test support and begin adding
new tutorial sections to make this concept more prominent.
Also fix some formatting issues in the tutorial, and start
moving more towards {opensql} - the "SQL" button thing
is cool but the tutorials mainly need to illustrate how
the SQL looks for things. Screens are bigger now and
we likely should move away from the SQL button almost
entirely.
CTE for the tutorial needed zzzeeksphinx fix to recognize
"WITH" so bumping that to latest release number in git.
Mike Bayer [Thu, 7 May 2020 21:13:35 +0000 (17:13 -0400)]
Warn when sorted_tables is not actually sorting
A warning is emitted when making use of the :attr:`.MetaData.sorted_tables`
attribute as well as the :func:`_schema.sort_tables` function, and the
given tables cannot be correctly sorted due to a cyclic dependency between
foreign key constraints. In this case, the functions will no longer sort
the involved tables by foreign key, and a warning will be emitted. Other
tables that are not part of the cycle will still be returned in dependency
order. Previously, the sorted_table routines would return a collection that
would unconditionally omit all foreign keys when a cycle was detected, and
no warning was emitted.
Mike Bayer [Wed, 6 May 2020 13:47:26 +0000 (09:47 -0400)]
Clarify create_engine encoding; update cx_Oracle
The create_engine()->encoding parameter is mostly irrelevant
under Python 3. make it clear this parameter is not generally
useful anymore and refer readers to the dialect documenation.
Update cx_Oracle documentation to feature many examples of
the encoding / nencoding parameters, remove extra detail that
is not generally useful and reorganize information into
more specific sections, de-emphasizing legacy / Python 2
specific sections.
Introduce :paramref:`_orm.relationship.sync_backref` flag in a relationship
to control if the synchronization events that mutate the in-Python
attributes are added.
This flag is implied in a ``viewonly=True`` relationship.
This supersedes the previous change #5149, that warned that ``viewonly=True``
relationship target of a back_populates or backref configuration would be
disallowed.
Mike Bayer [Mon, 4 May 2020 19:48:48 +0000 (15:48 -0400)]
Consider aliased=True, from_joinpoint as legacy
For a 1.4 / 1.3 merge, rewrite the documentation for
Query.join() to indicate calling forms that are now considered
legacy, including the use of strings in join(), sending a
series of join paths in one call, and using the aliased=True
flag. update the elementtree examples as well to use aliased()
(they are much simpler to understand this way too) and update
other links.
Also improve docs for aliased() and some other ORM targets
such as PropComparator.
Mike Bayer [Sun, 3 May 2020 23:35:54 +0000 (19:35 -0400)]
Baked query needs to spoil fully on uncachable option
Fixed issue in the area of where loader options such as selectinload()
interact with the baked query system, such that the caching of a query is
not supposed to occur if the loader options themselves have elements such
as with_polymorphic() objects in them that currently are not
cache-compatible. The baked loader could sometimes not fully invalidate
itself in these some of these scenarios leading to missed eager loads.
Mike Bayer [Mon, 4 May 2020 00:27:24 +0000 (20:27 -0400)]
Don't apply sets or similar to objects in IdentitySet
Modified the internal "identity set" implementation, which is a set that
hashes objects on their id() rather than their hash values, to not actually
call the ``__hash__()`` method of the objects, which are typically
user-mapped objects. Some methods were calling this method as a side
effect of the implementation.
Mike Bayer [Fri, 24 Apr 2020 19:34:19 +0000 (15:34 -0400)]
Integrate new Result into ORM query
The next step in the 2.0 ORM changes is to have the
ORM integrate with the new Result object fully.
this patch uses Result to represent ORM objects rather
than lists. public API to get at this Result is not
added yet. dogpile.cache and horizontal sharding
recipe/extensions have small adjustments to accommodate
this change.
Callcounts have fluctuated, some slightly better and
some slightly worse. A few have gone up by a bit,
however as the codebase is still in flux it is anticipated
there will be some performance gains later on as
ORM fetching is refined to no longer need to accommodate
for extensive aliasing. The addition of caching
will then change the entire story.
Mike Bayer [Sun, 26 Apr 2020 14:59:34 +0000 (10:59 -0400)]
Documentation updates for ResultProxy -> Result
This is based off of
I8091919d45421e3f53029b8660427f844fee0228 and includes
all documentation-only changes as a separate merge,
once the parent is merged.
Mike Bayer [Sat, 2 May 2020 00:30:18 +0000 (20:30 -0400)]
Handle non-string, non column expression in index for deferred attach
Fixed issue where an :class:`.Index` that is deferred in being associated
with a table, such as as when it contains a :class:`.Column` that is not
associated with any :class:`.Table` yet, would fail to attach correctly if
it also contained a non table-oriented expession.
Mike Bayer [Tue, 21 Apr 2020 16:51:13 +0000 (12:51 -0400)]
Propose Result as immediate replacement for ResultProxy
As progress is made on the _future.Result, including breaking
it out such that DBAPI behaviors are local to specific
implementations, it becomes apparent that the Result object
is a functional superset of ResultProxy and that basic
operations like fetchone(), fetchall(), and fetchmany()
behave pretty much exactly the same way on the new object.
Reorganize things so that ResultProxy is now referred to
as LegacyCursorResult, which subclasses CursorResult
that represents the DBAPI-cursor version of Result,
making use of a multiple inheritance pattern so that
the functionality of Result is also available in non-DBAPI
contexts, as will be necessary for some ORM
patterns.
Additionally propose the composition system for Result
that will form the basis for ORM-alternative result
systems such as horizontal sharding and dogpile cache.
As ORM results will soon be coming directly from
instances of Result, these extensions will instead
build their own ResultFetchStrategies that perform
the special steps to create composed or cached
result sets.
Also considering at the moment not emitting deprecation
warnings for fetchXYZ() methods; the immediate issue
is Keystone tests are calling upon it, but as the
implementations here are proving to be not in any
kind of conflict with how Result works, there's
not too much issue leaving them around and deprecating
at some later point.
fixes #4212 raise an error if selectable and flat/aliased are sent to with_polymorphic simultaenously
<!-- Provide a general summary of your proposed changes in the Title field above -->
Hi,
This is a proposed fix for Issue #4212
When the selectable argument is passed to with_polymorphic it is already aliased. Therefore an argument error should be raised if flat is also True. I have added an if statement to the with_polymorphic function, that raises an Argument Error if selectable is none/False AND if flat=True. I have also included a test.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [X ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Mon, 27 Apr 2020 20:51:43 +0000 (16:51 -0400)]
Alias the onclause if ORM join is to same polymorphic selectable
Fixed bug where using :func:`.with_polymorphic` as the target of a join via
:meth:`.RelationshipComparator.of_type` on a mapper that already has a
subquery-based with_polymorphic setting that's equivalent to the one
requested would not correctly alias the ON clause in the join.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
