Mike Bayer [Tue, 21 Jul 2020 16:36:20 +0000 (12:36 -0400)]
Allow Grouping to pass along proxy_set of element
Repaired an issue where the "ORDER BY" clause rendering a label name rather
than a complete expression, which is particularly important for SQL Server,
would fail to occur if the expression were enclosed in a parenthesized
grouping in some cases. This case has been added to test support.
Mike Bayer [Sun, 19 Jul 2020 21:39:14 +0000 (17:39 -0400)]
Revise setinputsizes approach
in order to support asyncpg as well as pg8000,
we need to revise setinputsizes to work for more cases as well
as adjust NativeForEmulated a bit to work more completely with
the INTERVAL datatype.
- put most of the setinputsizes work into the compiler where
the computation can be cached.
- support per-element setinputsizes for a tuple
- adjust TypeDecorator so that _unwrapped_dialect_impl
will honor a type that the dialect links to directly in
it's adaption mapping. Decouble _unwrapped_dialect_impl
from TypeDecorator._gen_dialect_impl() which has a different
purpose. This allows setinputsizes to do the right thing
with the INTERVAL datatype.
- test cases for Oracle with Variant continue to work
Mike Bayer [Mon, 13 Jul 2020 14:49:57 +0000 (10:49 -0400)]
test single and double quote inspection scenarios
Applied a sweep through all included dialects to ensure names that contain
single or double quotes are properly escaped when querying system tables,
for all :class:`.Inspector` methods that accept object names as an argument
(e.g. table names, view names, etc). SQLite and MSSQL contained two
quoting issues that were repaired.
mock in python 2.7 / 3.8 seems to have an "args" accessor
but in prior python 3 versions this is not the case,
so this is likely a recent addition to "mock" but wasn't
present in the python3 vendored versions.
Mike Bayer [Sat, 11 Jul 2020 22:54:15 +0000 (18:54 -0400)]
Further corrections to session docs
many more to come.
One issue is that references to documentation sections
inside of decorator-configured deprecation warnings will
also generate a meaningless token when the runtime warning
is emitted. It would be nice to improve upon this somehow.
Mike Bayer [Wed, 8 Jul 2020 18:31:17 +0000 (14:31 -0400)]
Convert remaining ORM APIs to support 2.0 style
This is kind of a mixed bag of all kinds to help get us
to 1.4 betas. The documentation stuff is a work in
progress. Lots of other relatively small changes to
APIs and things. More commits will follow to continue
improving the documentation and transitioning to the
1.4/2.0 hybrid documentation. In particular some refinements
to Session usage models so that it can match Engine's
scoping / transactional patterns, and a decision to
start moving away from "subtransactions" completely.
* add select().from_statement() to produce FromStatement in an
ORM context
* begin referring to select() that has "plugins" for the few edge
cases where select() will have ORM-only behaviors
* convert dynamic.AppenderQuery to its own object that can use
select(), though at the moment it uses Query to support legacy
join calling forms.
* custom query classes for AppenderQuery are replaced by
do_orm_execute() hooks for custom actions, a separate gerrit
will document this
* add Session.get() to replace query.get()
* Deprecate session.begin->subtransaction. propose within the
test suite a hypothetical recipe for apps that rely on this
pattern
* introduce Session construction level context manager,
sessionmaker context manager, rewrite the whole top of the
session_transaction.rst documentation. Establish context manager
patterns for Session that are identical to engine
* ensure same begin_nested() / commit() behavior as engine
* devise all new "join into an external transaction" recipe,
add test support for it, add rules into Session so it
just works, write new docs. need to ensure this doesn't
break anything
* vastly reduce the verbosity of lots of session docs as
I dont think people read this stuff and it's difficult
to keep current in any case
* constructs like case(), with_only_columns() really need
to move to *columns, add a coercion rule to just change
these.
* docs need changes everywhere I look. in_() is not in
the Core tutorial? how do people even know about it?
Remove tons of cruft from Select docs, etc.
* build a system for common ORM options like populate_existing
and autoflush to populate from execution options.
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
- Update the test so it looks at computed_default_table instead of
computed_column_table.
- Stop inspecting autoincrement; instead directly check for
default/non-default values for the relevant columns.
Fixes: #5414
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Wed, 8 Jul 2020 19:48:32 +0000 (15:48 -0400)]
Limit init_subclass test above python 3.6
The commit for I1654befe9eb1c8b8e7fc0784bdbe64284614f0ea #5357
runs the test on all Python 3 versions, however we need to limit
at least python 3.6 for this.
Mike Bayer [Wed, 8 Jul 2020 19:32:38 +0000 (15:32 -0400)]
Propose --nomemory by default
the memory tests are solely responsible for the largest chunk of
time and CPU energy taken up in running the tests. Regressions
in this area are nonexistent unless major changes are being
taken up. Try to revert them to False and see if a single
gerrit job can perhaps run these.
Mike Bayer [Wed, 8 Jul 2020 19:12:30 +0000 (15:12 -0400)]
Repair change of "future_select" caught in merge conflict
the recent change to select() apparently seems to have
been merged with other statements in the test_update_delete
test that was still using the old term.
Mike Bayer [Fri, 26 Jun 2020 20:15:19 +0000 (16:15 -0400)]
Add future=True to create_engine/Session; unify select()
Several weeks of using the future_select() construct
has led to the proposal there be just one select() construct
again which features the new join() method, and otherwise accepts
both the 1.x and 2.x argument styles. This would make
migration simpler and reduce confusion.
However, confusion may be increased by the fact that select().join()
is different Current thinking is we may be better off
with a few hard behavioral changes to old and relatively unknown APIs
rather than trying to play both sides within two extremely similar
but subtly different APIs. At the moment, the .join() thing seems
to be the only behavioral change that occurs without the user
taking any explicit steps. Session.execute() will still
behave the old way as we are adding a future flag.
This change also adds the "future" flag to Session() and
session.execute(), so that interpretation of the incoming statement,
as well as that the new style result is returned, does not
occur for existing applications unless they add the use
of this flag.
The change in general is moving the "removed in 2.0" system
further along where we want the test suite to fully pass
even if the SQLALCHEMY_WARN_20 flag is set.
Get many tests to pass when SQLALCHEMY_WARN_20 is set; this
should be ongoing after this patch merges.
Improve the RemovedIn20 warning; these are all deprecated
"since" 1.4, so ensure that's what the messages read.
Make sure the inforamtion link is on all warnings.
Add deprecation warnings for parameters present and
add warnings to all FromClause.select() types of methods.
Mike Bayer [Tue, 7 Jul 2020 15:26:39 +0000 (11:26 -0400)]
ensure we unwrap desc() /label() all the way w/ order by
The deprecated logic to move order_by expressions
up into the columns clause needed adjustment to accommodate
for a more deeply-wrapped structure when desc() + label()
are combined in an order by column. This structure
now comes from coercions in 1.4. it's not clear to me
at the moment why it's different from 1.3 but
this shouldn't really matter.
Added a ``**kw`` argument to the :meth:`.DeclarativeMeta.__init__` method.
This allows a class to support the :pep:`487` metaclass hook
``__init_subclass__``. Pull request courtesy Ewen Gillies.
1. move all discussion of how to customize connection
to the engines.rst section. have the section in pooling.rst
point to engines.
2. ensure all of the common use cases for connection modification
are listed out in order of level of modification. the use
case of modifying an existing connection is separate from that of
modifying how connect is called; ensure that poolevents.connect
is referred to just as prominently as dialectevents.do_connect.
3. completely replace any discussion of create_engine.creator, as this
hook does not offer anything beyond what do_connect() does.
"creator" is more about using a Pool object directly without an
Engine, which is no longer a documented use case.
Mike Bayer [Sun, 5 Jul 2020 17:44:58 +0000 (13:44 -0400)]
Ensure synchronize_session works with lambda statements
A few places have logic that assumes the top-level statement
is the actual UPDATE or DELETE which is not the case with a
lambda. Ensure the correct object is used. This
fixes issues specific to both "fetch" strategy
as well as "evaluate" strategy.
Mike Bayer [Mon, 16 Dec 2019 22:06:43 +0000 (17:06 -0500)]
introduce deferred lambdas
The coercions system allows us to add in lambdas as arguments
to Core and ORM elements without changing them at all. By allowing
the lambda to produce a deterministic cache key where we can also
cheat and yank out literal parameters means we can move towards
having 90% of "baked" functionality in a clearer way right in
Core / ORM.
As a second step, we can have whole statements inside the lambda,
and can then add generation with __add__(), so then we have
100% of "baked" functionality with full support of ad-hoc
literal values.
Adds some more short_selects tests for the moment for comparison.
Other tweaks inside cache key generation as we're trying to
approach a certain level of performance such that we can
remove the use of "baked" from the loader strategies.
As we have not yet closed #4639, however the caching feature
has been fully integrated as of b0cfa7379cf8513a821a3dbe3028c4965d9f85bd, we will also
add complete caching documentation here and close that issue
as well.
jonathan vanasco [Tue, 30 Jun 2020 16:31:58 +0000 (12:31 -0400)]
added semicolon (;) to improve warning message clarity
### Description
Added a semicolon to improve the clarity of warning message. I actually had a table named `backend`, and thought it was involved!
While updating the code, I noticed no test that directly tests for this warning message. There are tests for the `Can't sort tables for DROP;` prefix of this message and the `exc.CircularDependencyError`; and some tests for the `exc.CircularDependencyError` message itself. I couldn't find any test for this particular message though. (Just thought I'd bring that up)
Mike Bayer [Mon, 29 Jun 2020 19:45:20 +0000 (15:45 -0400)]
Include DATETIME / DateTime with the MySQL TIMESTAMP examples
To eliminate any remaining confusion, clarify that
DATETIME (as well as DateTime) and TIMESTAMP are treated
similarly with the MySQL dialect regarding ON UPDATE.
Mike Bayer [Mon, 29 Jun 2020 18:35:22 +0000 (14:35 -0400)]
Docuemnt current workarounds for MySQL TIMESTAMP
MySQL 8 no longer generates the DEFAULT or ON UPDATE
clauses for TIMESTAMP by default, hence users will begin to hit
this regularly. add warnings that this non-standard SQL is not
accommodated by server_onupdate and docuemnt the workaround
used in issues such as #4652.
Mike Bayer [Sun, 28 Jun 2020 16:47:32 +0000 (12:47 -0400)]
Remove _generate_path_cache_key()
loader options can now make a deterministic cache key based
on the structure they are given, and this accommodates for
aliased classes as well so that these cache keys are now
"safe". Have baked query call upon
the regular cache key method.
Mike Bayer [Sun, 28 Jun 2020 02:57:52 +0000 (22:57 -0400)]
Add an extra step to pool test_sync
Have observed CI failure with windows where not all
three connections got pulled out at the same time here
as the threads got serialized. make sure all three
connections get used.
Mike Bayer [Fri, 17 Apr 2020 14:55:08 +0000 (10:55 -0400)]
ORM executemany returning
Build on #5401 to allow the ORM to take advanage
of executemany INSERT + RETURNING.
Implemented the feature
updated tests
to support INSERT DEFAULT VALUES, needed to come up with
a new syntax for compiler INSERT INTO table (anycol) VALUES (DEFAULT)
which can then be iterated out for executemany.
Added graceful degrade to plain executemany for PostgreSQL <= 8.2
Renamed EXECUTEMANY_DEFAULT to EXECUTEMANY_PLAIN
Fix issue where unicode identifiers or parameter names wouldn't
work with execute_values() under Py2K, because we have to
encode the statement and therefore have to encode the
insert_single_values_expr too.
Correct issue from #5401 to support executemany + return_defaults
for a PK that is explicitly pre-generated, meaning we aren't actually
getting RETURNING but need to return it from compiled_parameters.
Mike Bayer [Tue, 23 Jun 2020 20:21:04 +0000 (16:21 -0400)]
Default psycopg2 executemany mode to "values_only"
The psycopg2 dialect now defaults to using the very performant
``execute_values()`` psycopg2 extension for compiled INSERT statements,
and also impements RETURNING support when this extension is used. This
allows INSERT statements that even include an autoincremented SERIAL
or IDENTITY value to run very fast while still being able to return the
newly generated primary key values. The ORM will then integrate this
new feature in a separate change.
Implements RETURNING for insert with executemany
Adds support to return_defaults() mode and inserted_primary_key
to support mutiple INSERTed rows, via return_defauls_rows
and inserted_primary_key_rows accessors.
within default execution context, new cached compiler
getters are used to fetch primary keys from rows
inserted_primary_key now returns a plain tuple. this
is not yet a row-like object however this can be
added.
Adds distinct "values_only" and "batch" modes, as
"values" has a lot of benefits but "batch" breaks
cursor.rowcount
psycopg2 minimum version 2.7 so we can remove the
large number of checks for very old versions of
psycopg2
simplify tests to no longer distinguish between
native and non-native json
Mike Bayer [Thu, 25 Jun 2020 15:12:40 +0000 (11:12 -0400)]
Use index name to determine if an index is for the PK
Fixed bug in Oracle dialect where indexes that contain the full set of
primary key columns would be mistaken as the primary key index itself,
which is omitted, even if there were multiples. The check has been refined
to compare the name of the primary key constraint against the index name
itself, rather than trying to guess based on the columns present in the
index.
ramonvg [Wed, 24 Jun 2020 13:51:56 +0000 (09:51 -0400)]
Fixes: #4556 - Test that prevents passing a filter to Session.query when running an update
### Description
There was a bug in 1.3.8 that allowed passing a `BinaryExpression` to `Session().query`. I assume it was fixed in https://github.com/sqlalchemy/sqlalchemy/commit/3ab2364e78641c4f0e4b6456afc2cbed39b0d0e6 since now it raises:
```
sqlalchemy.exc.ArgumentError: subject table for an INSERT, UPDATE or DELETE expected, got <sqlalchemy.sql.elements.BinaryExpression object at 0x7f3d90426370>.
```
I'm just adding a test that ensures the behavior. Fixes: [4556](https://github.com/sqlalchemy/sqlalchemy/issues/4556)
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Sun, 21 Jun 2020 16:21:21 +0000 (12:21 -0400)]
Propose using RETURNING for bulk updates, deletes
This patch makes several improvements in the area of
bulk updates and deletes as well as the new session mechanics.
RETURNING is now used for an UPDATE or DELETE statement
emitted for a diaelct that supports "full returning"
in order to satisfy the "fetch" strategy; this currently
includes PostgreSQL and SQL Server. The Oracle dialect
does not support RETURNING for more than one row,
so a new dialect capability "full_returning" is added
in addition to the existing "implicit_returning", indicating
this dialect supports RETURNING for zero or more rows,
not just a single identity row.
The "fetch" strategy will gracefully degrade to
the previous SELECT mechanics for dialects that do not
support RETURNING.
Additionally, the "fetch" strategy will attempt to use
evaluation for the VALUES that were UPDATEd, rather
than just expiring the updated attributes. Values should
be evalutable in all cases where the value is not
a SQL expression.
The new approach also incurs some changes in the
session.execute mechanics, where do_orm_execute() event
handlers can now be chained to each return results;
this is in turn used by the handler to detect on a
per-bind basis if the fetch strategy needs to
do a SELECT or if it can do RETURNING. A test suite is
added to test_horizontal_shard that breaks up a single
UPDATE or DELETE operation among multiple backends
where some are SQLite and don't support RETURNING and
others are PostgreSQL and do.
The session event mechanics are corrected
in terms of the "orm pre execute" hook, which now
receives a flag "is_reentrant" so that the two
ORM implementations for this can skip on their work
if they are being called inside of ORMExecuteState.invoke(),
where previously bulk update/delete were calling its
SELECT a second time.
In order for "fetch" to get the correct identity when
called as pre-execute, it also requests the identity_token
for each mapped instance which is now added as an optional
capability of a SELECT for ORM columns. the identity_token
that's placed by horizontal_sharding is now made available
within each result row, so that even when fetching a
merged result of plain rows we can tell which row belongs
to which identity token.
The evaluator that takes place within the ORM bulk update and delete for
synchronize_session="evaluate" now supports the IN and NOT IN operators.
Tuple IN is also supported.
Mike Bayer [Fri, 19 Jun 2020 14:25:42 +0000 (10:25 -0400)]
Propose black py27 + py35 mode for the rest of Py2K
py27 mode produces one failure for flake8 which is the
space added between exec and parenthesis. however apparently
we can add multiple versions to target-versions which allows
the exec() calls to come out in python 3 style.
The issue we want to improve is issues of trailing
commas being added. I'm not really able to get black to
consistently add or remove these trailing commas in any
case no matter what I set target-version towards.
Mike Bayer [Fri, 19 Jun 2020 04:32:00 +0000 (00:32 -0400)]
perf tweaks
- avoid abc checks in distill_20
- ColumnEntity subclasses are unique to their compile state and
have no querycontext specific state. They can do a simple memoize of their
fetch_column without using attributes, and they can memoize their
_getter() too so that it goes into the cache, just like
instance_processor() does.
- unify ORMColumnEntity and RawColumnEntity for the row processor part,
add some test coverage for the case where it is used in a from_statement
- do a faster generate if there are no memoized entries
- query._params is always immutabledict
Change-Id: I1e2dfe607a1749b5b434fc11f9348ee631501dfa
Mike Bayer [Fri, 12 Jun 2020 17:09:15 +0000 (13:09 -0400)]
Warn when transaction context manager ends on inactive transaction
if .rollback() or .commit() is called inside the transaction
context manager, the transaction object is deactivated.
the context manager continues but will not be able to correctly
fulfill it's closing state. Ensure a warning is emitted when
this happens.
Mike Bayer [Thu, 11 Jun 2020 18:31:57 +0000 (14:31 -0400)]
Add version token to error URL
the sqlalche.me redirector now supports the numerical version
code in the URL, e.g. /13/, /14/, /20/, etc., so that we can
redirect to the error codes for the appropriate version
of SQLAlchemy in use without going through the catch-all "latest"
link. If a particular version of the docs is no longer on the
site, the redirect will revert to falling through the "latest"
link (which ultimately lands on the current release version,
/13/ at the time of this writing).
Added "exists" to the list of reserved words for SQLite so that this word
will be quoted when used as a label or column name. Pull request courtesy
Thodoris Sotiropoulos.
Mike Bayer [Sun, 7 Jun 2020 00:40:43 +0000 (20:40 -0400)]
Turn on caching everywhere, add logging
A variety of caching issues found by running
all tests with statement caching turned on.
The cache system now has a more conservative approach where
any subclass of a SQL element will by default invalidate
the cache key unless it adds the flag inherit_cache=True
at the class level, or if it implements its own caching.
Add working caching to a few elements that were
omitted previously; fix some caching implementations
to suit lesser used edge cases such as json casts
and array slices.
Refine the way BaseCursorResult and CursorMetaData
interact with caching; to suit cases like Alembic
modifying table structures, don't cache the
cursor metadata if it were created against a
cursor.description using non-positional matching,
e.g. "select *". if a table re-ordered its columns
or added/removed, now that data is obsolete.
Additionally we have to adapt the cursor metadata
_keymap regardless of if we just processed
cursor.description, because if we ran against
a cached SQLCompiler we won't have the right
columns in _keymap.
Other refinements to how and when we do this
adaption as some weird cases
were exposed in the Postgresql dialect,
a text() construct that names just one column that
is not actually in the statement. Fixed that
also as it looks like a cut-and-paste artifact
that doesn't actually affect anything.
Various issues with re-use of compiled result maps
and cursor metadata in conjunction with tables being
changed, such as change in order of columns.
mappers can be cleared but the class remains, meaning
a mapper has to use itself as the cache key not the class.
lots of bound parameter / literal issues, due to Alembic
creating a straight subclass of bindparam that renders
inline directly. While we can update Alembic to not
do this, we have to assume other people might be doing
this, so bindparam() implements the inherit_cache=True
logic as well that was a bit involved.
turn on cache stats in logging.
Includes a fix to subqueryloader which moves all setup to
the create_row_processor() phase and elminates any storage
within the compiled context. This includes some changes
to create_row_processor() signature and a revising of the
technique used to determine if the loader can participate
in polymorphic queries, which is also applied to
selectinloading.
DML update.values() and ordered_values() now coerces the
keys as we have tests that pass an arbitrary class here
which only includes __clause_element__(), so the
key can't be cached unless it is coerced. this in turn
changed how composite attributes support bulk update
to use the standard approach of ClauseElement with
annotations that are parsed in the ORM context.
memory profiling successfully caught that the Session
from Query was getting passed into _statement_20()
so that was a big win for that test suite.
Apparently Compiler had .execute() and .scalar() methods
stuck on it, these date back to version 0.4 and there
was a single test in the PostgreSQL dialect tests
that exercised it for no apparent reason. Removed
these methods as well as the concept of a Compiler
holding onto a "bind".
Mike Bayer [Wed, 3 Jun 2020 21:38:35 +0000 (17:38 -0400)]
Convert bulk update/delete to new execution model
This reorganizes the BulkUD model in sqlalchemy.orm.persistence
to be based on the CompileState concept and to allow plain
update() / delete() to be passed to session.execute() where
the ORM synchronize session logic will take place.
Also gets "synchronize_session='fetch'" working with horizontal
sharding.
Adding a few more result.scalar_one() types of methods
as scalar_one() seems like what is normally desired.
### Description
There were a few remnant uses of master/slave in the code and docs. The project previously made a decision to move away from them to use modern and inclusive terminology.
This PR does not cover a bug or necessitate a documented entry into the changelog, so an issue ticket was not created.
### Checklist
This pull request is:
- [x] A documentation / typographical error fix
- [x] A short code fix
- [ ] A new feature implementation
Mike Bayer [Thu, 4 Jun 2020 17:28:21 +0000 (13:28 -0400)]
Document that type_coerce does not currently imply parenthesization
We've had a few issues where the current solution
is to use the self_group() method, so document that as
the current approach for the parenthesization use case.
Whether or not type_coerce() is changed later, this is
how it works at the moment.
Mike Bayer [Tue, 2 Jun 2020 18:21:03 +0000 (14:21 -0400)]
Default create_constraint to False
The :paramref:`.Enum.create_constraint` and
:paramref:`.Boolean.create_constraint` parameters now default to False,
indicating when a so-called "non-native" version of these two datatypes is
created, a CHECK constraint will not be generated by default. These CHECK
constraints present schema-management maintenance complexities that should
be opted in to, rather than being turned on by default.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
