Jesse Bakker [Mon, 17 Aug 2020 14:35:35 +0000 (10:35 -0400)]
Make discriminator column used by ConcreteBase configurable
The name of the virtual column used when using the
:class:`_declarative.AbstractConcreteBase` and
:class:`_declarative.ConcreteBase` classes can now be customized, to allow
for models that have a column that is actually named ``type``. Pull
request courtesy Jesse-Bakker.
Mike Bayer [Fri, 14 Aug 2020 16:07:14 +0000 (12:07 -0400)]
Provision on different drivers dynamically
We want TOX_POSTGRESQL and similar to be the fixed variable
that is configured from CI environment. These variables should refer
to database servers but individual drivers like asyncpg mysqlconnector
etc. should come from local tox.ini. add a new system to generate
per-driver URLs from a simple list of hostname-based URLs delivered
from CI environment.
Mike Bayer [Fri, 14 Aug 2020 02:44:33 +0000 (22:44 -0400)]
Use TOX_POSTGRESQL_MASTER for master
since 1.3 doesn't have updated pg8000 support nor
asyncpg support it needs to stay on the previous
TOX_POSTGRESQL variable, so make a new one that will
have the newer drivers in it.
We can now create asyncio endpoints that are then handled
in "implicit IO" form within the majority of the Core internals.
Then coroutines are re-exposed at the point at which we call
into asyncpg methods.
Patch includes:
* asyncpg dialect
* asyncio package
* engine, result, ORM session classes
* new test fixtures, tests
* some work with pep-484 and a short plugin for the
pyannotate package, which seems to have so-so results
RamonWill [Thu, 13 Aug 2020 21:04:13 +0000 (17:04 -0400)]
Test execution_options on Query object before compilation
A test that checks if the execution options are being set on the Query object before compilation
### Description
Since Issue #4670 has been open there have changes to the execution mechanics that resolved it. I have just created a test case that will enable the issue to be closed.
This pull request is:
- [ X] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Thu, 13 Aug 2020 14:43:53 +0000 (10:43 -0400)]
Further fixes for ticket 5470
The fix for #5470 didn't actually take into account that
the "distinct" logic in query was also doubling up the criteria.
Added many more tests. the 1.3 version here will be different
than 1.4 as the regression is not quite the same.
Mike Bayer [Wed, 12 Aug 2020 22:46:25 +0000 (18:46 -0400)]
Use importlib_metadata; add namespace for mariadb
The ``importlib_metadata`` library is used to scan for setuptools
entrypoints rather than pkg_resources. as importlib_metadata is a small
library that is included as of Python 3.8, the compatibility library is
installed as a dependency for Python versions older than 3.8.
Unfortunately setuptools "attr:" is broken because it tries to import
the module; seems like this is fixed as part of
https://github.com/pypa/setuptools/pull/1753 however this is too recent
to rely upon for now.
Added a new dialect token "mariadb" that may be used in place of "mysql" in
the :func:`_sa.create_engine` URL. This will deliver a MariaDB dialect
subclass of the MySQLDialect in use that forces the "is_mariadb" flag to
True. The dialect will raise an error if a server version string that does
not indicate MariaDB in use is received. This is useful for
MariaDB-specific testing scenarios as well as to support applications that
are hardcoding to MariaDB-only concepts. As MariaDB and MySQL featuresets
and usage patterns continue to diverge, this pattern may become more
prominent.
Mike Bayer [Thu, 13 Aug 2020 00:14:15 +0000 (20:14 -0400)]
Sweep through UPDATE ordered_values a second time
The fix in 180ae7c1a53385f72b0047496ac001ec5099cc3e
didn't do much as the code was not preserving parameter
order at all, in fact. Reworked stmt_parameters to be
delivered in the correct order up front and preserve
throughout crud.py which was not being done at all
before.
Mike Bayer [Mon, 10 Aug 2020 22:40:49 +0000 (18:40 -0400)]
Break scalars() and mappings() into separate objects
The issue of Result.fetchXXX() methods returning Row
objects unless filtering is applied will not provide a
clear enough API story when type annotations are applied,
so break out scalars/mappings into separate wrapper objects.
this makes some things more intuitive and other things a little
more bumpy. however the return type story is now clearer.
Mike Bayer [Tue, 11 Aug 2020 21:46:11 +0000 (17:46 -0400)]
Correct for update.ordered_values() and use in new test
The test for the parameter fix in c0685e5f41 was not working
deterministically on Python 2, so use ordered_values(), however
a second issue in ordered_values() was assuming each element was a
column, so also test for array-assignment expressions with
ordered_values.
Mike Bayer [Sat, 8 Aug 2020 17:03:17 +0000 (13:03 -0400)]
render INSERT/UPDATE column expressions up front; pass state
Fixes related to rendering of complex UPDATE DML
which was not correctly preserving positional parameter
order in conjunction with DML features that are only known
to work on the PostgreSQL database. Both pg8000
and asyncpg use positional parameters which is why these
issues are suddenly apparent.
crud.py now takes on the task of rendering the column
expressions for SET or VALUES so that for the very unusual
case that the column expression is a compound expression
that includes a bound parameter (namely an array index),
the bound parameter order is preserved.
Additionally, crud.py passes through the positional_names
keyword argument into bindparam_string() which is necessary
when CTEs are being rendered, as PG supports complex
CTE / INSERT / UPDATE scenarios.
Mike Bayer [Fri, 7 Aug 2020 18:51:33 +0000 (14:51 -0400)]
Don't link on_connect to first_connect event handler
Adjusted the dialect initialization process such that the
:meth:`_engine.Dialect.on_connect` is not called a second time on the first
connection. The hook is called first, then the
:meth:`_engine.Dialect.initialize` is called if that connection is the
first for that dialect, then no more events are called. This eliminates
the two calls to the "on_connect" function which can produce very difficult
debugging situations.
Mike Bayer [Fri, 7 Aug 2020 15:24:29 +0000 (11:24 -0400)]
Pass schema_translate_map from DDLCompiler to SQLCompiler
Fixed issue where the
:paramref:`_engine.Connection.execution_options.schema_translate_map`
feature would not take effect when the :meth:`_schema.Sequence.next_value`
function function for a :class:`_schema.Sequence` were used in the
:paramref:`_schema.Column.server_default` parameter and the create table
DDL were emitted.
Mike Bayer [Thu, 6 Aug 2020 19:53:17 +0000 (15:53 -0400)]
base all_orm_descriptors ordering on cls.__dict__ + cls.__mro__
Adjusted the workings of the :meth:`_orm.Mapper.all_orm_descriptors`
accessor to represent the attributes in the order that they are located in
a deterministic way, assuming the use of Python 3.6 or higher which
maintains the sorting order of class attributes based on how they were
declared. This sorting is not guaranteed to match the declared order of
attributes in all cases however; see the method documentation for the exact
scheme.
Mike Bayer [Sun, 28 Jun 2020 15:59:34 +0000 (11:59 -0400)]
Documentation updates for 1.4
* major additions to 1.4 migration doc; removed additional
verbosity regarding caching methodology and reorganized the
doc to present itself more as a "what's changed" guide
* as we now have a path for asyncio, update that doc so that
we aren't spreading obsolete information
* updates to the 2.0 migration guide with latest info, however
this is still an architecture doc and not a migration guide
yet, will need further rework.
* start really talking about 1.x vs. 2.0 style everywhere. Querying
is most of the docs so this is going to be a prominent
theme, start getting it to fit in
* Add introductory documentation for ORM example sections as these
are too sparse
* new documentation for do_orm_execute(), many separate sections,
adding deprecation notes to before_compile() and similar
* new example suites to illustrate do_orm_execute(),
with_loader_criteria()
* modernized horizontal sharding examples and added a separate
example to distinguish between multiple databases and single
database w/ multiple tables use case
* introducing DEEP ALCHEMY, will use zzzeeksphinx 1.1.6
Mike Bayer [Thu, 6 Aug 2020 01:47:43 +0000 (21:47 -0400)]
Implement relationship AND criteria; global loader criteria
Added the ability to add arbitrary criteria to the ON clause generated
by a relationship attribute in a query, which applies to methods such
as :meth:`_query.Query.join` as well as loader options like
:func:`_orm.joinedload`. Additionally, a "global" version of the option
allows limiting criteria to be applied to particular entities in
a query globally.
Documentation is minimal at this point, new examples will
be coming in a subsequent commit.
Some adjustments to execution options in how they are represented
in the ORMExecuteState as well as well as a few ORM tests that
forgot to get merged in a preceding commit.
Mike Bayer [Thu, 6 Aug 2020 00:32:02 +0000 (20:32 -0400)]
Fixes for MySQL 8
MySQL 8.0.19 has some changes to how it reports on display widths
for int types (seems like it omits it in some cases), and also
no longer cares about the length for YEAR. Another adjustment
to the ordering of constraints reported in one case also.
At least one CI machine is at 8.0.21 now.
Mike Bayer [Wed, 5 Aug 2020 21:58:48 +0000 (17:58 -0400)]
Convert lazy loader, selectinload, load_on_ident to lambda statements
Building on newly robust lambdas in
I29a513c98917b1d503abfdd61e6b6e8800851aa8,
convert key loading off of the "baked" system so that baked
is no longer used by the ORM.
Mike Bayer [Wed, 5 Aug 2020 20:42:26 +0000 (16:42 -0400)]
Robustness for lambdas, lambda statements
in order to accommodate relationship loaders
with lambda caching, a lot more is needed. This is
a full refactor of the lambda system such that it
now has two levels of caching; the first level caches what
can be known from the __code__ element, then the next level
of caching is against the lambda itself and the contents
of __closure__. This allows for the elements inside
the lambdas, like columns and entities, to change and
then be part of the cache key. Lazy/selectinloads' use of
baked queries had to add distinct cache key elements,
which was attempted here but overall things needed to be
more robust than that.
This commit is broken out from the very long and sprawling
commit at Id6b5c03b1ce9ddb7b280f66792212a0ef0a1c541 .
Mike Bayer [Sat, 1 Aug 2020 17:57:04 +0000 (13:57 -0400)]
Genericize str() for types
Remove lookup logic that attempts to locate a dialect for a type,
just use StrSQLTypeCompiler.
Cleaned up the internal ``str()`` for datatypes so that all types produce a
string representation without any dialect present, including that it works
for third-party dialect types without that dialect being present. The
string representation defaults to being the UPPERCASE name of that type
with nothing else.
Mike Bayer [Sat, 1 Aug 2020 19:05:53 +0000 (15:05 -0400)]
Establish future behavior for Session cascade backrefs, bind
The behavior of the :paramref:`_orm.relationship.cascade_backrefs` flag
will be reversed in 2.0 and set to ``False`` unconditionally, such that
backrefs don't cascade save-update operations from a forwards-assignment to
a backwards assignment. A 2.0 deprecation warning is emitted when the
parameter is left at its default of ``True`` at the point at which such a
cascade operation actually takes place. The new behavior can be
established as always by setting the flag to ``False`` on a specific
:func:`_orm.relationship`, or more generally can be set up across the board
by setting the the :paramref:`_orm.Session.future` flag to True.
Additionally in the interests of expediency, this commit will also
move Session away from making use of bound metadata if the future=True
flag is set. An application that sets future=True should ideally
have to change as little else as possible for full 2.0 behavior.
Mike Bayer [Sat, 9 May 2020 14:55:19 +0000 (10:55 -0400)]
Unpin pydocstyle, minimum for flake8-docstrings 1.3.1
Per https://gitlab.com/pycqa/flake8-docstrings/-/issues/36#note_199635329
flake8-docstrings has repaired the pydocstyle issue as of
1.3.1, so unpin pydocstyle.
Mike Bayer [Tue, 28 Jul 2020 20:46:53 +0000 (16:46 -0400)]
Consider default FROM DUAL for MySQL
MySQL claims it doesn't require FROM DUAL for no
FROM clause even though the issue at #5481 locates
a case which requires one. See if FROM DUAL the same
way as Oracle without attempting to guess is potentially
feasible.
- in create wheel set minimum versions of setuptools and wheel to avoid
failure in python 3.5 with metadata configured in setup.cfg
- update action versions
- test also cext in the pull requests
Jim Bosch [Sun, 26 Jul 2020 20:50:14 +0000 (16:50 -0400)]
Ensure is_comparison passed for PG RANGE op() methods
Fixed issue where the return type for the various RANGE comparison
operators would itself be the same RANGE type rather than BOOLEAN, which
would cause an undesirable result in the case that a
:class:`.TypeDecorator` that defined result-processing behavior were in
use. Pull request courtesy Jim Bosch.
Mike Bayer [Sun, 26 Jul 2020 17:17:56 +0000 (13:17 -0400)]
Improve delete cascade and passive deletes sections
Add cross linking from Core to ORM and also add new sections
and examples regarding many-to-many. Move the section
out of "collections" and into the "cascades" chapter
where it's more likely to be found.
PyODBCConnector.initialize just super from Connector, no need to redeclare, so I guess that's why they are commented before.
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
Remove the useless comment code lines.
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Tue, 21 Jul 2020 16:36:20 +0000 (12:36 -0400)]
Allow Grouping to pass along proxy_set of element
Repaired an issue where the "ORDER BY" clause rendering a label name rather
than a complete expression, which is particularly important for SQL Server,
would fail to occur if the expression were enclosed in a parenthesized
grouping in some cases. This case has been added to test support.
Mike Bayer [Sun, 19 Jul 2020 21:39:14 +0000 (17:39 -0400)]
Revise setinputsizes approach
in order to support asyncpg as well as pg8000,
we need to revise setinputsizes to work for more cases as well
as adjust NativeForEmulated a bit to work more completely with
the INTERVAL datatype.
- put most of the setinputsizes work into the compiler where
the computation can be cached.
- support per-element setinputsizes for a tuple
- adjust TypeDecorator so that _unwrapped_dialect_impl
will honor a type that the dialect links to directly in
it's adaption mapping. Decouble _unwrapped_dialect_impl
from TypeDecorator._gen_dialect_impl() which has a different
purpose. This allows setinputsizes to do the right thing
with the INTERVAL datatype.
- test cases for Oracle with Variant continue to work
Mike Bayer [Mon, 13 Jul 2020 14:49:57 +0000 (10:49 -0400)]
test single and double quote inspection scenarios
Applied a sweep through all included dialects to ensure names that contain
single or double quotes are properly escaped when querying system tables,
for all :class:`.Inspector` methods that accept object names as an argument
(e.g. table names, view names, etc). SQLite and MSSQL contained two
quoting issues that were repaired.
mock in python 2.7 / 3.8 seems to have an "args" accessor
but in prior python 3 versions this is not the case,
so this is likely a recent addition to "mock" but wasn't
present in the python3 vendored versions.
Mike Bayer [Sat, 11 Jul 2020 22:54:15 +0000 (18:54 -0400)]
Further corrections to session docs
many more to come.
One issue is that references to documentation sections
inside of decorator-configured deprecation warnings will
also generate a meaningless token when the runtime warning
is emitted. It would be nice to improve upon this somehow.
Mike Bayer [Wed, 8 Jul 2020 18:31:17 +0000 (14:31 -0400)]
Convert remaining ORM APIs to support 2.0 style
This is kind of a mixed bag of all kinds to help get us
to 1.4 betas. The documentation stuff is a work in
progress. Lots of other relatively small changes to
APIs and things. More commits will follow to continue
improving the documentation and transitioning to the
1.4/2.0 hybrid documentation. In particular some refinements
to Session usage models so that it can match Engine's
scoping / transactional patterns, and a decision to
start moving away from "subtransactions" completely.
* add select().from_statement() to produce FromStatement in an
ORM context
* begin referring to select() that has "plugins" for the few edge
cases where select() will have ORM-only behaviors
* convert dynamic.AppenderQuery to its own object that can use
select(), though at the moment it uses Query to support legacy
join calling forms.
* custom query classes for AppenderQuery are replaced by
do_orm_execute() hooks for custom actions, a separate gerrit
will document this
* add Session.get() to replace query.get()
* Deprecate session.begin->subtransaction. propose within the
test suite a hypothetical recipe for apps that rely on this
pattern
* introduce Session construction level context manager,
sessionmaker context manager, rewrite the whole top of the
session_transaction.rst documentation. Establish context manager
patterns for Session that are identical to engine
* ensure same begin_nested() / commit() behavior as engine
* devise all new "join into an external transaction" recipe,
add test support for it, add rules into Session so it
just works, write new docs. need to ensure this doesn't
break anything
* vastly reduce the verbosity of lots of session docs as
I dont think people read this stuff and it's difficult
to keep current in any case
* constructs like case(), with_only_columns() really need
to move to *columns, add a coercion rule to just change
these.
* docs need changes everywhere I look. in_() is not in
the Core tutorial? how do people even know about it?
Remove tons of cruft from Select docs, etc.
* build a system for common ORM options like populate_existing
and autoflush to populate from execution options.
<!-- Provide a general summary of your proposed changes in the Title field above -->
### Description
<!-- Describe your changes in detail -->
- Update the test so it looks at computed_default_table instead of
computed_column_table.
- Stop inspecting autoincrement; instead directly check for
default/non-default values for the relevant columns.
Fixes: #5414
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Wed, 8 Jul 2020 19:48:32 +0000 (15:48 -0400)]
Limit init_subclass test above python 3.6
The commit for I1654befe9eb1c8b8e7fc0784bdbe64284614f0ea #5357
runs the test on all Python 3 versions, however we need to limit
at least python 3.6 for this.
Mike Bayer [Wed, 8 Jul 2020 19:32:38 +0000 (15:32 -0400)]
Propose --nomemory by default
the memory tests are solely responsible for the largest chunk of
time and CPU energy taken up in running the tests. Regressions
in this area are nonexistent unless major changes are being
taken up. Try to revert them to False and see if a single
gerrit job can perhaps run these.
Mike Bayer [Wed, 8 Jul 2020 19:12:30 +0000 (15:12 -0400)]
Repair change of "future_select" caught in merge conflict
the recent change to select() apparently seems to have
been merged with other statements in the test_update_delete
test that was still using the old term.
Mike Bayer [Fri, 26 Jun 2020 20:15:19 +0000 (16:15 -0400)]
Add future=True to create_engine/Session; unify select()
Several weeks of using the future_select() construct
has led to the proposal there be just one select() construct
again which features the new join() method, and otherwise accepts
both the 1.x and 2.x argument styles. This would make
migration simpler and reduce confusion.
However, confusion may be increased by the fact that select().join()
is different Current thinking is we may be better off
with a few hard behavioral changes to old and relatively unknown APIs
rather than trying to play both sides within two extremely similar
but subtly different APIs. At the moment, the .join() thing seems
to be the only behavioral change that occurs without the user
taking any explicit steps. Session.execute() will still
behave the old way as we are adding a future flag.
This change also adds the "future" flag to Session() and
session.execute(), so that interpretation of the incoming statement,
as well as that the new style result is returned, does not
occur for existing applications unless they add the use
of this flag.
The change in general is moving the "removed in 2.0" system
further along where we want the test suite to fully pass
even if the SQLALCHEMY_WARN_20 flag is set.
Get many tests to pass when SQLALCHEMY_WARN_20 is set; this
should be ongoing after this patch merges.
Improve the RemovedIn20 warning; these are all deprecated
"since" 1.4, so ensure that's what the messages read.
Make sure the inforamtion link is on all warnings.
Add deprecation warnings for parameters present and
add warnings to all FromClause.select() types of methods.
Imply `sync_backref` flag in a viewonly relationship
Update :paramref:`_orm.relationship.sync_backref` flag in a relationship
to make it implicitly set to False in ``viewonly=True`` relationships,
preventing synchronization events in all cases.
Mike Bayer [Tue, 7 Jul 2020 15:26:39 +0000 (11:26 -0400)]
ensure we unwrap desc() /label() all the way w/ order by
The deprecated logic to move order_by expressions
up into the columns clause needed adjustment to accommodate
for a more deeply-wrapped structure when desc() + label()
are combined in an order by column. This structure
now comes from coercions in 1.4. it's not clear to me
at the moment why it's different from 1.3 but
this shouldn't really matter.
Added a ``**kw`` argument to the :meth:`.DeclarativeMeta.__init__` method.
This allows a class to support the :pep:`487` metaclass hook
``__init_subclass__``. Pull request courtesy Ewen Gillies.
1. move all discussion of how to customize connection
to the engines.rst section. have the section in pooling.rst
point to engines.
2. ensure all of the common use cases for connection modification
are listed out in order of level of modification. the use
case of modifying an existing connection is separate from that of
modifying how connect is called; ensure that poolevents.connect
is referred to just as prominently as dialectevents.do_connect.
3. completely replace any discussion of create_engine.creator, as this
hook does not offer anything beyond what do_connect() does.
"creator" is more about using a Pool object directly without an
Engine, which is no longer a documented use case.
projects / thirdparty / sqlalchemy / sqlalchemy.git / log
