Mike Bayer [Tue, 7 Apr 2020 21:37:14 +0000 (17:37 -0400)]
Use dot-separated name resolution for relationship target
The string argument accepted as the first positional argument by the
:func:`.relationship` function when using the Declarative API is no longer
interpreted using the Python ``eval()`` function; instead, the name is dot
separated and the names are looked up directly in the name resolution
dictionary without treating the value as a Python expression. However,
passing a string argument to the other :func:`.relationship` parameters
that necessarily must accept Python expressions will still use ``eval()``;
the documentation has been clarified to ensure that there is no ambiguity
that this is in use.
Mike Bayer [Tue, 7 Apr 2020 23:21:01 +0000 (19:21 -0400)]
Gracefully skip isolation level if no row returned
Fixed issue in MySQL dialect when connecting to a psuedo-MySQL database
such as that provided by ProxySQL, the up front check for isolation level
when it returns no row will not prevent the dialect from continuing to
connect. A warning is emitted that the isolation level could not be
detected.
Federico Caselli [Tue, 24 Mar 2020 20:06:04 +0000 (21:06 +0100)]
Add length parameter in `Enum`
The `Enum` type now supports the parameter `Enum.length`
to specify the length of the VARCHAR column to create when using
non native enums by setting `Enum.native_enum` to `False`
Mike Bayer [Fri, 3 Apr 2020 20:00:22 +0000 (16:00 -0400)]
Key subqueryloaders on the property object, not string key
Fixed bug in :func:`.orm.selectinload` loading option where two or more
loaders that represent different relationships with the same string key
name as referenced from a single :func:`.orm.with_polymorphic` construct
with multiple subclass mappers would fail to invoke each subqueryload
separately, instead making use of a single string-based slot that would
prevent the other loaders from being invoked.
Add github action workflow to run tests on master and on pr to master
Trying the pr to check if it works right away
### Description
<!-- Describe your changes in detail -->
### Checklist
<!-- go over following points. check them with an `x` if they do apply, (they turn into clickable checkboxes once the PR is submitted, so no need to do everything at once)
-->
This pull request is:
- [ ] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [x] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
Mike Bayer [Fri, 3 Apr 2020 00:45:44 +0000 (20:45 -0400)]
Run autoflush for column attribute load operations
The "autoflush" behavior of :class:`.Query` will now trigger for nearly
all ORM level attribute load operations, including when a deferred
column is loaded as well as when an expired column is loaded. Previously,
autoflush on load of expired or unloaded attributes was limited to
relationship-bound attributes only. However, this led to the issue
where column-based attributes that also depended on other rows, or even
other columns in the same row, in order to express the correct value,
would show an effectively stale value when accessed as there could be
pending changes in the session left to be flushed. Autoflush
is now disabled only in some cases where attributes are being unexpired in
the context of a history operation.
Mike Bayer [Wed, 1 Apr 2020 22:31:16 +0000 (18:31 -0400)]
Repair caching / traversals for values
The test suite wasn't running the copy_internals most fixtures,
enable that and try to get all cases working.
Set up selectable.values to do tuple conversion within compilation
step. at the same time, disable caching for selectable.values
for the moment and make it equivalent to dml_multi_values.
fix cache / compare / copy cases for dml_values and dml_multi_values
which weren't fully tested or covered.
Trying to see if by making the cache key memoized, we
still can have the older "identity" form of caching
which is the cheapest of all, at the same time as the
newer "cache key each time" version that is not nearly
as cheap; but still much cheaper than no caching at all.
Also needed is a per-execution update of _keymap when
we invoke from a cached select, so that Column objects
that are anonymous or otherwise adapted will match up.
this is analogous to the adaption of bound parameters
from the cache key.
Adds test coverage for the keymap / construct_params()
changes related to caching. Also hones performance
to a large extent for statement construction and
cache key generation.
Also includes a new memoized attribute
approach that vastly simplifies the previous approach
of "group_expirable_memoized_property" and finally
integrates cleanly with _clone(), _generate(), etc.
no more hardcoding of attributes is needed, as well
as that most _reset_memoization() calls are no longer
needed as the reset is inherent in a _generate() call;
this also has dramatic performance improvements.
Mike Bayer [Wed, 1 Apr 2020 17:51:35 +0000 (13:51 -0400)]
Fix databases -> dialects
The change in I5343a2d270ed5a8c654b9fe13dff40cdf54649ed
left the name "databases" present in one spot, which is usually
not reached unless tests are being run on only partial files
where not every dialect has been imported.
Mike Bayer [Mon, 30 Mar 2020 15:04:24 +0000 (11:04 -0400)]
Remove ORDER BY pk from subqueryload, selectinload
Modified the queries used by subqueryload and selectinload to no longer
ORDER BY the primary key of the parent entity; this ordering was there to
allow the rows as they come in to be copied into lists directly with a
minimal level of Python-side collation. However, these ORDER BY clauses
can negatively impact the performance of the query as in many scenarios
these columns are derived from a subquery or are otherwise not actual
primary key columns such that SQL planners cannot make use of indexes. The
Python-side collation uses the native itertools.group_by() to collate the
incoming rows, and has been modified to allow multiple
row-groups-per-parent to be assembled together using list.extend(), which
should still allow for relatively fast Python-side performance. There will
still be an ORDER BY present for a relationship that includes an explicit
order_by parameter, however this is the only ORDER BY that will be added to
the query for both kinds of loading.
Mike Bayer [Sun, 29 Mar 2020 18:24:39 +0000 (14:24 -0400)]
Add a third labeling mode for SELECT statements
Enhanced the disambiguating labels feature of the
:func:`~.sql.expression.select` construct such that when a select statement
is used in a subquery, repeated column names from different tables are now
automatically labeled with a unique label name, without the need to use the
full "apply_labels()" feature that conbines tablename plus column name.
The disambigated labels are available as plain string keys in the .c
collection of the subquery, and most importantly the feature allows an ORM
:func:`.orm.aliased` construct against the combination of an entity and an
arbitrary subquery to work correctly, targeting the correct columns despite
same-named columns in the source tables, without the need for an "apply
labels" warning.
The existing labeling style is now called
LABEL_STYLE_TABLENAME_PLUS_COL. This labeling style will remain used
throughout the ORM as has been the case for over a decade, however,
the new disambiguation scheme could theoretically replace this scheme
entirely. The new scheme would dramatically alter how SQL looks
when rendered from the ORM to be more succinct but arguably harder
to read.
The tablename_columnname scheme used by Join.c is unaffected here,
as that's still hardcoded to that scheme.
Federico Caselli [Tue, 24 Mar 2020 21:55:46 +0000 (22:55 +0100)]
String compiler can now literal compile datetime objects
Add ability to literal compile a :class:`DateTime`, :class:`Date`
or :class:"Time" when using the string dialect for debugging purposes.
This change does not impact real dialect implementation that retain
their current behavior.
Mike Bayer [Sun, 29 Mar 2020 16:06:29 +0000 (12:06 -0400)]
Repair queries from #5134 to ORDER BY outside the subquery
ORDER BY has to be on the outermost query to guarantee ordering
as these tests were failing for SQL Server. Also add
new tests that don't use from_self() as this is also going to
be removed in 2.0.
Also propose some from_self() alternatives. References #5221
Federico Caselli [Sat, 28 Mar 2020 10:04:44 +0000 (11:04 +0100)]
Fix typo in resultproxy.c and test compatibility with python 3.5
- Fix typo in resultproxy.c that would error on windows.
- add -Wundef to C flags when linux is detected so that undefined
symbols emit a warning
- a few adjustments for tests to succeed on python 3.5
- note minimum version still documented here as 3.4 but this should
move to at least 3.5 if not 3.6 for SQLAlchemy 1.4
Mike Bayer [Thu, 26 Mar 2020 21:26:57 +0000 (17:26 -0400)]
introduce "autobegin" concept for Connection
because engine.connect() and engine.begin() should feature
identical internal behavior, with the sole exception that
one rolls back and the end and the other commits at the end,
while also supporting execution options like transaction isolation level
at the connection level, include that engine.connect() will
return a connection that uses autobegin in the same way as the
session will. This is solely to support the "begin" event
noting that a transaction is begun which is tracked on the
connection. Behavior and design should be very similar to that
of the ORM session and "Transaction" should no longer be a very
explicit object.
Mike Bayer [Wed, 25 Mar 2020 15:34:19 +0000 (11:34 -0400)]
Correct ambiguous func / class links
:func:`.sql.expression.select`, :func:`.sql.expression.insert`
and :class:`.sql.expression.Insert` were hitting many ambiguous
symbol errors, due to future.select, as well as the PG/MySQL
variants of Insert.
Mike Bayer [Mon, 23 Mar 2020 18:52:05 +0000 (14:52 -0400)]
Convert schema_translate to a post compile
Revised the :paramref:`.Connection.execution_options.schema_translate_map`
feature such that the processing of the SQL statement to receive a specific
schema name occurs within the execution phase of the statement, rather than
at the compile phase. This is to support the statement being efficiently
cached. Previously, the current schema being rendered into the statement
for a particular run would be considered as part of the cache key itself,
meaning that for a run against hundreds of schemas, there would be hundreds
of cache keys, rendering the cache much less performant. The new behavior
is that the rendering is done in a similar manner as the "post compile"
rendering added in 1.4 as part of :ticket:`4645`, :ticket:`4808`.
Gord Thompson [Fri, 20 Dec 2019 00:58:52 +0000 (19:58 -0500)]
Implement SQL VALUES in core.
Added a core :class:`Values` object that enables a VALUES construct
to be used in the FROM clause of an SQL statement for databases that
support it (mainly PostgreSQL and SQL Server).
Mike Bayer [Tue, 24 Mar 2020 16:51:53 +0000 (12:51 -0400)]
Reword changelog for issue #5207
SQLAlchemy can remain using setuptools even when pep-517 is
the standard installation process. We are omitting the file
for now because it leads to new pip installation issues that need
to be worked out in the greater pip community before SQLAlchemy
opts into this system.
Mike Bayer [Sun, 2 Feb 2020 16:39:37 +0000 (11:39 -0500)]
Remove deprecated elements from selectable.py; remove lockmode
Removed autocommit and legacy "for update" / "lockmode" elements
from selectable.py / query.py. lockmode was removed from
selectable in 693938dd6fb2f3ee3e031aed4c62355ac97f3ceb
however was not removed from the ORM.
Also removes the ignore_nonexistent_tables option on
join().
Gord Thompson [Thu, 12 Mar 2020 18:54:37 +0000 (12:54 -0600)]
Implement autocommit isolation level for cx_oracle
As part of this change Oracle also gets the concept of a
default isolation level, however since Oracle does not provide a
fixed method of knowing what the isolation level would be without a
server side transaction actually in progress, for now we hardcode
just to "READ COMMITTED".
Enhanced the test suite for isolation level testing in the dialect
test suite and added features to requirements so that the supported
isolation levels can be reported generically for dialects.
Co-authored-by: Mike Bayer <mike_mp@zzzcomputing.com> Fixes: #5200
Change-Id: I2c4d49da9ff80ccc228c21e196ec9a961de53478
Mike Bayer [Sat, 21 Mar 2020 21:26:24 +0000 (17:26 -0400)]
Test instance for matching class hierarchy on get_from_identity
Fixed issue where a lazyload that uses session-local "get" against a target
many-to-one relationship where an object with the correct primary key is
present, however it's an instance of a sibling class, does not correctly
return None as is the case when the lazy loader actually emits a load for
that row.
Federico Caselli [Sat, 14 Mar 2020 13:02:44 +0000 (14:02 +0100)]
Deprecate plain string in execute and introduce `exec_driver_sql`
Execution of literal sql string is deprecated in the
:meth:`.Connection.execute` and a warning is raised when used stating
that it will be coerced to :func:`.text` in a future release.
To execute a raw sql string the new connection method
:meth:`.Connection.exec_driver_sql` was added, that will retain the previous
behavior, passing the string to the DBAPI driver unchanged.
Usage of scalar or tuple positional parameters in :meth:`.Connection.execute`
is also deprecated.
Mike Bayer [Sat, 21 Mar 2020 14:16:47 +0000 (10:16 -0400)]
Remove pyproject.toml from distribution
SQLAlchemy does not want to opt-in to pep-517 at this time
as this would require a custom build backend interface
which we have not built yet, and the standard is not
widely adopted at this time in any case. Per
[1] [2], the presence of this file indicates a positive opt-in
to pep-517, so it must be omitted from source distributions.
Mike Bayer [Fri, 20 Mar 2020 20:40:47 +0000 (16:40 -0400)]
Clarify Alembic vs. Migrate
Migrate is fully legacy at this point so continue to mention
it to provide clarity, but ensure it isn't suggested
as a real alternative. It's unclear if Migrate will be able
to support SQLAlchemy 2.0.
Mike Bayer [Fri, 20 Mar 2020 19:21:10 +0000 (15:21 -0400)]
Emphasize context managers when working with Core
Prep the main documentation for the changes coming up in 1.4
by first removing references to engine.execute() outside
of the "connectionless" section, and using context managers
in all cases. For features that have always been confusing
and are going away, add a note that this feature will
be going away.
Mike Bayer [Fri, 20 Mar 2020 15:17:47 +0000 (11:17 -0400)]
Correct misleading guidance on multiprocessing
Link the connections intro to the dedicated section
on multiprocessing rather than stating that a separate
engine per process is needed, since this is inaccurate
and vague.
mike bayer [Wed, 18 Mar 2020 23:05:20 +0000 (19:05 -0400)]
Don't include PG INCLUDE columns as regular index columns
Fixed issue where a "covering" index, e.g. those which have an INCLUDE
clause, would be reflected including all the columns in INCLUDE clause as
regular columns. A warning is now emitted if these additional columns are
detected indicating that they are currently ignored. Note that full
support for "covering" indexes is part of :ticket:`4458`. Pull request
courtesy Marat Sharafutdinov.
Mike Bayer [Fri, 13 Mar 2020 22:37:25 +0000 (18:37 -0400)]
import the module as passed before we look in sys.modules
in I2e291eba4297867fc0ddb5d875b9f7af34751d01, the importer
is importing tokens[0:-1] and then the final token as the name;
this doesn't always work for the two example modules of "xml.dom"
and "wsgiref.simple_server". import the complete path we will
pull from sys.modules directly.
Federico Caselli [Sat, 14 Mar 2020 12:57:42 +0000 (13:57 +0100)]
Support inspection of computed column
Added support for reflection of "computed" columns, which are now returned
as part of the structure returned by :meth:`.Inspector.get_columns`.
When reflecting full :class:`.Table` objects, computed columns will
be represented using the :class:`.Computed` construct.
Also improve the documentation in :meth:`Inspector.get_columns`, correctly
listing all the returned keys.
jonathan vanasco [Sat, 14 Mar 2020 12:27:43 +0000 (08:27 -0400)]
updated doc string to show `NOT (EXISTS` instead of `NOT EXISTS`, as …
…that is what SqlAlchemy now renders.
SqlAlchemy currently renders `NOT (EXISTS` not `NOT EXISTS`. This fixes the example SQL.
### Description
The change is a docstring only.
### Checklist
This pull request is:
- [x] A documentation / typographical error fix
- Good to go, no issue or tests are needed
- [ ] A short code fix
- please include the issue number, and create an issue if none exists, which
must include a complete example of the issue. one line code fixes without an
issue and demonstration will not be accepted.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests. one line code fixes without tests will not be accepted.
- [ ] A new feature implementation
- please include the issue number, and create an issue if none exists, which must
include a complete example of how the feature would look.
- Please include: `Fixes: #<issue number>` in the commit message
- please include tests.
2. add some exclusion rules to allow the sqlite_file target to work;
add to tox.
3. add reap dbs target for SQLite, repair SQLite drop_db routine
which also wasn't doing the right thing for memory databases
etc.
4. Fix logging in provision files, as the main provision logger
is the one that's enabled by reap_dbs and maybe others, have all
the provision files use the provision logger.
Mike Bayer [Fri, 13 Mar 2020 03:11:18 +0000 (23:11 -0400)]
Include schema in all_tab_comments query
Fixed regression / incorrect fix caused by fix for :ticket:`5146` where the
Oracle dialect reads from the "all_tab_comments" view to get table comments
but fails to accommodate for the current owner of the table being
requested, causing it to read the wrong comment if multiple tables of the
same name exist in multiple schemas.
Mike Bayer [Thu, 12 Mar 2020 23:44:37 +0000 (19:44 -0400)]
Dont raise on pytest deprecation warnings
py.test 5.4.0 emits deprecation warnings for pytest.Class.
make sure we don't raise for these, and log the code that will
be used for 5.4.0 when we bump requirements.
Mike Bayer [Wed, 11 Mar 2020 14:41:12 +0000 (10:41 -0400)]
Repair broken call to sys.exc_info()
Fixed regression in 1.3.14 due to :ticket:`4849` where a sys.exc_info()
call failed to be invoked correctly when a flush error would occur. Test
coverage has been added for this exception case.
Mike Bayer [Tue, 10 Mar 2020 22:48:42 +0000 (18:48 -0400)]
Reword implicit left join error; ensure deterministic FROM for columns
Adjusted the error message emitted by :meth:`.Query.join` when a left hand
side can't be located that the :meth:`.Query.select_from` method is the
best way to resolve the issue. Also, within the 1.3 series, used a
deterministic ordering when determining the FROM clause from a given column
entity passed to :class:`.Query` so that the same expression is determined
each time.