From: Mike Bayer Date: Thu, 13 Jun 2019 16:37:22 +0000 (-0400) Subject: SelectBase no longer a FromClause X-Git-Tag: rel_1_4_0b1~806^2 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=ef7ff058eb67d73ebeac7b125ab2a7806e14629c;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git SelectBase no longer a FromClause As part of the SQLAlchemy 2.0 migration project, a conceptual change has been made to the role of the :class:`.SelectBase` class hierarchy, which is the root of all "SELECT" statement constructs, in that they no longer serve directly as FROM clauses, that is, they no longer subclass :class:`.FromClause`. For end users, the change mostly means that any placement of a :func:`.select` construct in the FROM clause of another :func:`.select` requires first that it be wrapped in a subquery first, which historically is through the use of the :meth:`.SelectBase.alias` method, and is now also available through the use of :meth:`.SelectBase.subquery`. This was usually a requirement in any case since several databases don't accept unnamed SELECT subqueries in their FROM clause in any case. See the documentation in this change for lots more detail. Fixes: #4617 Change-Id: I0f6174ee24b9a1a4529168e52e855e12abd60667 --- diff --git a/doc/build/changelog/migration_14.rst b/doc/build/changelog/migration_14.rst index 3127ca977b..f0e99e7eab 100644 --- a/doc/build/changelog/migration_14.rst +++ b/doc/build/changelog/migration_14.rst @@ -7,6 +7,183 @@ What's New in SQLAlchemy 1.4? This document describes changes between SQLAlchemy version 1.3 and SQLAlchemy version 1.4. + Version 1.4 is taking on a different focus than other SQLAlchemy releases + in that it is in many ways attempting to serve as a potential migration + point for a more dramatic series of API changes currently planned for + release 2.0 of SQLAlchemy. The focus of SQLAlchemy 2.0 is a modernized + and slimmed down API that removes lots of usage patterns that have long + been discouraged, as well as mainstreams the best ideas in SQLAlchemy as + first class API features, with the goal being that there is much less + ambiguity in how the API is to be used, as well as that a series of + implicit behaviors and rarely-used API flags that complicate the internals + and hinder performance will be removed. + +API Changes - Core +================== + +.. _change_4617: + +A SELECT statement is no longer implicitly considered to be a FROM clause +-------------------------------------------------------------------------- + +This change is one of the larger conceptual changes in SQLAlchemy in many years, +however it is hoped that the end user impact is relatively small, as the change +more closely matches what databases like MySQL and PostgreSQL require in any case. + +The most immediate noticeable impact is that a :func:`.select` can no longer +be embedded inside of another :func:`.select` directly, without explicitly +turning the inner :func:`.select` into a subquery first. This is historically +performed by using the :meth:`.SelectBase.alias` method, which remains, however +is more explicitly suited by using a new method :meth:`.SelectBase.subquery`; +both methods do the same thing. The object returned is now :class:`.Subquery`, +which is very similar to the :class:`.Alias` object and shares a common +base :class:`.AliasedReturnsRows`. + +That is, this will now raise:: + + stmt1 = select([user.c.id, user.c.name]) + stmt2 = select([addresses, stmt1]).select_from(addresses.join(stmt1)) + +Raising:: + + sqlalchemy.exc.ArgumentError: Column expression or FROM clause expected, + got <...Select object ...>. To create a FROM clause from a object, use the .subquery() method. + +The correct calling form is instead:: + + sq1 = select([user.c.id, user.c.name]).subquery() + stmt2 = select([addresses, sq1]).select_from(addresses.join(sq1)) + +Noting above that the :meth:`.SelectBase.subquery` method is essentially +equivalent to using the :meth:`.SelectBase.alias` method. + +The above calling form is typically required in any case as the call to +:meth:`.SelectBase.subquery` or :meth:`.SelectBase.alias` is needed to +ensure the subquery has a name. The MySQL and PostgreSQL databases do not +accept unnamed subqueries in the FROM clause and they are of limited use +on other platforms; this is described further below. + +Along with the above change, the general capability of :func:`.select` and +related constructs to create unnamed subqueries, which means a FROM subquery +that renders without any name i.e. "AS somename", has been removed, and the +ability of the :func:`.select` construct to implicitly create subqueries +without explicit calling code to do so is mostly deprecated. In the above +example, as has always been the case, using the :meth:`.SelectBase.alias` +method as well as the new :meth:`.SelectBase.subquery` method without passing a +name will generate a so-called "anonymous" name, which is the familiar +``anon_1`` name we see in SQLAlchemy queries:: + + SELECT + addresses.id, addresses.email, addresses.user_id, + anon_1.id, anon_1.name + FROM + addresses JOIN + (SELECT users.id AS id, users.name AS name FROM users) AS anon_1 + ON addresses.user_id = anon_1.id + +Unnamed subqueries in the FROM clause (which note are different from +so-called "scalar subqueries" which take the place of a column expression +in the columns clause or WHERE clause) are of extremely limited use in SQL, +and their production in SQLAlchemy has mostly presented itself as an +undesirable behavior that needs to be worked around. For example, +both the MySQL and PostgreSQL outright reject the usage of unnamed subqueries:: + + # MySQL / MariaDB: + + MariaDB [(none)]> select * from (select 1); + ERROR 1248 (42000): Every derived table must have its own alias + + + # PostgreSQL: + + test=> select * from (select 1); + ERROR: subquery in FROM must have an alias + LINE 1: select * from (select 1); + ^ + HINT: For example, FROM (SELECT ...) [AS] foo. + +A database like SQLite accepts them, however it is still often the case that +the names produced from such a subquery are too ambiguous to be useful:: + + sqlite> CREATE TABLE a(id integer); + sqlite> CREATE TABLE b(id integer); + sqlite> SELECT * FROM a JOIN (SELECT * FROM b) ON a.id=id; + Error: ambiguous column name: id + sqlite> SELECT * FROM a JOIN (SELECT * FROM b) ON a.id=b.id; + Error: no such column: b.id + + # use a name + sqlite> SELECT * FROM a JOIN (SELECT * FROM b) AS anon_1 ON a.id=anon_1.id; + +Due to the above limitations, there are very few places in SQLAlchemy where +such a query form was valid; the one exception was within the Oracle dialect +where they were used to create OFFSET / LIMIT subqueries as Oracle does not +support these keywords directly; this implementation has been replaced by +one which uses anonymous subqueries. Throughout the ORM, exception cases +that detect where a SELECT statement would be SELECTed from either encourage +the user to, or implicitly create, an anonymously named subquery; it is hoped +by moving to an all-explicit subquery much of the complexity incurred by +these areas can be removed. + +As :class:`.SelectBase` objects are no longer :class:`.FromClause` objects, +attributes like the ``.c`` attribute as well as methods like ``.select()``, +``.join()``, and ``.outerjoin()`` upon :class:`.SelectBase` are now +deprecated, as these methods all imply implicit production of a subquery. +Instead, as is already what the vast majority of applications have to do +in any case, invoking :meth:`.SelectBase.alias` or :meth:`.SelectBase.subquery` +will provide for a :class:`.Subquery` object that provides all these attributes, +as it is part of the :class:`.FromClause` hierarchy. In the interim, these +methods are still available, however they now produce an anonymously named +subquery rather than an unnamed one, and this subquery is distinct from the +:class:`.SelectBase` construct itself. + +In place of the ``.c`` attribute, a new attribute :attr:`.SelectBase.selected_columns` +is added. This attribute resolves to a column collection that is what most +people hope that ``.c`` does (but does not), which is to reference the columns +that are in the columns clause of the SELECT statement. A common beginner mistake +is code such as the following:: + + stmt = select([users]) + stmt = stmt.where(stmt.c.name == 'foo') + +The above code appears intuitive and that it would generate +"SELECT * FROM users WHERE name='foo'", however veteran SQLAlchemy users will +recognize that it in fact generates a useless subquery resembling +"SELECT * FROM (SELECT * FROM users) WHERE name='foo'". + +The new :attr:`.SelectBase.selected_columns` attribute however **does** suit +the use case above, as in a case like the above it links directly to the columns +present in the ``users.c`` collection:: + + stmt = select([users]) + stmt = stmt.where(stmt.selected_columns.name == 'foo') + +There is of course the notion that perhaps ``.c`` on :class:`.SelectBase` could +simply act the way :attr:`.SelectBase.selected_columns` does above, however in +light of the fact that ``.c`` is strongly associated with the :class:`.FromClause` +hierarchy, meaning that it is a set of columns that can be directly in the +FROM clause of another SELECT, it's better that a column collection that +serves an entirely different purpose have a new name. + +In the bigger picture, the reason this change is being made now is towards the +goal of unifying the ORM :class:`.Query` object into the :class:`.SelectBase` +hierarchy in SQLAlchemy 2.0, so that the ORM will have a "``select()``" +construct that extends directly from the existing :func:`.select` object, +having the same methods and behaviors except that it will have additional ORM +functionality. All statement objects in Core will also be fully cacheable +using a new system that resembles "baked queries" except that it will work +transparently for all statements across Core and ORM. In order to achieve +this, the Core class hierarchy needs to be refined to behave in such a way that +is more easily compatible with the ORM, and the ORM class hierarchy needs to be +refined so that it is more compatible with Core. + + +:ticket:`4617` + + + + Behavioral Changes - ORM ======================== diff --git a/doc/build/changelog/unreleased_14/4617_implicit_subquery.rst b/doc/build/changelog/unreleased_14/4617_implicit_subquery.rst index 6eee63cf90..32ad9a2cd7 100644 --- a/doc/build/changelog/unreleased_14/4617_implicit_subquery.rst +++ b/doc/build/changelog/unreleased_14/4617_implicit_subquery.rst @@ -2,13 +2,37 @@ :tags: change, sql :tickets: 4617 - Added new method :meth:`.SelectBase.subquery`, which creates a subquery - that is essentially the same thing as what calling - :meth:`.FromClause.alias` has always done, e.g. creates a named subquery. - This method is intended to roughly mirror the same role as that of - :meth:`.Query.subquery`. The :meth:`.SelectBase.alias` method is - being kept for the time being as essentially the same function as that - of :meth:`.SelectBase.subquery`. + As part of the SQLAlchemy 2.0 migration project, a conceptual change has + been made to the role of the :class:`.SelectBase` class hierarchy, + which is the root of all "SELECT" statement constructs, in that they no + longer serve directly as FROM clauses, that is, they no longer subclass + :class:`.FromClause`. For end users, the change mostly means that any + placement of a :func:`.select` construct in the FROM clause of another + :func:`.select` requires first that it be wrapped in a subquery first, + which historically is through the use of the :meth:`.SelectBase.alias` + method, and is now also available through the use of + :meth:`.SelectBase.subquery`. This was usually a requirement in any + case since several databases don't accept unnamed SELECT subqueries + in their FROM clause in any case. + + .. seealso:: + + :ref:`change_4617` + +.. change:: + :tags: change, sql + :tickets: 4617 + + Added a new Core class :class:`.Subquery`, which takes the place of + :class:`.Alias` when creating named subqueries against a :class:`.SelectBase` + object. :class:`.Subquery` acts in the same way as :class:`.Alias` + and is produced from the :meth:`.SelectBase.subquery` method; for + ease of use and backwards compatibility, the :meth:`.SelectBase.alias` + method is synonymous with this new method. + + .. seealso:: + + :ref:`change_4617` .. change:: :tags: change, orm @@ -22,12 +46,26 @@ :class:`.Query` object is passed directly to these functions and others, the ORM is typically coercing them to be a subquery by calling the :meth:`.SelectBase.alias` method automatically (which is now superceded by - the :meth:`.SelectBase.subquery method). The historical reason is that - most databases other than SQLite don't allow a SELECT of a SELECT without - the inner SELECT being a named subuqery in any case; going forward, - SQLAlchemy Core is moving towards no longer considering a SELECT statement - that isn't inside a subquery to be a "FROM" clause, that is, an object that - can be selected from, in the first place, as part of a larger change to - unify the interfaces for :func:`.select` and :meth:`.Query`. The change is - intended to encourage code to make explicit those places where these - subqueries have normally been implicitly created. + the :meth:`.SelectBase.subquery` method). See the migration notes linked + below for further details. + + .. seealso:: + + :ref:`change_4617` + +.. change:: + :tags: bug, sql + :tickets: 4617 + + The ORDER BY clause of a :class:`.CompoundSelect`, e.g. UNION, EXCEPT, etc. + will not render the table name associated with a given column when applying + :meth:`.CompoundSelect.order_by` in terms of a :class:`.Table` - bound + column. Most databases require that the names in the ORDER BY clause be + expressed as label names only which are matched to names in the first + SELECT statement. The change is related to :ticket:`4617` in that a + previous workaround was to refer to the ``.c`` attribute of the + :class:`.CompoundSelect` in order to get at a column that has no table + name. As the subquery is now named, this change allows both the workaround + to continue to work, as well as allows table-bound columns as well as the + :attr:`.CompoundSelect.selected_columns` collections to be usable in the + :meth:`.CompoundSelect.order_by` method. \ No newline at end of file diff --git a/doc/build/changelog/unreleased_14/4617_scalar.rst b/doc/build/changelog/unreleased_14/4617_scalar.rst index 3f22414f7f..d47b3e464a 100644 --- a/doc/build/changelog/unreleased_14/4617_scalar.rst +++ b/doc/build/changelog/unreleased_14/4617_scalar.rst @@ -3,17 +3,19 @@ :tickets: 4617 The :meth:`.SelectBase.as_scalar` and :meth:`.Query.as_scalar` methods have - been renamed to :meth:`.SelectBase.scalar_subquery` and :meth:`.Query.scalar_subquery`, - respectively. The old names continue to exist within 1.4 series with a deprecation - warning. In addition, the implicit coercion of :class:`.SelectBase`, :class:`.Alias`, - and other SELECT oriented objects into scalar subqueries when evaluated in a column - context is also deprecated, and emits a warning that the :meth:`.SelectBase.scalar_subquery` - method should be called explicitly. This warning will in a later major release - become an error, however the message will always be clear when :meth:`.SelectBase.scalar_subquery` - needs to be invoked. The latter part of the change is for clarity and to reduce the - implicit decisionmaking by the query coercion system. + been renamed to :meth:`.SelectBase.scalar_subquery` and + :meth:`.Query.scalar_subquery`, respectively. The old names continue to + exist within 1.4 series with a deprecation warning. In addition, the + implicit coercion of :class:`.SelectBase`, :class:`.Alias`, and other + SELECT oriented objects into scalar subqueries when evaluated in a column + context is also deprecated, and emits a warning that the + :meth:`.SelectBase.scalar_subquery` method should be called explicitly. + This warning will in a later major release become an error, however the + message will always be clear when :meth:`.SelectBase.scalar_subquery` needs + to be invoked. The latter part of the change is for clarity and to reduce + the implicit decisionmaking by the query coercion system. - This change is part of the larger change to convert :func:`.select` objects to no - longer be directly part of the "from clause" class hierarchy, which also includes - an overhaul of the clause coercion system. + This change is part of the larger change to convert :func:`.select` objects + to no longer be directly part of the "from clause" class hierarchy, which + also includes an overhaul of the clause coercion system. diff --git a/doc/build/core/selectable.rst b/doc/build/core/selectable.rst index 57e389722e..14099b6c76 100644 --- a/doc/build/core/selectable.rst +++ b/doc/build/core/selectable.rst @@ -29,8 +29,6 @@ elements are themselves :class:`.ColumnElement` subclasses). .. autofunction:: select -.. autofunction:: subquery - .. autofunction:: sqlalchemy.sql.expression.table .. autofunction:: tablesample @@ -43,6 +41,10 @@ elements are themselves :class:`.ColumnElement` subclasses). :members: :inherited-members: +.. autoclass:: AliasedReturnsRows + :members: + :inherited-members: + .. autoclass:: CompoundSelect :members: :inherited-members: @@ -92,6 +94,10 @@ elements are themselves :class:`.ColumnElement` subclasses). :members: :inherited-members: +.. autoclass:: Subquery + :members: + :inherited-members: + .. autoclass:: TableClause :members: :inherited-members: diff --git a/doc/build/core/tutorial.rst b/doc/build/core/tutorial.rst index 0d98f34191..6e389518d0 100644 --- a/doc/build/core/tutorial.rst +++ b/doc/build/core/tutorial.rst @@ -1097,9 +1097,15 @@ FROM clause multiple times. In the case of a SELECT statement, it provides a parent name for the columns represented by the statement, allowing them to be referenced relative to this name. -In SQLAlchemy, any :class:`.Table`, :func:`.select` construct, or other -selectable can be turned into an alias or named subquery using the -:meth:`.FromClause.alias` method, which produces a :class:`.Alias` construct. +In SQLAlchemy, any :class:`.Table` or other :class:`.FromClause` based +selectable can be turned into an alias using :meth:`.FromClause.alias` method, +which produces an :class:`.Alias` construct. :class:`.Alias` is a +:class:`.FromClause` object that refers to a mapping of :class:`.Column` +objects via its :attr:`.FromClause.c` collection, and can be used within the +FROM clause of any subsequent SELECT statement, by referring to its column +elements in the columns or WHERE clause of the statement, or through explicit +placement in the FROM clause, either directly or within a join. + As an example, suppose we know that our user ``jack`` has two particular email addresses. How can we locate jack based on the combination of those two addresses? To accomplish this, we'd use a join to the ``addresses`` table, @@ -1142,17 +1148,21 @@ to the :meth:`.FromClause.alias` method:: >>> a1 = addresses.alias('a1') -Aliases can of course be used for anything which you can SELECT from, -including SELECT statements themselves, by converting the SELECT statement -into a named subquery. The :meth:`.SelectBase.alias` method performs this -role. We can self-join the ``users`` table -back to the :func:`.select` we've created by making an alias of the entire -statement: +SELECT-oriented constructs which extend from :class:`.SelectBase` may be turned +into aliased subqueries using the :meth:`.SelectBase.subquery` method, which +produces a :class:`.Subquery` construct; for ease of use, there is also a +:meth:`.SelectBase.alias` method that is synonymous with +:class:`.SelectBase.subquery`. Like :class:`.Alias`, :class:`.Subquery` is +also a :class:`.FromClause` object that may be part of any enclosing SELECT +using the same techniques one would use for a :class:`.Alias`. + +We can self-join the ``users`` table back to the :func:`.select` we've created +by making :class:`.Subquery` of the entire statement: .. sourcecode:: pycon+sql - >>> addresses_subq = s.alias() - >>> s = select([users.c.name]).where(users.c.id == addresses_subq.c.id) + >>> address_subq = s.subquery() + >>> s = select([users.c.name]).where(users.c.id == address_subq.c.id) {sql}>>> conn.execute(s).fetchall() SELECT users.name FROM users, @@ -1165,6 +1175,10 @@ statement: ('jack@msn.com', 'jack@yahoo.com') {stop}[(u'jack',)] +.. versionchanged:: 1.4 Added the :class:`.Subquery` object and created more of a + separation between an "alias" of a FROM clause and a named subquery of a + SELECT. See :ref:`change_4617`. + Using Joins =========== @@ -1571,7 +1585,7 @@ module level functions :func:`~.expression.union` and UNION SELECT addresses.id, addresses.user_id, addresses.email_address FROM addresses - WHERE addresses.email_address LIKE ? ORDER BY addresses.email_address + WHERE addresses.email_address LIKE ? ORDER BY email_address ('foo@bar.com', '%@yahoo.com') {stop}[(1, 1, u'jack@yahoo.com')] @@ -1604,7 +1618,7 @@ Also available, though not supported on all databases, are A common issue with so-called "compound" selectables arises due to the fact that they nest with parenthesis. SQLite in particular doesn't like a statement that starts with parenthesis. So when nesting a "compound" inside a -"compound", it's often necessary to apply ``.alias().select()`` to the first +"compound", it's often necessary to apply ``.subquery().select()`` to the first element of the outermost compound, if that element is also a compound. For example, to nest a "union" and a "select" inside of "except\_", SQLite will want the "union" to be stated as a subquery: @@ -1617,7 +1631,7 @@ want the "union" to be stated as a subquery: ... where(addresses.c.email_address.like('%@yahoo.com')), ... addresses.select(). ... where(addresses.c.email_address.like('%@msn.com')) - ... ).alias().select(), # apply subquery here + ... ).subquery().select(), # apply subquery here ... addresses.select(addresses.c.email_address.like('%@msn.com')) ... ) {sql}>>> conn.execute(u).fetchall() @@ -1653,6 +1667,36 @@ want the "union" to be stated as a subquery: :func:`.except_all` +Ordering Unions +^^^^^^^^^^^^^^^ + +UNION and other set constructs have a special case when it comes to ordering +the results. As the UNION consists of several SELECT statements, to ORDER the +whole result usually requires that an ORDER BY clause refer to column names but +not specific tables. As in the previous examples, we used +``.order_by(addresses.c.email_address)`` but SQLAlchemy rendered the ORDER BY +without using the table name. A generalized way to apply ORDER BY to a union +is also to refer to the :attr:`.CompoundSelect.selected_columns` collection in +order to access the column expressions which are synonymous with the columns +selected from the first SELECT; the SQLAlchemy compiler will ensure these will +be rendered without table names:: + + >>> u = union( + ... addresses.select(). + ... where(addresses.c.email_address == 'foo@bar.com'), + ... addresses.select(). + ... where(addresses.c.email_address.like('%@yahoo.com')), + ... ) + >>> u = u.order_by(u.selected_columns.email_address) + >>> print(u) + SELECT addresses.id, addresses.user_id, addresses.email_address + FROM addresses + WHERE addresses.email_address = :email_address_1 + UNION SELECT addresses.id, addresses.user_id, addresses.email_address + FROM addresses + WHERE addresses.email_address LIKE :email_address_2 ORDER BY email_address + + .. _scalar_selects: Scalar Selects diff --git a/doc/build/errors.rst b/doc/build/errors.rst index fa2da99d02..38f30d0558 100644 --- a/doc/build/errors.rst +++ b/doc/build/errors.rst @@ -474,7 +474,6 @@ The solution is to access the :class:`.Column` directly using the CheckConstraint(cprop.expression > 5), ) - .. _error_2afi: This Compiled object is not bound to any Engine or Connection @@ -586,6 +585,62 @@ Since "b" is required, pass it as ``None`` so that the INSERT may proceed:: :ref:`execute_multiple` +.. _error_89ve: + +Expected FROM clause, got Select. To create a FROM clause, use the .subquery() method +-------------------------------------------------------------------------------------- + +This refers to a change made as of SQLAlchemy 1.4 where a SELECT statement as generated +by a function such as :func:`.select`, but also including things like unions and textual +SELECT expressions are no longer considered to be :class:`.FromClause` objects and +can't be placed directly in the FROM clause of another SELECT statement without them +being wrapped in a :class:`.Subquery` first. This is a major conceptual change in the +Core and the full rationale is discussed at :ref:`change_4617`. + +Given an example as:: + + m = MetaData() + t = Table( + 't', m, + Column('a', Integer), + Column('b', Integer), + Column('c', Integer) + ) + stmt = select([t]) + +Above, ``stmt`` represents a SELECT statement. The error is produced when we want +to use ``stmt`` directly as a FROM clause in another SELECT, such as if we +attempted to select from it:: + + new_stmt_1 = select([stmt]) + +Or if we wanted to use it in a FROM clause such as in a JOIN:: + + new_stmt_2 = select([some_table]).select_from(some_table.join(stmt)) + +In previous versions of SQLAlchemy, using a SELECT inside of another SELECT +would produce a parenthesized, unnamed subquery. In most cases, this form of +SQL is not very useful as databases like MySQL and PostgreSQL require that +subqueries in FROM clauses have named aliases, which means using the +:meth:`.SelectBase.alias` method or as of 1.4 using the +:meth:`.SelectBase.subquery` method to produce this. On other databases, it +is still much clearer for the subquery to have a name to resolve any ambiguity +on future references to column names inside the subquery. + +Beyond the above practical reasons, there are a lot of other SQLAlchemy-oriented +reasons the change is being made. The correct form of the above two statements +therefore requires that :meth:`.SelectBase.subquery` is used:: + + subq = stmt.subquery() + + new_stmt_1 = select([subq]) + + new_stmt_2 = select([some_table]).select_from(some_table.join(subq)) + +.. seealso:: + + :ref:`change_4617` + Object Relational Mapping ========================= diff --git a/lib/sqlalchemy/dialects/mssql/base.py b/lib/sqlalchemy/dialects/mssql/base.py index b2bcc280c5..86116e6a17 100644 --- a/lib/sqlalchemy/dialects/mssql/base.py +++ b/lib/sqlalchemy/dialects/mssql/base.py @@ -1760,7 +1760,7 @@ class MSSQLCompiler(compiler.SQLCompiler): select, column, asfrom ) - def for_update_clause(self, select): + def for_update_clause(self, select, **kw): # "FOR UPDATE" is only allowed on "DECLARE CURSOR" which # SQLAlchemy doesn't use return "" diff --git a/lib/sqlalchemy/dialects/oracle/base.py b/lib/sqlalchemy/dialects/oracle/base.py index 33f9c86594..842730c5bf 100644 --- a/lib/sqlalchemy/dialects/oracle/base.py +++ b/lib/sqlalchemy/dialects/oracle/base.py @@ -886,7 +886,8 @@ class OracleCompiler(compiler.SQLCompiler): [ c for c in inner_subquery.c - if orig_select.corresponding_column(c) is not None + if orig_select.selected_columns.corresponding_column(c) + is not None ] ) if ( @@ -939,17 +940,22 @@ class OracleCompiler(compiler.SQLCompiler): limitselect._is_wrapper = True if for_update is not None and for_update.of: - + limitselect_cols = limitselect.selected_columns for elem in for_update.of: - if limitselect.corresponding_column(elem) is None: + if ( + limitselect_cols.corresponding_column(elem) + is None + ): limitselect = limitselect.column(elem) limit_subquery = limitselect.alias() + origselect_cols = orig_select.selected_columns offsetselect = sql.select( [ c for c in limit_subquery.c - if orig_select.corresponding_column(c) is not None + if origselect_cols.corresponding_column(c) + is not None ] ) diff --git a/lib/sqlalchemy/engine/result.py b/lib/sqlalchemy/engine/result.py index 740565f5e1..480d086ff3 100644 --- a/lib/sqlalchemy/engine/result.py +++ b/lib/sqlalchemy/engine/result.py @@ -300,7 +300,7 @@ class ResultMetaData(object): # if we did a pure positional match, then reset the # original "expression element" back to the "unambiguous" # entry. This is a new behavior in 1.1 which impacts - # TextAsFrom but also straight compiled SQL constructs. + # TextualSelect but also straight compiled SQL constructs. if not self.matched_on_name: self._keymap.update( [ @@ -363,7 +363,7 @@ class ResultMetaData(object): The remaining fairly common case is that of the textual SQL that includes at least partial column information; this is when - we use a :class:`.TextAsFrom` construct. This construct may have + we use a :class:`.TextualSelect` construct. This construct may have unordered or ordered column information. In the ordered case, we merge the cursor.description and the compiled construct's information positionally, and warn if there are additional description names @@ -386,7 +386,7 @@ class ResultMetaData(object): SQLAlchemy for all cases up through te 0.9 series. Positional matching for compiled SQL expressions was introduced in 1.0 as a major performance feature, and positional matching for textual - :class:`.TextAsFrom` objects in 1.1. As name matching is no longer + :class:`.TextualSelect` objects in 1.1. As name matching is no longer a common case, it was acceptable to factor it into smaller generator- oriented methods that are easier to understand, but incur slightly more performance overhead. diff --git a/lib/sqlalchemy/orm/query.py b/lib/sqlalchemy/orm/query.py index c73a8147cf..13402e7f42 100644 --- a/lib/sqlalchemy/orm/query.py +++ b/lib/sqlalchemy/orm/query.py @@ -253,11 +253,6 @@ class Query(object): "expected when the base alias is being set." ) fa.append(info.selectable) - elif not info.is_clause_element or not info._is_from_clause: - raise sa_exc.ArgumentError( - "argument is not a mapped class, mapper, " - "aliased(), or FromClause instance." - ) else: from_obj = coercions.expect( roles.StrictFromClauseRole, from_obj, allow_select=True @@ -271,7 +266,9 @@ class Query(object): if ( set_base_alias and len(self._from_obj) == 1 - and isinstance(select_from_alias, expression.Alias) + and isinstance( + select_from_alias, sql.selectable.AliasedReturnsRows + ) ): equivs = self.__all_equivs() self._from_obj_alias = sql_util.ColumnAdapter( @@ -2302,7 +2299,14 @@ class Query(object): if ( len(keys) == 2 and isinstance( - keys[0], (expression.FromClause, type, AliasedClass) + keys[0], + ( + # note this would be FromClause once + # coercion of SELECT is removed + expression.Selectable, + type, + AliasedClass, + ), ) and isinstance( keys[1], @@ -2761,7 +2765,9 @@ class Query(object): # if the destination selectable is a plain select(), # turn it into an alias(). if isinstance(right_selectable, expression.SelectBase): - right_selectable = right_selectable.alias() + right_selectable = coercions.expect( + roles.FromClauseRole, right_selectable + ) need_adapter = True # make the right hand side target into an ORM entity @@ -2781,7 +2787,8 @@ class Query(object): and ( right_mapper.with_polymorphic and isinstance( - right_mapper._with_polymorphic_selectable, expression.Alias + right_mapper._with_polymorphic_selectable, + expression.AliasedReturnsRows, ) or overlap # test for overlap: @@ -3201,13 +3208,10 @@ class Query(object): """ statement = coercions.expect(roles.SelectStatementRole, statement) - if not isinstance( + # TODO: coercions above should have this handled + assert isinstance( statement, (expression.TextClause, expression.SelectBase) - ): - raise sa_exc.ArgumentError( - "from_statement accepts text(), select(), " - "and union() objects only." - ) + ) self._statement = statement diff --git a/lib/sqlalchemy/orm/strategies.py b/lib/sqlalchemy/orm/strategies.py index b0dffe5ddf..7e150414b7 100644 --- a/lib/sqlalchemy/orm/strategies.py +++ b/lib/sqlalchemy/orm/strategies.py @@ -1629,7 +1629,6 @@ class JoinedLoader(AbstractRelationshipLoader): # the object becomes shared among threads. this prevents # races for column identities. inspect(to_adapt).selectable.c - self._aliased_class_pool.append(to_adapt) return self._aliased_class_pool[idx] diff --git a/lib/sqlalchemy/orm/util.py b/lib/sqlalchemy/orm/util.py index 80d4b26b20..4b4fa40523 100644 --- a/lib/sqlalchemy/orm/util.py +++ b/lib/sqlalchemy/orm/util.py @@ -825,7 +825,9 @@ def aliased(element, alias=None, name=None, flat=False, adapt_on_names=False): raise sa_exc.ArgumentError( "adapt_on_names only applies to ORM elements" ) - return element._anonymous_fromclause(name=name, flat=flat) + return coercions.expect( + roles.AnonymizedFromClauseRole, element, name=name, flat=flat + ) else: return AliasedClass( element, diff --git a/lib/sqlalchemy/sql/__init__.py b/lib/sqlalchemy/sql/__init__.py index 00cafd8ff8..bf5468aa56 100644 --- a/lib/sqlalchemy/sql/__init__.py +++ b/lib/sqlalchemy/sql/__init__.py @@ -58,6 +58,7 @@ from .expression import quoted_name # noqa from .expression import Select # noqa from .expression import select # noqa from .expression import Selectable # noqa +from .expression import Subquery # noqa from .expression import subquery # noqa from .expression import table # noqa from .expression import TableClause # noqa diff --git a/lib/sqlalchemy/sql/base.py b/lib/sqlalchemy/sql/base.py index 9df0c932f9..a84843c4bc 100644 --- a/lib/sqlalchemy/sql/base.py +++ b/lib/sqlalchemy/sql/base.py @@ -11,6 +11,7 @@ import itertools +import operator import re from .visitors import ClauseVisitor @@ -51,6 +52,38 @@ def _generative(fn, *args, **kw): return self +def _clone(element, **kw): + return element._clone() + + +def _expand_cloned(elements): + """expand the given set of ClauseElements to be the set of all 'cloned' + predecessors. + + """ + return itertools.chain(*[x._cloned_set for x in elements]) + + +def _cloned_intersection(a, b): + """return the intersection of sets a and b, counting + any overlap between 'cloned' predecessors. + + The returned set is in terms of the entities present within 'a'. + + """ + all_overlap = set(_expand_cloned(a)).intersection(_expand_cloned(b)) + return set( + elem for elem in a if all_overlap.intersection(elem._cloned_set) + ) + + +def _cloned_difference(a, b): + all_overlap = set(_expand_cloned(a)).intersection(_expand_cloned(b)) + return set( + elem for elem in a if not all_overlap.intersection(elem._cloned_set) + ) + + class _DialectArgView(util.collections_abc.MutableMapping): """A dictionary view of dialect-level arguments in the form _. @@ -486,6 +519,97 @@ class ColumnCollection(util.OrderedProperties): def __str__(self): return repr([str(c) for c in self]) + def corresponding_column(self, column, require_embedded=False): + """Given a :class:`.ColumnElement`, return the exported + :class:`.ColumnElement` object from this :class:`.ColumnCollection` + which corresponds to that original :class:`.ColumnElement` via a common + ancestor column. + + :param column: the target :class:`.ColumnElement` to be matched + + :param require_embedded: only return corresponding columns for + the given :class:`.ColumnElement`, if the given + :class:`.ColumnElement` is actually present within a sub-element + of this :class:`.Selectable`. Normally the column will match if + it merely shares a common ancestor with one of the exported + columns of this :class:`.Selectable`. + + .. seealso:: + + :meth:`.Selectable.corresponding_column` - invokes this method + against the collection returned by + :attr:`.Selectable.exported_columns`. + + .. versionchanged:: 1.4 the implementation for ``corresponding_column`` + was moved onto the :class:`.ColumnCollection` itself. + + """ + + def embedded(expanded_proxy_set, target_set): + for t in target_set.difference(expanded_proxy_set): + if not set(_expand_cloned([t])).intersection( + expanded_proxy_set + ): + return False + return True + + # don't dig around if the column is locally present + if self.contains_column(column): + return column + col, intersect = None, None + target_set = column.proxy_set + cols = self._all_columns + for c in cols: + expanded_proxy_set = set(_expand_cloned(c.proxy_set)) + i = target_set.intersection(expanded_proxy_set) + if i and ( + not require_embedded + or embedded(expanded_proxy_set, target_set) + ): + if col is None: + + # no corresponding column yet, pick this one. + + col, intersect = c, i + elif len(i) > len(intersect): + + # 'c' has a larger field of correspondence than + # 'col'. i.e. selectable.c.a1_x->a1.c.x->table.c.x + # matches a1.c.x->table.c.x better than + # selectable.c.x->table.c.x does. + + col, intersect = c, i + elif i == intersect: + # they have the same field of correspondence. see + # which proxy_set has fewer columns in it, which + # indicates a closer relationship with the root + # column. Also take into account the "weight" + # attribute which CompoundSelect() uses to give + # higher precedence to columns based on vertical + # position in the compound statement, and discard + # columns that have no reference to the target + # column (also occurs with CompoundSelect) + + col_distance = util.reduce( + operator.add, + [ + sc._annotations.get("weight", 1) + for sc in col._uncached_proxy_set() + if sc.shares_lineage(column) + ], + ) + c_distance = util.reduce( + operator.add, + [ + sc._annotations.get("weight", 1) + for sc in c._uncached_proxy_set() + if sc.shares_lineage(column) + ], + ) + if c_distance < col_distance: + col, intersect = c, i + return col + def replace(self, column): """add the given column to this collection, removing unaliased versions of this column as well as existing columns with the @@ -619,6 +743,26 @@ class ColumnCollection(util.OrderedProperties): return ImmutableColumnCollection(self._data, self._all_columns) +class SeparateKeyColumnCollection(ColumnCollection): + """Column collection that maintains a string name separate from the + column itself""" + + def __init__(self, cols_plus_names=None): + super(ColumnCollection, self).__init__() + object.__setattr__(self, "_all_columns", []) + if cols_plus_names: + self.update(cols_plus_names) + + def replace(self, column): + raise NotImplementedError() + + def add(self, column): + raise NotImplementedError() + + def remove(self, column): + raise NotImplementedError() + + class ImmutableColumnCollection(util.ImmutableProperties, ColumnCollection): def __init__(self, data, all_columns): util.ImmutableProperties.__init__(self, data) diff --git a/lib/sqlalchemy/sql/clause_compare.py b/lib/sqlalchemy/sql/clause_compare.py index 0ea981f1e2..50b1df99ed 100644 --- a/lib/sqlalchemy/sql/clause_compare.py +++ b/lib/sqlalchemy/sql/clause_compare.py @@ -291,7 +291,7 @@ class StructureComparatorStrategy(object): return True - def compare_text_as_from(self, left, right, **kw): + def compare_textual_select(self, left, right, **kw): self.compare_stack.extendleft( util.zip_longest(left.column_args, right.column_args) ) diff --git a/lib/sqlalchemy/sql/coercions.py b/lib/sqlalchemy/sql/coercions.py index 39e6628e42..64d9f0f968 100644 --- a/lib/sqlalchemy/sql/coercions.py +++ b/lib/sqlalchemy/sql/coercions.py @@ -124,18 +124,24 @@ class RoleImpl(object): return element def _implicit_coercions(self, element, resolved, argname=None, **kw): - self._raise_for_expected(element, argname) + self._raise_for_expected(element, argname, resolved) - def _raise_for_expected(self, element, argname=None): + def _raise_for_expected( + self, element, argname=None, resolved=None, advice=None, code=None + ): if argname: - raise exc.ArgumentError( - "%s expected for argument %r; got %r." - % (self.name, argname, element) + msg = "%s expected for argument %r; got %r." % ( + self.name, + argname, + element, ) else: - raise exc.ArgumentError( - "%s expected, got %r." % (self.name, element) - ) + msg = "%s expected, got %r." % (self.name, element) + + if advice: + msg += " " + advice + + raise exc.ArgumentError(msg, code=code) class _StringOnly(object): @@ -150,7 +156,7 @@ class _ReturnsStringKey(object): if isinstance(original_element, util.string_types): return original_element else: - self._raise_for_expected(original_element, argname) + self._raise_for_expected(original_element, argname, resolved) def _literal_coercion(self, element, **kw): return element @@ -172,15 +178,13 @@ class _ColumnCoercions(object): if resolved._is_select_statement: self._warn_for_scalar_subquery_coercion() return resolved.scalar_subquery() - elif ( - resolved._is_from_clause - and isinstance(resolved, selectable.Alias) - and resolved.element._is_select_statement + elif resolved._is_from_clause and isinstance( + resolved, selectable.Subquery ): self._warn_for_scalar_subquery_coercion() return resolved.element.scalar_subquery() else: - self._raise_for_expected(original_element, argname) + self._raise_for_expected(original_element, argname, resolved) def _no_text_coercion( @@ -236,6 +240,30 @@ class _CoerceLiterals(object): self._raise_for_expected(element, argname) +class _SelectIsNotFrom(object): + def _raise_for_expected(self, element, argname=None, resolved=None, **kw): + if isinstance(element, roles.SelectStatementRole) or isinstance( + resolved, roles.SelectStatementRole + ): + advice = ( + "To create a " + "FROM clause from a %s object, use the .subquery() method." + % (element.__class__) + ) + code = "89ve" + else: + advice = code = None + + return super(_SelectIsNotFrom, self)._raise_for_expected( + element, + argname=argname, + resolved=resolved, + advice=advice, + code=code, + **kw + ) + + class ExpressionElementImpl( _ColumnCoercions, RoleImpl, roles.ExpressionElementRole ): @@ -287,7 +315,7 @@ class InElementImpl(RoleImpl, roles.InElementRole): else: return resolved.select() else: - self._raise_for_expected(original_element, argname) + self._raise_for_expected(original_element, argname, resolved) def _literal_coercion(self, element, expr, operator, **kw): if isinstance(element, collections_abc.Iterable) and not isinstance( @@ -412,7 +440,7 @@ class TruncatedLabelImpl(_StringOnly, RoleImpl, roles.TruncatedLabelRole): if isinstance(original_element, util.string_types): return resolved else: - self._raise_for_expected(original_element, argname) + self._raise_for_expected(original_element, argname, resolved) def _literal_coercion(self, element, argname=None, **kw): """coerce the given value to :class:`._truncated_label`. @@ -447,7 +475,7 @@ class LimitOffsetImpl(RoleImpl, roles.LimitOffsetRole): if resolved is None: return None else: - self._raise_for_expected(element, argname) + self._raise_for_expected(element, argname, resolved) def _literal_coercion(self, element, name, type_, **kw): if element is None: @@ -474,10 +502,12 @@ class LabeledColumnExprImpl( if isinstance(new, roles.ExpressionElementRole): return new.label(None) else: - self._raise_for_expected(original_element, argname) + self._raise_for_expected(original_element, argname, resolved) -class ColumnsClauseImpl(_CoerceLiterals, RoleImpl, roles.ColumnsClauseRole): +class ColumnsClauseImpl( + _SelectIsNotFrom, _CoerceLiterals, RoleImpl, roles.ColumnsClauseRole +): _coerce_consts = True _coerce_numerics = True @@ -526,21 +556,40 @@ class SelectStatementImpl( if resolved._is_text_clause: return resolved.columns() else: - self._raise_for_expected(original_element, argname) + self._raise_for_expected(original_element, argname, resolved) class HasCTEImpl(ReturnsRowsImpl, roles.HasCTERole): pass -class FromClauseImpl(_NoTextCoercion, RoleImpl, roles.FromClauseRole): +class FromClauseImpl( + _SelectIsNotFrom, _NoTextCoercion, RoleImpl, roles.FromClauseRole +): def _implicit_coercions( - self, original_element, resolved, argname=None, **kw + self, + original_element, + resolved, + argname=None, + explicit_subquery=False, + allow_select=True, + **kw ): - if resolved._is_text_clause: + if resolved._is_select_statement: + if explicit_subquery: + return resolved.subquery() + elif allow_select: + util.warn_deprecated( + "Implicit coercion of SELECT and textual SELECT " + "constructs into FROM clauses is deprecated; please call " + ".subquery() on any Core select or ORM Query object in " + "order to produce a subquery object." + ) + return resolved._implicit_subquery + elif resolved._is_text_clause: return resolved else: - self._raise_for_expected(original_element, argname) + self._raise_for_expected(original_element, argname, resolved) class StrictFromClauseImpl(FromClauseImpl, roles.StrictFromClauseRole): @@ -559,16 +608,16 @@ class StrictFromClauseImpl(FromClauseImpl, roles.StrictFromClauseRole): "on any Core select or ORM Query object in order to produce a " "subquery object." ) - return resolved.subquery() + return resolved._implicit_subquery else: - self._raise_for_expected(original_element, argname) + self._raise_for_expected(original_element, argname, resolved) class AnonymizedFromClauseImpl( StrictFromClauseImpl, roles.AnonymizedFromClauseRole ): - def _post_coercion(self, element, flat=False, **kw): - return element.alias(flat=flat) + def _post_coercion(self, element, flat=False, name=None, **kw): + return element.alias(name=name, flat=flat) class DMLSelectImpl(_NoTextCoercion, RoleImpl, roles.DMLSelectRole): @@ -584,17 +633,28 @@ class DMLSelectImpl(_NoTextCoercion, RoleImpl, roles.DMLSelectRole): else: return resolved.select() else: - self._raise_for_expected(original_element, argname) + self._raise_for_expected(original_element, argname, resolved) class CompoundElementImpl( _NoTextCoercion, RoleImpl, roles.CompoundElementRole ): - def _implicit_coercions(self, original_element, resolved, argname=None): - if resolved._is_from_clause: - return resolved + def _raise_for_expected(self, element, argname=None, resolved=None, **kw): + if isinstance(element, roles.FromClauseRole): + if element._is_subquery: + advice = ( + "Use the plain select() object without " + "calling .subquery() or .alias()." + ) + else: + advice = ( + "To SELECT from any FROM clause, use the .select() method." + ) else: - self._raise_for_expected(original_element, argname) + advice = None + return super(CompoundElementImpl, self)._raise_for_expected( + element, argname=argname, resolved=resolved, advice=advice, **kw + ) _impl_lookup = {} diff --git a/lib/sqlalchemy/sql/compiler.py b/lib/sqlalchemy/sql/compiler.py index 7922054f88..13219ee687 100644 --- a/lib/sqlalchemy/sql/compiler.py +++ b/lib/sqlalchemy/sql/compiler.py @@ -488,7 +488,7 @@ class SQLCompiler(Compiled): """ if False, means we can't be sure the list of entries in _result_columns is actually the rendered order. Usually - True unless using an unordered TextAsFrom. + True unless using an unordered TextualSelect. """ _numeric_binds = False @@ -916,8 +916,8 @@ class SQLCompiler(Compiled): ), ) - def visit_text_as_from( - self, taf, compound_index=None, asfrom=False, parens=True, **kw + def visit_textual_select( + self, taf, compound_index=None, asfrom=False, **kw ): toplevel = not self.stack @@ -943,10 +943,7 @@ class SQLCompiler(Compiled): add_to_result_map=self._add_to_result_map, ) - text = self.process(taf.element, **kw) - if asfrom and parens: - text = "(%s)" % text - return text + return self.process(taf.element, **kw) def visit_null(self, expr, **kw): return "NULL" @@ -1120,7 +1117,7 @@ class SQLCompiler(Compiled): return func.clause_expr._compiler_dispatch(self, **kwargs) def visit_compound_select( - self, cs, asfrom=False, parens=True, compound_index=0, **kwargs + self, cs, asfrom=False, compound_index=0, **kwargs ): toplevel = not self.stack entry = self._default_stack_entry if toplevel else self.stack[-1] @@ -1143,16 +1140,13 @@ class SQLCompiler(Compiled): text = (" " + keyword + " ").join( ( c._compiler_dispatch( - self, - asfrom=asfrom, - parens=False, - compound_index=i, - **kwargs + self, asfrom=asfrom, compound_index=i, **kwargs ) for i, c in enumerate(cs.selects) ) ) + kwargs["include_table"] = False text += self.group_by_clause(cs, **dict(asfrom=asfrom, **kwargs)) text += self.order_by_clause(cs, **kwargs) text += ( @@ -1165,10 +1159,7 @@ class SQLCompiler(Compiled): text = self._render_cte_clause() + text self.stack.pop(-1) - if asfrom and parens: - return "(" + text + ")" - else: - return text + return text def _get_operator_dispatch(self, operator_, qualifier1, qualifier2): attrname = "visit_%s_%s%s" % ( @@ -1682,8 +1673,11 @@ class SQLCompiler(Compiled): if self.positional: kwargs["positional_names"] = self.cte_positional[cte] = [] - text += " AS \n" + cte.element._compiler_dispatch( - self, asfrom=True, **kwargs + assert kwargs.get("subquery", False) is False + text += " AS \n(%s)" % ( + cte.element._compiler_dispatch( + self, asfrom=True, **kwargs + ), ) if cte._suffixes: @@ -1713,8 +1707,28 @@ class SQLCompiler(Compiled): ashint=False, iscrud=False, fromhints=None, + subquery=False, + lateral=False, + enclosing_alias=None, **kwargs ): + if enclosing_alias is not None and enclosing_alias.element is alias: + inner = alias.element._compiler_dispatch( + self, + asfrom=asfrom, + ashint=ashint, + iscrud=iscrud, + fromhints=fromhints, + lateral=lateral, + enclosing_alias=alias, + **kwargs + ) + if subquery and (asfrom or lateral): + inner = "(%s)" % (inner,) + return inner + else: + enclosing_alias = kwargs["enclosing_alias"] = alias + if asfrom or ashint: if isinstance(alias.name, elements._truncated_label): alias_name = self._truncated_identifier("alias", alias.name) @@ -1724,12 +1738,15 @@ class SQLCompiler(Compiled): if ashint: return self.preparer.format_alias(alias, alias_name) elif asfrom: - ret = alias.element._compiler_dispatch( - self, asfrom=True, **kwargs - ) + self.get_render_as_alias_suffix( - self.preparer.format_alias(alias, alias_name) + inner = alias.element._compiler_dispatch( + self, asfrom=True, lateral=lateral, **kwargs ) + if subquery: + inner = "(%s)" % (inner,) + ret = inner + self.get_render_as_alias_suffix( + self.preparer.format_alias(alias, alias_name) + ) if fromhints and alias in fromhints: ret = self.format_from_hint_text( ret, alias, fromhints[alias], iscrud @@ -1737,7 +1754,14 @@ class SQLCompiler(Compiled): return ret else: - return alias.element._compiler_dispatch(self, **kwargs) + # note we cancel the "subquery" flag here as well + return alias.element._compiler_dispatch( + self, lateral=lateral, **kwargs + ) + + def visit_subquery(self, subquery, **kw): + kw["subquery"] = True + return self.visit_alias(subquery, **kw) def visit_lateral(self, lateral, **kw): kw["lateral"] = True @@ -2004,7 +2028,6 @@ class SQLCompiler(Compiled): self, select, asfrom=False, - parens=True, fromhints=None, compound_index=0, nested_join_translation=False, @@ -2027,7 +2050,6 @@ class SQLCompiler(Compiled): text = self.visit_select( transformed_select, asfrom=asfrom, - parens=parens, fromhints=fromhints, compound_index=compound_index, nested_join_translation=True, @@ -2138,10 +2160,7 @@ class SQLCompiler(Compiled): self.stack.pop(-1) - if (asfrom or lateral) and parens: - return "(" + text + ")" - else: - return text + return text def _setup_select_hints(self, select): byfrom = dict( @@ -2371,7 +2390,7 @@ class SQLCompiler(Compiled): ) return dialect_hints, table_text - def visit_insert(self, insert_stmt, asfrom=False, **kw): + def visit_insert(self, insert_stmt, **kw): toplevel = not self.stack self.stack.append( @@ -2475,10 +2494,7 @@ class SQLCompiler(Compiled): self.stack.pop(-1) - if asfrom: - return "(" + text + ")" - else: - return text + return text def update_limit_clause(self, update_stmt): """Provide a hook for MySQL to add LIMIT to the UPDATE""" @@ -2508,7 +2524,7 @@ class SQLCompiler(Compiled): "criteria within UPDATE" ) - def visit_update(self, update_stmt, asfrom=False, **kw): + def visit_update(self, update_stmt, **kw): toplevel = not self.stack extra_froms = update_stmt._extra_froms @@ -2605,10 +2621,7 @@ class SQLCompiler(Compiled): self.stack.pop(-1) - if asfrom: - return "(" + text + ")" - else: - return text + return text @util.memoized_property def _key_getters_for_crud_column(self): @@ -2633,7 +2646,7 @@ class SQLCompiler(Compiled): def delete_table_clause(self, delete_stmt, from_table, extra_froms): return from_table._compiler_dispatch(self, asfrom=True, iscrud=True) - def visit_delete(self, delete_stmt, asfrom=False, **kw): + def visit_delete(self, delete_stmt, **kw): toplevel = not self.stack crud._setup_crud_params(self, delete_stmt, crud.ISDELETE, **kw) @@ -2702,10 +2715,7 @@ class SQLCompiler(Compiled): self.stack.pop(-1) - if asfrom: - return "(" + text + ")" - else: - return text + return text def visit_savepoint(self, savepoint_stmt): return "SAVEPOINT %s" % self.preparer.format_savepoint(savepoint_stmt) diff --git a/lib/sqlalchemy/sql/dml.py b/lib/sqlalchemy/sql/dml.py index c7d83fc12b..5a9be7c629 100644 --- a/lib/sqlalchemy/sql/dml.py +++ b/lib/sqlalchemy/sql/dml.py @@ -47,6 +47,10 @@ class UpdateBase( _prefixes = () named_with_column = False + def _generate_fromclause_column_proxies(self, fromclause): + for col in self._returning: + col._make_proxy(fromclause) + def _process_colparams(self, parameters): def process_single(p): if isinstance(p, (list, tuple)): diff --git a/lib/sqlalchemy/sql/elements.py b/lib/sqlalchemy/sql/elements.py index b902ef4b4f..5b44422227 100644 --- a/lib/sqlalchemy/sql/elements.py +++ b/lib/sqlalchemy/sql/elements.py @@ -22,6 +22,7 @@ from . import operators from . import roles from . import type_api from .annotation import Annotated +from .base import _clone from .base import _generative from .base import Executable from .base import Immutable @@ -36,10 +37,6 @@ from .. import inspection from .. import util -def _clone(element, **kw): - return element._clone() - - def collate(expression, collation): """Return the clause ``expression COLLATE collation``. @@ -415,6 +412,11 @@ class ClauseElement(roles.SQLRole, Visitable): """ return self + def _ungroup(self): + """Return this :class:`.ClauseElement` without any groupings.""" + + return self + @util.dependencies("sqlalchemy.engine.default") def compile(self, default, bind=None, dialect=None, **kw): """Compile this SQL expression. @@ -821,6 +823,16 @@ class ColumnElement( and other.name == self.name ) + @util.memoized_property + def _proxy_key(self): + if self.key: + return self.key + else: + try: + return str(self) + except exc.UnsupportedCompilationError: + return self.anon_label + def _make_proxy( self, selectable, name=None, name_is_truncatable=False, **kw ): @@ -831,14 +843,7 @@ class ColumnElement( """ if name is None: name = self.anon_label - if self.key: - key = self.key - else: - try: - key = str(self) - except exc.UnsupportedCompilationError: - key = self.anon_label - + key = self._proxy_key else: key = name co = ColumnClause( @@ -1619,8 +1624,14 @@ class TextClause( @util.dependencies("sqlalchemy.sql.selectable") def columns(self, selectable, *cols, **types): - """Turn this :class:`.TextClause` object into a :class:`.TextAsFrom` - object that can be embedded into another statement. + r"""Turn this :class:`.TextClause` object into a + :class:`.TextualSelect` object that serves the same role as a SELECT + statement. + + The :class:`.TextualSelect` is part of the :class:`.SelectBase` + hierarchy and can be embedded into another statement by using the + :meth:`.TextualSelect.subquery` method to produce a :class:`.Subquery` + object, which can then be SELECTed from. This function essentially bridges the gap between an entirely textual SELECT statement and the SQL expression language concept @@ -1629,7 +1640,7 @@ class TextClause( from sqlalchemy.sql import column, text stmt = text("SELECT id, name FROM some_table") - stmt = stmt.columns(column('id'), column('name')).alias('st') + stmt = stmt.columns(column('id'), column('name')).subquery('st') stmt = select([mytable]).\ select_from( @@ -1638,8 +1649,10 @@ class TextClause( Above, we pass a series of :func:`.column` elements to the :meth:`.TextClause.columns` method positionally. These :func:`.column` - elements now become first class elements upon the :attr:`.TextAsFrom.c` - column collection, just like any other selectable. + elements now become first class elements upon the + :attr:`.TextualSelect.selected_columns` column collection, which then + become part of the :attr:`.Subquery.c` collection after + :meth:`.TextualSelect.subquery` is invoked. The column expressions we pass to :meth:`.TextClause.columns` may also be typed; when we do so, these :class:`.TypeEngine` objects become @@ -1697,17 +1710,22 @@ class TextClause( the column expressions are passed purely positionally. The :meth:`.TextClause.columns` method provides a direct - route to calling :meth:`.FromClause.alias` as well as + route to calling :meth:`.FromClause.subquery` as well as :meth:`.SelectBase.cte` against a textual SELECT statement:: stmt = stmt.columns(id=Integer, name=String).cte('st') stmt = select([sometable]).where(sometable.c.id == stmt.c.id) - .. versionadded:: 0.9.0 :func:`.text` can now be converted into a - fully featured "selectable" construct using the - :meth:`.TextClause.columns` method. + :param \*cols: A series of :class:`.ColumnElement` objects, typically + :class:`.Column` objects from a :class:`.Table` or ORM level + column-mapped attributes, representing a set of columns that this + textual string will SELECT from. + :param \**types: A mapping of string names to :class:`.TypeEngine` + type objects indicating the datatypes to use for names that are + SELECTed from the textual string. Prefer to use the ``\*cols`` + argument as it also indicates positional ordering. """ positional_input_cols = [ @@ -1720,7 +1738,7 @@ class TextClause( ColumnClause(key, type_) for key, type_ in types.items() ] - return selectable.TextAsFrom( + return selectable.TextualSelect( self, positional_input_cols + keyed_input_cols, positional=bool(positional_input_cols) and not keyed_input_cols, @@ -3291,19 +3309,26 @@ class IndexExpression(BinaryExpression): pass -class Grouping(ColumnElement): - """Represent a grouping within a column expression""" +class GroupedElement(ClauseElement): + """Represent any parenthesized expression""" __visit_name__ = "grouping" - def __init__(self, element): - self.element = element - self.type = getattr(element, "type", type_api.NULLTYPE) - def self_group(self, against=None): # type: (Optional[Any]) -> ClauseElement return self + def _ungroup(self): + return self.element._ungroup() + + +class Grouping(GroupedElement, ColumnElement): + """Represent a grouping within a column expression""" + + def __init__(self, element): + self.element = element + self.type = getattr(element, "type", type_api.NULLTYPE) + @util.memoized_property def _is_implicitly_boolean(self): return self.element._is_implicitly_boolean @@ -4351,14 +4376,6 @@ class quoted_name(util.MemoizedSlots, util.text_type): return "'%s'" % backslashed -def _expand_cloned(elements): - """expand the given set of ClauseElements to be the set of all 'cloned' - predecessors. - - """ - return itertools.chain(*[x._cloned_set for x in elements]) - - def _select_iterables(elements): """expand tables into individual columns in the given list of column expressions. @@ -4367,26 +4384,6 @@ def _select_iterables(elements): return itertools.chain(*[c._select_iterable for c in elements]) -def _cloned_intersection(a, b): - """return the intersection of sets a and b, counting - any overlap between 'cloned' predecessors. - - The returned set is in terms of the entities present within 'a'. - - """ - all_overlap = set(_expand_cloned(a)).intersection(_expand_cloned(b)) - return set( - elem for elem in a if all_overlap.intersection(elem._cloned_set) - ) - - -def _cloned_difference(a, b): - all_overlap = set(_expand_cloned(a)).intersection(_expand_cloned(b)) - return set( - elem for elem in a if not all_overlap.intersection(elem._cloned_set) - ) - - def _find_columns(clause): """locate Column objects within the given expression.""" diff --git a/lib/sqlalchemy/sql/expression.py b/lib/sqlalchemy/sql/expression.py index b04355cf5d..7ce822669c 100644 --- a/lib/sqlalchemy/sql/expression.py +++ b/lib/sqlalchemy/sql/expression.py @@ -16,6 +16,7 @@ class. __all__ = [ "Alias", + "AliasedReturnsRows", "any_", "all_", "ClauseElement", @@ -76,6 +77,7 @@ __all__ = [ "union_all", "update", "within_group", + "Subquery", "TableSample", "tablesample", ] @@ -132,6 +134,7 @@ from .functions import Function # noqa from .functions import FunctionElement # noqa from .functions import modifier # noqa from .selectable import Alias # noqa +from .selectable import AliasedReturnsRows # noqa from .selectable import CompoundSelect # noqa from .selectable import CTE # noqa from .selectable import Exists # noqa @@ -148,10 +151,12 @@ from .selectable import ScalarSelect # noqa from .selectable import Select # noqa from .selectable import Selectable # noqa from .selectable import SelectBase # noqa +from .selectable import Subquery # noqa from .selectable import subquery # noqa from .selectable import TableClause # noqa from .selectable import TableSample # noqa from .selectable import TextAsFrom # noqa +from .selectable import TextualSelect # noqa from .visitors import Visitable # noqa from ..util.langhelpers import public_factory # noqa diff --git a/lib/sqlalchemy/sql/roles.py b/lib/sqlalchemy/sql/roles.py index 053bd71460..55c52d4014 100644 --- a/lib/sqlalchemy/sql/roles.py +++ b/lib/sqlalchemy/sql/roles.py @@ -95,6 +95,8 @@ class InElementRole(SQLRole): class FromClauseRole(ColumnsClauseRole): _role_name = "FROM expression, such as a Table or alias() object" + _is_subquery = False + @property def _hide_froms(self): raise NotImplementedError() @@ -134,7 +136,7 @@ class StatementRole(CoerceTextStatementRole): class ReturnsRowsRole(StatementRole): _role_name = ( - "Row returning expression such as a SELECT, or an " + "Row returning expression such as a SELECT, a FROM clause, or an " "INSERT/UPDATE/DELETE with RETURNING" ) @@ -142,6 +144,12 @@ class ReturnsRowsRole(StatementRole): class SelectStatementRole(ReturnsRowsRole): _role_name = "SELECT construct or equivalent text() construct" + def subquery(self): + raise NotImplementedError( + "All SelectStatementRole objects should implement a " + ".subquery() method." + ) + class HasCTERole(ReturnsRowsRole): pass diff --git a/lib/sqlalchemy/sql/selectable.py b/lib/sqlalchemy/sql/selectable.py index 591086a467..2263073c49 100644 --- a/lib/sqlalchemy/sql/selectable.py +++ b/lib/sqlalchemy/sql/selectable.py @@ -12,7 +12,6 @@ SQL tables and derived rowsets. import collections import itertools -import operator from operator import attrgetter from . import coercions @@ -20,6 +19,10 @@ from . import operators from . import roles from . import type_api from .annotation import Annotated +from .base import _clone +from .base import _cloned_difference +from .base import _cloned_intersection +from .base import _expand_cloned from .base import _from_objects from .base import _generative from .base import ColumnCollection @@ -27,17 +30,15 @@ from .base import ColumnSet from .base import Executable from .base import Generative from .base import Immutable +from .base import SeparateKeyColumnCollection from .coercions import _document_text_coercion from .elements import _anonymous_label -from .elements import _clone -from .elements import _cloned_difference -from .elements import _cloned_intersection -from .elements import _expand_cloned from .elements import _select_iterables from .elements import and_ from .elements import BindParameter from .elements import ClauseElement from .elements import ClauseList +from .elements import GroupedElement from .elements import Grouping from .elements import literal_column from .elements import True_ @@ -52,20 +53,22 @@ class _OffsetLimitParam(BindParameter): return self.effective_value +@util.deprecated( + "1.4", + "The standalone :func:`.subquery` function is deprecated " + "and will be removed in a future release. Use select().subquery().", +) def subquery(alias, *args, **kwargs): - r"""Return an :class:`.Alias` object derived + r"""Return an :class:`.Subquery` object derived from a :class:`.Select`. - name - alias name + :param name: the alias name for the subquery - \*args, \**kwargs - - all other arguments are delivered to the - :func:`select` function. + :param \*args, \**kwargs: all other arguments are passed through to the + :func:`.select` function. """ - return Select(*args, **kwargs).alias(alias) + return Select(*args, **kwargs).subquery(alias) class ReturnsRows(roles.ReturnsRowsRole, ClauseElement): @@ -90,19 +93,12 @@ class ReturnsRows(roles.ReturnsRowsRole, ClauseElement): @property def selectable(self): - raise NotImplementedError( - "This object is a base ReturnsRows object, but is not a " - "FromClause so has no .c. collection." - ) + raise NotImplementedError() class Selectable(ReturnsRows): """mark a class as being selectable. - This class is legacy as of 1.4 as the concept of a SQL construct which - "returns rows" is more generalized than one which can be the subject - of a SELECT. - """ __visit_name__ = "selectable" @@ -113,6 +109,93 @@ class Selectable(ReturnsRows): def selectable(self): return self + @property + def exported_columns(self): + """A :class:`.ColumnCollection` that represents the "exported" + columns of this :class:`.Selectable`. + + The "exported" columns represent the collection of + :class:`.ColumnElement` expressions that are rendered by this SQL + construct. There are two primary varieties which are the + "FROM clause columns" of a FROM clause, such as a table, join, + or subquery, and the "SELECTed columns", which are the columns in + the "columns clause" of a SELECT statement. + + .. versionadded:: 1.4 + + .. seealso: + + :attr:`.FromClause.exported_columns` + + :attr:`.SelectBase.exported_columns` + """ + + raise NotImplementedError() + + def _refresh_for_new_column(self, column): + raise NotImplementedError() + + def lateral(self, name=None): + """Return a LATERAL alias of this :class:`.Selectable`. + + The return value is the :class:`.Lateral` construct also + provided by the top-level :func:`~.expression.lateral` function. + + .. versionadded:: 1.1 + + .. seealso:: + + :ref:`lateral_selects` - overview of usage. + + """ + return Lateral._construct(self, name) + + @util.deprecated( + "1.4", + message="The :meth:`.Selectable.replace_selectable` method is " + "deprecated, and will be removed in a future release. Similar " + "functionality is available via the sqlalchemy.sql.visitors module.", + ) + @util.dependencies("sqlalchemy.sql.util") + def replace_selectable(self, sqlutil, old, alias): + """replace all occurrences of FromClause 'old' with the given Alias + object, returning a copy of this :class:`.FromClause`. + + """ + + return sqlutil.ClauseAdapter(alias).traverse(self) + + def corresponding_column(self, column, require_embedded=False): + """Given a :class:`.ColumnElement`, return the exported + :class:`.ColumnElement` object from the + :attr:`.Selectable.exported_columns` + collection of this :class:`.Selectable` which corresponds to that + original :class:`.ColumnElement` via a common ancestor + column. + + :param column: the target :class:`.ColumnElement` to be matched + + :param require_embedded: only return corresponding columns for + the given :class:`.ColumnElement`, if the given + :class:`.ColumnElement` is actually present within a sub-element + of this :class:`.Selectable`. Normally the column will match if + it merely shares a common ancestor with one of the exported + columns of this :class:`.Selectable`. + + .. seealso:: + + :attr:`.Selectable.exported_columns` - the + :class:`.ColumnCollection` that is used for the operation + + :meth:`.ColumnCollection.corresponding_column` - implementation + method. + + """ + + return self.exported_columns.corresponding_column( + column, require_embedded + ) + class HasPrefixes(object): _prefixes = () @@ -211,7 +294,7 @@ class HasSuffixes(object): ) -class FromClause(roles.FromClauseRole, Selectable): +class FromClause(roles.AnonymizedFromClauseRole, Selectable): """Represent an element that can be used within the ``FROM`` clause of a ``SELECT`` statement. @@ -243,9 +326,8 @@ class FromClause(roles.FromClauseRole, Selectable): """ - is_selectable = has_selectable = True + is_selectable = True _is_from_clause = True - _is_text_as_from = False _is_join = False def _translate_schema(self, effective_schema, map_): @@ -400,21 +482,6 @@ class FromClause(roles.FromClauseRole, Selectable): return Alias._construct(self, name) - def lateral(self, name=None): - """Return a LATERAL alias of this :class:`.FromClause`. - - The return value is the :class:`.Lateral` construct also - provided by the top-level :func:`~.expression.lateral` function. - - .. versionadded:: 1.1 - - .. seealso:: - - :ref:`lateral_selects` - overview of usage. - - """ - return Lateral._construct(self, name) - def tablesample(self, sampling, name=None, seed=None): """Return a TABLESAMPLE alias of this :class:`.FromClause`. @@ -452,125 +519,43 @@ class FromClause(roles.FromClauseRole, Selectable): """ return self._cloned_set.intersection(other._cloned_set) - @util.dependencies("sqlalchemy.sql.util") - def replace_selectable(self, sqlutil, old, alias): - """replace all occurrences of FromClause 'old' with the given Alias - object, returning a copy of this :class:`.FromClause`. - - """ - - return sqlutil.ClauseAdapter(alias).traverse(self) + @property + def description(self): + """a brief description of this FromClause. - def correspond_on_equivalents(self, column, equivalents): - """Return corresponding_column for the given column, or if None - search for a match in the given dictionary. + Used primarily for error message formatting. """ - col = self.corresponding_column(column, require_embedded=True) - if col is None and col in equivalents: - for equiv in equivalents[col]: - nc = self.corresponding_column(equiv, require_embedded=True) - if nc: - return nc - return col + return getattr(self, "name", self.__class__.__name__ + " object") - def corresponding_column(self, column, require_embedded=False): - """Given a :class:`.ColumnElement`, return the exported - :class:`.ColumnElement` object from this :class:`.Selectable` - which corresponds to that original - :class:`~sqlalchemy.schema.Column` via a common ancestor - column. + def _reset_exported(self): + """delete memoized collections when a FromClause is cloned.""" - :param column: the target :class:`.ColumnElement` to be matched + self._memoized_property.expire_instance(self) - :param require_embedded: only return corresponding columns for - the given :class:`.ColumnElement`, if the given - :class:`.ColumnElement` is actually present within a sub-element - of this :class:`.FromClause`. Normally the column will match if - it merely shares a common ancestor with one of the exported - columns of this :class:`.FromClause`. + def _generate_fromclause_column_proxies(self, fromclause): + for col in self.c: + col._make_proxy(fromclause) - """ + @property + def exported_columns(self): + """A :class:`.ColumnCollection` that represents the "exported" + columns of this :class:`.Selectable`. - def embedded(expanded_proxy_set, target_set): - for t in target_set.difference(expanded_proxy_set): - if not set(_expand_cloned([t])).intersection( - expanded_proxy_set - ): - return False - return True + The "exported" columns for a :class:`.FromClause` object are synonymous + with the :attr:`.FromClause.columns` collection. - # don't dig around if the column is locally present - if self.c.contains_column(column): - return column - col, intersect = None, None - target_set = column.proxy_set - cols = self.c._all_columns - for c in cols: - expanded_proxy_set = set(_expand_cloned(c.proxy_set)) - i = target_set.intersection(expanded_proxy_set) - if i and ( - not require_embedded - or embedded(expanded_proxy_set, target_set) - ): - if col is None: - - # no corresponding column yet, pick this one. - - col, intersect = c, i - elif len(i) > len(intersect): - - # 'c' has a larger field of correspondence than - # 'col'. i.e. selectable.c.a1_x->a1.c.x->table.c.x - # matches a1.c.x->table.c.x better than - # selectable.c.x->table.c.x does. - - col, intersect = c, i - elif i == intersect: - - # they have the same field of correspondence. see - # which proxy_set has fewer columns in it, which - # indicates a closer relationship with the root - # column. Also take into account the "weight" - # attribute which CompoundSelect() uses to give - # higher precedence to columns based on vertical - # position in the compound statement, and discard - # columns that have no reference to the target - # column (also occurs with CompoundSelect) - - col_distance = util.reduce( - operator.add, - [ - sc._annotations.get("weight", 1) - for sc in col._uncached_proxy_set() - if sc.shares_lineage(column) - ], - ) - c_distance = util.reduce( - operator.add, - [ - sc._annotations.get("weight", 1) - for sc in c._uncached_proxy_set() - if sc.shares_lineage(column) - ], - ) - if c_distance < col_distance: - col, intersect = c, i - return col + .. versionadded:: 1.4 - @property - def description(self): - """a brief description of this FromClause. + .. seealso: - Used primarily for error message formatting. + :attr:`.Selectable.exported_columns` - """ - return getattr(self, "name", self.__class__.__name__ + " object") + :attr:`.SelectBase.exported_columns` - def _reset_exported(self): - """delete memoized collections when a FromClause is cloned.""" - self._memoized_property.expire_instance(self) + """ + return self.columns @_memoized_property def columns(self): @@ -660,15 +645,10 @@ class FromClause(roles.FromClauseRole, Selectable): derivations. """ - if not self._cols_populated: - return None - elif column.key in self.columns and self.columns[column.key] is column: - return column - else: - return None + self._reset_exported() -class Join(roles.AnonymizedFromClauseRole, FromClause): +class Join(FromClause): """represent a ``JOIN`` construct between two :class:`.FromClause` elements. @@ -811,23 +791,15 @@ class Join(roles.AnonymizedFromClauseRole, FromClause): (c for c in columns if c.primary_key), self.onclause ) ) - self._columns.update((col._label, col) for col in columns) + self._columns.update((col._key_label, col) for col in columns) self.foreign_keys.update( itertools.chain(*[col.foreign_keys for col in columns]) ) def _refresh_for_new_column(self, column): - col = self.left._refresh_for_new_column(column) - if col is None: - col = self.right._refresh_for_new_column(column) - if col is not None: - if self._cols_populated: - self._columns[col._label] = col - self.foreign_keys.update(col.foreign_keys) - if col.primary_key: - self.primary_key.add(col) - return col - return None + super(Join, self)._refresh_for_new_column(column) + self.left._refresh_for_new_column(column) + self.right._refresh_for_new_column(column) def _copy_internals(self, clone=_clone, **kw): self._reset_exported() @@ -947,6 +919,9 @@ class Join(roles.AnonymizedFromClauseRole, FromClause): def _joincond_scan_left_right( cls, a, a_subset, b, consider_as_foreign_keys ): + a = coercions.expect(roles.FromClauseRole, a) + b = coercions.expect(roles.FromClauseRole, b) + constraints = collections.defaultdict(list) for left in (a_subset, a): @@ -1186,23 +1161,20 @@ class Join(roles.AnonymizedFromClauseRole, FromClause): ) -class Alias(roles.AnonymizedFromClauseRole, FromClause): - """Represents an table or selectable alias (AS). - - Represents an alias, as typically applied to any table or - sub-select within a SQL statement using the ``AS`` keyword (or - without the keyword on certain databases such as Oracle). - - This object is constructed from the :func:`~.expression.alias` module - level function as well as the :meth:`.FromClause.alias` method available - on all :class:`.FromClause` subclasses. - - """ - - __visit_name__ = "alias" - named_with_column = True +# FromClause -> +# AliasedReturnsRows +# -> Alias only for FromClause +# -> Subquery only for SelectBase +# -> CTE only for HasCTE -> SelectBase, DML +# -> Lateral -> FromClause, but we accept SelectBase +# w/ non-deprecated coercion +# -> TableSample -> only for FromClause +class AliasedReturnsRows(FromClause): + """Base class of aliases against tables, subqueries, and other + selectables.""" _is_from_container = True + named_with_column = True def __init__(self, *arg, **kw): raise NotImplementedError( @@ -1224,59 +1196,12 @@ class Alias(roles.AnonymizedFromClauseRole, FromClause): return obj @classmethod - def _factory(cls, selectable, name=None, flat=False): - """Return an :class:`.Alias` object. - - An :class:`.Alias` represents any :class:`.FromClause` - with an alternate name assigned within SQL, typically using the ``AS`` - clause when generated, e.g. ``SELECT * FROM table AS aliasname``. - - Similar functionality is available via the - :meth:`~.FromClause.alias` method - available on all :class:`.FromClause` subclasses. In terms of a - SELECT object as generated from the :func:`.select` function, the - :meth:`.SelectBase.alias` method returns an :class:`.Alias` or - similar object which represents a named, parenthesized subquery. - - When an :class:`.Alias` is created from a :class:`.Table` object, - this has the effect of the table being rendered - as ``tablename AS aliasname`` in a SELECT statement. - - For :func:`.select` objects, the effect is that of creating a named - subquery, i.e. ``(select ...) AS aliasname``. - - The ``name`` parameter is optional, and provides the name - to use in the rendered SQL. If blank, an "anonymous" name - will be deterministically generated at compile time. - Deterministic means the name is guaranteed to be unique against - other constructs used in the same statement, and will also be the - same name for each successive compilation of the same statement - object. - - :param selectable: any :class:`.FromClause` subclass, - such as a table, select statement, etc. - - :param name: string name to be assigned as the alias. - If ``None``, a name will be deterministically generated - at compile time. - - :param flat: Will be passed through to if the given selectable - is an instance of :class:`.Join` - see :meth:`.Join.alias` - for details. - - .. versionadded:: 0.9.0 - - """ - return coercions.expect(roles.FromClauseRole, selectable).alias( - name=name, flat=flat - ) + def _factory(cls, returnsrows, name=None): + """Base factory method. Subclasses need to provide this.""" + raise NotImplementedError() def _init(self, selectable, name=None): - self.wrapped = selectable - if isinstance(selectable, Alias): - selectable = selectable.element - assert not isinstance(selectable, Alias) - + self.element = selectable self.supports_execution = selectable.supports_execution if self.supports_execution: self._execution_options = selectable._execution_options @@ -1288,18 +1213,14 @@ class Alias(roles.AnonymizedFromClauseRole, FromClause): and selectable.named_with_column ): name = getattr(selectable, "name", None) + if isinstance(name, _anonymous_label): + name = None name = _anonymous_label("%%(%d %s)s" % (id(self), name or "anon")) self.name = name - def self_group(self, against=None): - if ( - isinstance(against, CompoundSelect) - and isinstance(self.element, Select) - and self.element._needs_parens_for_grouping() - ): - return FromGrouping(self) - - return super(Alias, self).self_group(against=against) + def _refresh_for_new_column(self, column): + super(AliasedReturnsRows, self)._refresh_for_new_column(column) + self.element._refresh_for_new_column(column) @property def description(self): @@ -1319,18 +1240,7 @@ class Alias(roles.AnonymizedFromClauseRole, FromClause): return self.element.is_derived_from(fromclause) def _populate_column_collection(self): - for col in self.wrapped.columns._all_columns: - col._make_proxy(self) - - def _refresh_for_new_column(self, column): - col = self.wrapped._refresh_for_new_column(column) - if col is not None: - if not self._cols_populated: - return None - else: - return col._make_proxy(self) - else: - return None + self.element._generate_fromclause_column_proxies(self) def _copy_internals(self, clone=_clone, **kw): # don't apply anything to an aliased Table @@ -1339,17 +1249,13 @@ class Alias(roles.AnonymizedFromClauseRole, FromClause): if isinstance(self.element, TableClause): return self._reset_exported() - self.wrapped = clone(self.wrapped, **kw) - if isinstance(self.wrapped, Alias): - self.element = self.wrapped.element - else: - self.element = self.wrapped + self.element = clone(self.element, **kw) def get_children(self, column_collections=True, **kw): if column_collections: for c in self.c: yield c - yield self.wrapped + yield self.element def _cache_key(self, **kw): return (self.__class__, self.element._cache_key(**kw), self._orig_name) @@ -1363,7 +1269,71 @@ class Alias(roles.AnonymizedFromClauseRole, FromClause): return self.element.bind -class Lateral(Alias): +class Alias(AliasedReturnsRows): + """Represents an table or selectable alias (AS). + + Represents an alias, as typically applied to any table or + sub-select within a SQL statement using the ``AS`` keyword (or + without the keyword on certain databases such as Oracle). + + This object is constructed from the :func:`~.expression.alias` module + level function as well as the :meth:`.FromClause.alias` method available + on all :class:`.FromClause` subclasses. + + """ + + __visit_name__ = "alias" + + @classmethod + def _factory(cls, selectable, name=None, flat=False): + """Return an :class:`.Alias` object. + + An :class:`.Alias` represents any :class:`.FromClause` + with an alternate name assigned within SQL, typically using the ``AS`` + clause when generated, e.g. ``SELECT * FROM table AS aliasname``. + + Similar functionality is available via the + :meth:`~.FromClause.alias` method + available on all :class:`.FromClause` subclasses. In terms of a + SELECT object as generated from the :func:`.select` function, the + :meth:`.SelectBase.alias` method returns an :class:`.Alias` or + similar object which represents a named, parenthesized subquery. + + When an :class:`.Alias` is created from a :class:`.Table` object, + this has the effect of the table being rendered + as ``tablename AS aliasname`` in a SELECT statement. + + For :func:`.select` objects, the effect is that of creating a named + subquery, i.e. ``(select ...) AS aliasname``. + + The ``name`` parameter is optional, and provides the name + to use in the rendered SQL. If blank, an "anonymous" name + will be deterministically generated at compile time. + Deterministic means the name is guaranteed to be unique against + other constructs used in the same statement, and will also be the + same name for each successive compilation of the same statement + object. + + :param selectable: any :class:`.FromClause` subclass, + such as a table, select statement, etc. + + :param name: string name to be assigned as the alias. + If ``None``, a name will be deterministically generated + at compile time. + + :param flat: Will be passed through to if the given selectable + is an instance of :class:`.Join` - see :meth:`.Join.alias` + for details. + + .. versionadded:: 0.9.0 + + """ + return coercions.expect( + roles.FromClauseRole, selectable, allow_select=True + ).alias(name=name, flat=flat) + + +class Lateral(AliasedReturnsRows): """Represent a LATERAL subquery. This object is constructed from the :func:`~.expression.lateral` module @@ -1404,12 +1374,12 @@ class Lateral(Alias): :ref:`lateral_selects` - overview of usage. """ - return coercions.expect(roles.FromClauseRole, selectable).lateral( - name=name - ) + return coercions.expect( + roles.FromClauseRole, selectable, explicit_subquery=True + ).lateral(name=name) -class TableSample(Alias): +class TableSample(AliasedReturnsRows): """Represent a TABLESAMPLE clause. This object is constructed from the :func:`~.expression.tablesample` module @@ -1485,7 +1455,7 @@ class TableSample(Alias): return functions.func.system(self.sampling) -class CTE(Generative, HasSuffixes, Alias): +class CTE(Generative, HasSuffixes, AliasedReturnsRows): """Represent a Common Table Expression. The :class:`.CTE` object is obtained using the @@ -1531,15 +1501,6 @@ class CTE(Generative, HasSuffixes, Alias): [clone(elem, **kw) for elem in self._restates] ) - @util.dependencies("sqlalchemy.sql.dml") - def _populate_column_collection(self, dml): - if isinstance(self.element, dml.UpdateBase): - for col in self.element._returning: - col._make_proxy(self) - else: - for col in self.element.columns._all_columns: - col._make_proxy(self) - def alias(self, name=None, flat=False): """Return an :class:`.Alias` of this :class:`.CTE`. @@ -1748,13 +1709,26 @@ class HasCTE(roles.HasCTERole): return CTE._construct(self, name=name, recursive=recursive) -class FromGrouping(FromClause): - """Represent a grouping of a FROM clause""" +class Subquery(AliasedReturnsRows): + __visit_name__ = "subquery" - __visit_name__ = "grouping" + _is_subquery = True + + @classmethod + def _factory(cls, selectable, name=None): + """Return a :class:`.Subquery` object. + + """ + return coercions.expect( + roles.SelectStatementRole, selectable + ).subquery(name=name) + + +class FromGrouping(GroupedElement, FromClause): + """Represent a grouping of a FROM clause""" def __init__(self, element): - self.element = element + self.element = coercions.expect(roles.FromClauseRole, element) def _init_collections(self): pass @@ -1794,9 +1768,6 @@ class FromGrouping(FromClause): def _from_objects(self): return self.element._from_objects - def __getattr__(self, attr): - return getattr(self.element, attr) - def __getstate__(self): return {"element": self.element} @@ -1804,7 +1775,7 @@ class FromGrouping(FromClause): self.element = state["element"] -class TableClause(roles.AnonymizedFromClauseRole, Immutable, FromClause): +class TableClause(Immutable, FromClause): """Represents a minimal "table" construct. This is a lightweight table object that has only a name and a @@ -1870,6 +1841,9 @@ class TableClause(roles.AnonymizedFromClauseRole, Immutable, FromClause): for c in columns: self.append_column(c) + def _refresh_for_new_column(self, column): + pass + def _init_collections(self): pass @@ -2065,19 +2039,122 @@ class SelectBase( roles.InElementRole, HasCTE, Executable, - FromClause, + Selectable, ): """Base class for SELECT statements. This includes :class:`.Select`, :class:`.CompoundSelect` and - :class:`.TextAsFrom`. + :class:`.TextualSelect`. """ _is_select_statement = True + _memoized_property = util.group_expirable_memoized_property() + + def _reset_memoizations(self): + self._memoized_property.expire_instance(self) + + def _generate_fromclause_column_proxies(self, fromclause): + # type: (FromClause) + raise NotImplementedError() + + def _refresh_for_new_column(self, column): + self._reset_memoizations() + + @property + def selected_columns(self): + """A :class:`.ColumnCollection` representing the columns that + this SELECT statement or similar construct returns in its result set. + + This collection differs from the :attr:`.FromClause.columns` collection + of a :class:`.FromClause` in that the columns within this collection + cannot be directly nested inside another SELECT statement; a subquery + must be applied first which provides for the necessary parenthesization + required by SQL. + + .. versionadded:: 1.4 + + """ + raise NotImplementedError() + + @property + def exported_columns(self): + """A :class:`.ColumnCollection` that represents the "exported" + columns of this :class:`.Selectable`. + + The "exported" columns for a :class:`.SelectBase` object are synonymous + with the :attr:`.SelectBase.selected_columns` collection. + + .. versionadded:: 1.4 + + .. seealso: + + :attr:`.Selectable.exported_columns` + + :attr:`.FromClause.exported_columns` + + + """ + return self.selected_columns + + @property + @util.deprecated( + "1.4", + "The :attr:`.SelectBase.c` and :attr:`.SelectBase.columns` attributes " + "are deprecated and will be removed in a future release; these " + "attributes implicitly create a subquery that should be explicit. " + "Please call :meth:`.SelectBase.subquery` first in order to create " + "a subquery, which then contains this attribute. To access the " + "columns that this SELECT object SELECTs " + "from, use the :attr:`.SelectBase.selected_columns` attribute.", + ) + def c(self): + return self._implicit_subquery.columns + + @property + def columns(self): + return self.c + + @util.deprecated( + "1.4", + "The :meth:`.SelectBase.select` method is deprecated " + "and will be removed in a future release; this method implicitly " + "creates a subquery that should be explicit. " + "Please call :meth:`.SelectBase.subquery` first in order to create " + "a subquery, which then can be seleted.", + ) + def select(self, *arg, **kw): + return self._implicit_subquery.select(*arg, **kw) + + @util.deprecated( + "1.4", + "The :meth:`.SelectBase.join` method is deprecated " + "and will be removed in a future release; this method implicitly " + "creates a subquery that should be explicit. " + "Please call :meth:`.SelectBase.subquery` first in order to create " + "a subquery, which then can be seleted.", + ) + def join(self, *arg, **kw): + return self._implicit_subquery.join(*arg, **kw) + + @util.deprecated( + "1.4", + "The :meth:`.SelectBase.outerjoin` method is deprecated " + "and will be removed in a future release; this method implicitly " + "creates a subquery that should be explicit. " + "Please call :meth:`.SelectBase.subquery` first in order to create " + "a subquery, which then can be seleted.", + ) + def outerjoin(self, *arg, **kw): + return self._implicit_subquery.outerjoin(*arg, **kw) + + @_memoized_property + def _implicit_subquery(self): + return self.subquery() + @util.deprecated( "1.4", "The :meth:`.SelectBase.as_scalar` method is deprecated and will be " @@ -2117,6 +2194,21 @@ class SelectBase( """ return self.scalar_subquery().label(name) + def lateral(self, name=None): + """Return a LATERAL alias of this :class:`.Selectable`. + + The return value is the :class:`.Lateral` construct also + provided by the top-level :func:`~.expression.lateral` function. + + .. versionadded:: 1.1 + + .. seealso:: + + :ref:`lateral_selects` - overview of usage. + + """ + return Lateral._factory(self, name) + @_generative @util.deprecated( "0.6", @@ -2141,7 +2233,7 @@ class SelectBase( s = self.__class__.__new__(self.__class__) s.__dict__ = self.__dict__.copy() - s._reset_exported() + s._reset_memoizations() return s @property @@ -2182,7 +2274,78 @@ class SelectBase( .. versionadded:: 1.4 """ - return self.alias() + return Subquery._construct(self, name) + + def alias(self, name=None, flat=False): + """Return a named subquery against this :class:`.SelectBase`. + + For a :class:`.SelectBase` (as opposed to a :class:`.FromClause`), + this returns a :class:`.Subquery` object which behaves mostly the + same as the :class:`.Alias` object that is used with a + :class:`.FromClause`. + + .. versionchanged:: 1.4 The :meth:`.SelectBase.alias` method is now + a synonym for the :meth:`.SelectBase.subquery` method. + + """ + return self.subquery(name=name) + + +class SelectStatementGrouping(GroupedElement, SelectBase): + """Represent a grouping of a :class:`.SelectBase`. + + This differs from :class:`.Subquery` in that we are still + an "inner" SELECT statement, this is strictly for grouping inside of + compound selects. + + """ + + __visit_name__ = "grouping" + + def __init__(self, element): + # type: (SelectBase) + self.element = coercions.expect(roles.SelectStatementRole, element) + + @property + def select_statement(self): + return self.element + + def get_children(self, **kwargs): + return (self.element,) + + def self_group(self, against=None): + # type: (Optional[Any]) -> FromClause + return self + + def _generate_fromclause_column_proxies(self, subquery): + self.element._generate_fromclause_column_proxies(subquery) + + def _generate_proxy_for_new_column(self, column, subquery): + return self.element._generate_proxy_for_new_column(subquery) + + @property + def selected_columns(self): + """A :class:`.ColumnCollection` representing the columns that + the embedded SELECT statement returns in its result set. + + .. versionadded:: 1.4 + + .. seealso:: + + :ref:`.SelectBase.selected_columns` + + """ + return self.element.selected_columns + + def _copy_internals(self, clone=_clone, **kw): + self.element = clone(self.element, **kw) + + def _cache_key(self, **kw): + return (SelectStatementGrouping, self.element._cache_key(**kw)) + + @property + def _from_objects(self): + return self.element._from_objects class GenerativeSelect(SelectBase): @@ -2191,7 +2354,7 @@ class GenerativeSelect(SelectBase): This serves as the base for :class:`.Select` and :class:`.CompoundSelect` where elements such as ORDER BY, GROUP BY can be added and column - rendering can be controlled. Compare to :class:`.TextAsFrom`, which, + rendering can be controlled. Compare to :class:`.TextualSelect`, which, while it subclasses :class:`.SelectBase` and is also a SELECT construct, represents a fixed textual string which cannot be altered at this level, only wrapped as a subquery. @@ -2199,7 +2362,7 @@ class GenerativeSelect(SelectBase): .. versionadded:: 0.9.0 :class:`.GenerativeSelect` was added to provide functionality specific to :class:`.Select` and :class:`.CompoundSelect` while allowing :class:`.SelectBase` to be - used for other SELECT-like objects, e.g. :class:`.TextAsFrom`. + used for other SELECT-like objects, e.g. :class:`.TextualSelect`. """ @@ -2591,8 +2754,8 @@ class CompoundSelect(GenerativeSelect): s = coercions.expect(roles.CompoundElementRole, s) if not numcols: - numcols = len(s.c._all_columns) - elif len(s.c._all_columns) != numcols: + numcols = len(s.selected_columns) + elif len(s.selected_columns) != numcols: raise exc.ArgumentError( "All selectables passed to " "CompoundSelect must have identical numbers of " @@ -2600,9 +2763,9 @@ class CompoundSelect(GenerativeSelect): "#%d has %d" % ( 1, - len(self.selects[0].c._all_columns), + len(self.selects[0].selected_columns), n + 1, - len(s.c._all_columns), + len(s.selected_columns), ) ) @@ -2610,9 +2773,12 @@ class CompoundSelect(GenerativeSelect): GenerativeSelect.__init__(self, **kwargs) - @property + @SelectBase._memoized_property def _label_resolve_dict(self): - d = dict((c.key, c) for c in self.c) + # TODO: this is hacky and slow + hacky_subquery = self.subquery() + hacky_subquery.named_with_column = False + d = dict((c.key, c) for c in hacky_subquery.c) return d, d, d @classmethod @@ -2727,7 +2893,8 @@ class CompoundSelect(GenerativeSelect): return self.selects[0]._scalar_type() def self_group(self, against=None): - return FromGrouping(self) + # type: (Optional[Any]) -> FromClause + return SelectStatementGrouping(self) def is_derived_from(self, fromclause): for s in self.selects: @@ -2735,50 +2902,59 @@ class CompoundSelect(GenerativeSelect): return True return False - def _populate_column_collection(self): - for cols in zip(*[s.c._all_columns for s in self.selects]): - - # this is a slightly hacky thing - the union exports a - # column that resembles just that of the *first* selectable. - # to get at a "composite" column, particularly foreign keys, - # you have to dig through the proxies collection which we - # generate below. We may want to improve upon this, such as - # perhaps _make_proxy can accept a list of other columns - # that are "shared" - schema.column can then copy all the - # ForeignKeys in. this would allow the union() to have all - # those fks too. - - proxy = cols[0]._make_proxy( - self, - name=cols[0]._label if self.use_labels else None, - key=cols[0]._key_label if self.use_labels else None, - ) - - # hand-construct the "_proxies" collection to include all - # derived columns place a 'weight' annotation corresponding - # to how low in the list of select()s the column occurs, so - # that the corresponding_column() operation can resolve - # conflicts - proxy._proxies = [ - c._annotate({"weight": i + 1}) for (i, c) in enumerate(cols) + def _generate_fromclause_column_proxies(self, subquery): + + # this is a slightly hacky thing - the union exports a + # column that resembles just that of the *first* selectable. + # to get at a "composite" column, particularly foreign keys, + # you have to dig through the proxies collection which we + # generate below. We may want to improve upon this, such as + # perhaps _make_proxy can accept a list of other columns + # that are "shared" - schema.column can then copy all the + # ForeignKeys in. this would allow the union() to have all + # those fks too. + select_0 = self.selects[0] + if self.use_labels: + select_0 = select_0.apply_labels() + select_0._generate_fromclause_column_proxies(subquery) + + # hand-construct the "_proxies" collection to include all + # derived columns place a 'weight' annotation corresponding + # to how low in the list of select()s the column occurs, so + # that the corresponding_column() operation can resolve + # conflicts + for subq_col, select_cols in zip( + subquery.c._all_columns, + zip(*[s.selected_columns for s in self.selects]), + ): + subq_col._proxies = [ + c._annotate({"weight": i + 1}) + for (i, c) in enumerate(select_cols) ] def _refresh_for_new_column(self, column): - for s in self.selects: - s._refresh_for_new_column(column) + super(CompoundSelect, self)._refresh_for_new_column(column) + for select in self.selects: + select._refresh_for_new_column(column) - if not self._cols_populated: - return None + @property + def selected_columns(self): + """A :class:`.ColumnCollection` representing the columns that + this SELECT statement or similar construct returns in its result set. - raise NotImplementedError( - "CompoundSelect constructs don't support " - "addition of columns to underlying " - "selectables" - ) + For a :class:`.CompoundSelect`, the + :attr:`.CompoundSelect.selected_columns` attribute returns the selected + columns of the first SELECT statement contined within the series of + statements within the set operation. + + .. versionadded:: 1.4 + + """ + return self.selects[0].selected_columns def _copy_internals(self, clone=_clone, **kw): super(CompoundSelect, self)._copy_internals(clone, **kw) - self._reset_exported() + self._reset_memoizations() self.selects = [clone(s, **kw) for s in self.selects] if hasattr(self, "_col_map"): del self._col_map @@ -2790,11 +2966,9 @@ class CompoundSelect(GenerativeSelect): if getattr(self, attr) is not None: setattr(self, attr, clone(getattr(self, attr), **kw)) - def get_children(self, column_collections=True, **kwargs): - return ( - (column_collections and list(self.c) or []) - + [self._order_by_clause, self._group_by_clause] - + list(self.selects) + def get_children(self, **kwargs): + return [self._order_by_clause, self._group_by_clause] + list( + self.selects ) def _cache_key(self, **kw): @@ -3142,7 +3316,7 @@ class Select(HasPrefixes, HasSuffixes, GenerativeSelect): else (), self._from_obj, ): - if item is self: + if item._is_subquery and item.element is self: raise exc.InvalidRequestError( "select() construct refers to itself as a FROM" ) @@ -3415,16 +3589,15 @@ class Select(HasPrefixes, HasSuffixes, GenerativeSelect): if getattr(self, attr) is not None: setattr(self, attr, clone(getattr(self, attr), **kw)) - # erase exported column list, _froms collection, + # erase _froms collection, # etc. - self._reset_exported() + self._reset_memoizations() - def get_children(self, column_collections=True, **kwargs): + def get_children(self, **kwargs): """return child elements as per the ClauseElement specification.""" return ( - (column_collections and list(self.columns) or []) - + self._raw_columns + self._raw_columns + list(self._froms) + [ x @@ -3594,7 +3767,7 @@ class Select(HasPrefixes, HasSuffixes, GenerativeSelect): asked to select both from ``table1`` as well as itself. """ - self._reset_exported() + self._reset_memoizations() rc = [] for c in columns: c = coercions.expect(roles.ColumnsClauseRole, c) @@ -3789,7 +3962,7 @@ class Select(HasPrefixes, HasSuffixes, GenerativeSelect): :class:`.Select` object. """ - self._reset_exported() + self._reset_memoizations() column = coercions.expect(roles.ColumnsClauseRole, column) if isinstance(column, ScalarSelect): @@ -3821,7 +3994,7 @@ class Select(HasPrefixes, HasSuffixes, GenerativeSelect): """ - self._reset_exported() + self._reset_memoizations() self._whereclause = and_(True_._ifnone(self._whereclause), whereclause) def append_having(self, having): @@ -3835,7 +4008,7 @@ class Select(HasPrefixes, HasSuffixes, GenerativeSelect): :term:`method chaining`. """ - self._reset_exported() + self._reset_memoizations() self._having = and_(True_._ifnone(self._having), having) def append_from(self, fromclause): @@ -3847,10 +4020,60 @@ class Select(HasPrefixes, HasSuffixes, GenerativeSelect): standard :term:`method chaining`. """ - self._reset_exported() + self._reset_memoizations() fromclause = coercions.expect(roles.FromClauseRole, fromclause) self._from_obj = self._from_obj.union([fromclause]) + @_memoized_property + def selected_columns(self): + """A :class:`.ColumnCollection` representing the columns that + this SELECT statement or similar construct returns in its result set. + + This collection differs from the :attr:`.FromClause.columns` collection + of a :class:`.FromClause` in that the columns within this collection + cannot be directly nested inside another SELECT statement; a subquery + must be applied first which provides for the necessary parenthesization + required by SQL. + + For a :func:`.select` construct, the collection here is exactly what + would be rendered inside the "SELECT" statement, and the + :class:`.ColumnElement` objects are directly present as they were + given, e.g.:: + + col1 = column('q', Integer) + col2 = column('p', Integer) + stmt = select([col1, col2]) + + Above, ``stmt.selected_columns`` would be a collection that contains + the ``col1`` and ``col2`` objects directly. For a statement that is + against a :class:`.Table` or other :class:`.FromClause`, the collection + will use the :class:`.ColumnElement` objects that are in the + :attr:`.FromClause.c` collection of the from element. + + .. versionadded:: 1.4 + + """ + names = set() + + def name_for_col(c): + # we use key_label since this name is intended for targeting + # within the ColumnCollection only, it's not related to SQL + # rendering which always uses column name for SQL label names + if self.use_labels: + name = c._key_label + else: + name = c._proxy_key + if name in names: + name = c.anon_label + else: + names.add(name) + return name + + return SeparateKeyColumnCollection( + (name_for_col(c), c) + for c in util.unique_list(_select_iterables(self._raw_columns)) + ).as_immutable() + @_memoized_property def _columns_plus_names(self): if self.use_labels: @@ -3877,7 +4100,9 @@ class Select(HasPrefixes, HasSuffixes, GenerativeSelect): for c in util.unique_list(_select_iterables(self._raw_columns)) ] - def _populate_column_collection(self): + def _generate_fromclause_column_proxies(self, subquery): + keys_seen = set() + for name, c in self._columns_plus_names: if not hasattr(c, "_make_proxy"): continue @@ -3885,27 +4110,15 @@ class Select(HasPrefixes, HasSuffixes, GenerativeSelect): key = None elif self.use_labels: key = c._key_label - if key is not None and key in self.c: + if key is not None and key in keys_seen: key = c.anon_label + keys_seen.add(key) else: key = None - c._make_proxy(self, key=key, name=name, name_is_truncatable=True) - def _refresh_for_new_column(self, column): - for fromclause in self._froms: - col = fromclause._refresh_for_new_column(column) - if col is not None: - if col in self.inner_columns and self._cols_populated: - our_label = col._key_label if self.use_labels else col.key - if our_label not in self.c: - return col._make_proxy( - self, - name=col._label if self.use_labels else None, - key=col._key_label if self.use_labels else None, - name_is_truncatable=True, - ) - return None - return None + c._make_proxy( + subquery, key=key, name=name, name_is_truncatable=True + ) def _needs_parens_for_grouping(self): return ( @@ -3928,7 +4141,8 @@ class Select(HasPrefixes, HasSuffixes, GenerativeSelect): and not self._needs_parens_for_grouping() ): return self - return FromGrouping(self) + else: + return SelectStatementGrouping(self) def union(self, other, **kwargs): """return a SQL UNION of this select() construct against the given @@ -4030,7 +4244,6 @@ class Exists(UnaryExpression): """ - __visit_name__ = UnaryExpression.__visit_name__ _from_objects = [] def __init__(self, *args, **kwargs): @@ -4056,7 +4269,7 @@ class Exists(UnaryExpression): else: if not args: args = ([literal_column("*")],) - s = Select(*args, **kwargs).scalar_subquery().self_group() + s = Select(*args, **kwargs).scalar_subquery() UnaryExpression.__init__( self, @@ -4066,17 +4279,26 @@ class Exists(UnaryExpression): wraps_column_expression=True, ) + def _regroup(self, fn): + element = self.element._ungroup() + element = fn(element) + return element.self_group(against=operators.exists) + def select(self, whereclause=None, **params): return Select([self], whereclause, **params) def correlate(self, *fromclause): e = self._clone() - e.element = self.element.correlate(*fromclause).self_group() + e.element = self._regroup( + lambda element: element.correlate(*fromclause) + ) return e def correlate_except(self, *fromclause): e = self._clone() - e.element = self.element.correlate_except(*fromclause).self_group() + e.element = self._regroup( + lambda element: element.correlate_except(*fromclause) + ) return e def select_from(self, clause): @@ -4086,7 +4308,7 @@ class Exists(UnaryExpression): """ e = self._clone() - e.element = self.element.select_from(clause).self_group() + e.element = self._regroup(lambda element: element.select_from(clause)) return e def where(self, clause): @@ -4095,12 +4317,11 @@ class Exists(UnaryExpression): """ e = self._clone() - e.element = self.element.where(clause).self_group() + e.element = self._regroup(lambda element: element.where(clause)) return e -# TODO: rename to TextualSelect, this is not a FROM clause -class TextAsFrom(SelectBase): +class TextualSelect(SelectBase): """Wrap a :class:`.TextClause` construct within a :class:`.SelectBase` interface. @@ -4108,20 +4329,22 @@ class TextAsFrom(SelectBase): and other FROM-like capabilities such as :meth:`.FromClause.alias`, :meth:`.SelectBase.cte`, etc. - The :class:`.TextAsFrom` construct is produced via the + The :class:`.TextualSelect` construct is produced via the :meth:`.TextClause.columns` method - see that method for details. - .. versionadded:: 0.9.0 + .. versionchanged:: 1.4 the :class:`.TextualSelect` class was renamed + from ``TextAsFrom``, to more correctly suit its role as a + SELECT-oriented object and not a FROM clause. .. seealso:: :func:`.text` - :meth:`.TextClause.columns` + :meth:`.TextClause.columns` - primary creation interface. """ - __visit_name__ = "text_as_from" + __visit_name__ = "textual_select" _is_textual = True @@ -4130,6 +4353,26 @@ class TextAsFrom(SelectBase): self.column_args = columns self.positional = positional + @SelectBase._memoized_property + def selected_columns(self): + """A :class:`.ColumnCollection` representing the columns that + this SELECT statement or similar construct returns in its result set. + + This collection differs from the :attr:`.FromClause.columns` collection + of a :class:`.FromClause` in that the columns within this collection + cannot be directly nested inside another SELECT statement; a subquery + must be applied first which provides for the necessary parenthesization + required by SQL. + + For a :class:`.TextualSelect` construct, the collection contains the + :class:`.ColumnElement` objects that were passed to the constructor, + typically via the :meth:`.TextClause.columns` method. + + .. versionadded:: 1.4 + + """ + return ColumnCollection(*self.column_args).as_immutable() + @property def _bind(self): return self.element._bind @@ -4138,22 +4381,19 @@ class TextAsFrom(SelectBase): def bindparams(self, *binds, **bind_as_values): self.element = self.element.bindparams(*binds, **bind_as_values) - def _populate_column_collection(self): + def _generate_fromclause_column_proxies(self, fromclause): for c in self.column_args: - c._make_proxy(self) + c._make_proxy(fromclause) def _copy_internals(self, clone=_clone, **kw): - self._reset_exported() + self._reset_memoizations() self.element = clone(self.element, **kw) - def get_children(self, column_collections=True, **kw): - if column_collections: - for c in self.column_args: - yield c - yield self.element + def get_children(self, **kw): + return [self.element] def _cache_key(self, **kw): - return (TextAsFrom, self.element._cache_key(**kw)) + tuple( + return (TextualSelect, self.element._cache_key(**kw)) + tuple( col._cache_key(**kw) for col in self.column_args ) @@ -4161,6 +4401,10 @@ class TextAsFrom(SelectBase): return self.column_args[0].type +TextAsFrom = TextualSelect +"""Backwards compatibility with the previous name""" + + class AnnotatedFromClause(Annotated): def __init__(self, element, values): # force FromClause to generate their internal diff --git a/lib/sqlalchemy/sql/util.py b/lib/sqlalchemy/sql/util.py index d90b3f158b..dd2c7c1fb5 100644 --- a/lib/sqlalchemy/sql/util.py +++ b/lib/sqlalchemy/sql/util.py @@ -17,10 +17,10 @@ from . import visitors from .annotation import _deep_annotate # noqa from .annotation import _deep_deannotate # noqa from .annotation import _shallow_annotate # noqa +from .base import _expand_cloned from .base import _from_objects from .base import ColumnSet from .ddl import sort_tables # noqa -from .elements import _expand_cloned from .elements import _find_columns # noqa from .elements import _label_reference from .elements import _textual_label_reference diff --git a/lib/sqlalchemy/testing/suite/__init__.py b/lib/sqlalchemy/testing/suite/__init__.py index 465a6cfc57..4c71157cd5 100644 --- a/lib/sqlalchemy/testing/suite/__init__.py +++ b/lib/sqlalchemy/testing/suite/__init__.py @@ -1,5 +1,6 @@ from .test_cte import * # noqa from .test_ddl import * # noqa +from .test_deprecations import * # noqa from .test_dialect import * # noqa from .test_insert import * # noqa from .test_reflection import * # noqa diff --git a/lib/sqlalchemy/testing/suite/test_deprecations.py b/lib/sqlalchemy/testing/suite/test_deprecations.py new file mode 100644 index 0000000000..d0202a0a95 --- /dev/null +++ b/lib/sqlalchemy/testing/suite/test_deprecations.py @@ -0,0 +1,144 @@ +from .. import config +from .. import fixtures +from ..assertions import eq_ +from ..schema import Column +from ..schema import Table +from ... import Integer +from ... import select +from ... import testing +from ... import union + + +class DeprecatedCompoundSelectTest(fixtures.TablesTest): + __backend__ = True + + @classmethod + def define_tables(cls, metadata): + Table( + "some_table", + metadata, + Column("id", Integer, primary_key=True), + Column("x", Integer), + Column("y", Integer), + ) + + @classmethod + def insert_data(cls): + config.db.execute( + cls.tables.some_table.insert(), + [ + {"id": 1, "x": 1, "y": 2}, + {"id": 2, "x": 2, "y": 3}, + {"id": 3, "x": 3, "y": 4}, + {"id": 4, "x": 4, "y": 5}, + ], + ) + + def _assert_result(self, select, result, params=()): + eq_(config.db.execute(select, params).fetchall(), result) + + def test_plain_union(self): + table = self.tables.some_table + s1 = select([table]).where(table.c.id == 2) + s2 = select([table]).where(table.c.id == 3) + + u1 = union(s1, s2) + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns " + "attributes are deprecated" + ): + self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + + # note we've had to remove one use case entirely, which is this + # one. the Select gets its FROMS from the WHERE clause and the + # columns clause, but not the ORDER BY, which means the old ".c" system + # allowed you to "order_by(s.c.foo)" to get an unnamed column in the + # ORDER BY without adding the SELECT into the FROM and breaking the + # query. Users will have to adjust for this use case if they were doing + # it before. + def _dont_test_select_from_plain_union(self): + table = self.tables.some_table + s1 = select([table]).where(table.c.id == 2) + s2 = select([table]).where(table.c.id == 3) + + u1 = union(s1, s2).alias().select() + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns " + "attributes are deprecated" + ): + self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + + @testing.requires.order_by_col_from_union + @testing.requires.parens_in_union_contained_select_w_limit_offset + def test_limit_offset_selectable_in_unions(self): + table = self.tables.some_table + s1 = ( + select([table]) + .where(table.c.id == 2) + .limit(1) + .order_by(table.c.id) + ) + s2 = ( + select([table]) + .where(table.c.id == 3) + .limit(1) + .order_by(table.c.id) + ) + + u1 = union(s1, s2).limit(2) + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns " + "attributes are deprecated" + ): + self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + + @testing.requires.parens_in_union_contained_select_wo_limit_offset + def test_order_by_selectable_in_unions(self): + table = self.tables.some_table + s1 = select([table]).where(table.c.id == 2).order_by(table.c.id) + s2 = select([table]).where(table.c.id == 3).order_by(table.c.id) + + u1 = union(s1, s2).limit(2) + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns " + "attributes are deprecated" + ): + self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + + def test_distinct_selectable_in_unions(self): + table = self.tables.some_table + s1 = select([table]).where(table.c.id == 2).distinct() + s2 = select([table]).where(table.c.id == 3).distinct() + + u1 = union(s1, s2).limit(2) + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns " + "attributes are deprecated" + ): + self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + + def test_limit_offset_aliased_selectable_in_unions(self): + table = self.tables.some_table + s1 = ( + select([table]) + .where(table.c.id == 2) + .limit(1) + .order_by(table.c.id) + .alias() + .select() + ) + s2 = ( + select([table]) + .where(table.c.id == 3) + .limit(1) + .order_by(table.c.id) + .alias() + .select() + ) + + u1 = union(s1, s2).limit(2) + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns " + "attributes are deprecated" + ): + self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) diff --git a/lib/sqlalchemy/testing/suite/test_select.py b/lib/sqlalchemy/testing/suite/test_select.py index d265851419..dabb30f9b8 100644 --- a/lib/sqlalchemy/testing/suite/test_select.py +++ b/lib/sqlalchemy/testing/suite/test_select.py @@ -268,7 +268,9 @@ class CompoundSelectTest(fixtures.TablesTest): s2 = select([table]).where(table.c.id == 3) u1 = union(s1, s2) - self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + self._assert_result( + u1.order_by(u1.selected_columns.id), [(2, 2, 3), (3, 3, 4)] + ) def test_select_from_plain_union(self): table = self.tables.some_table @@ -276,7 +278,9 @@ class CompoundSelectTest(fixtures.TablesTest): s2 = select([table]).where(table.c.id == 3) u1 = union(s1, s2).alias().select() - self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + self._assert_result( + u1.order_by(u1.selected_columns.id), [(2, 2, 3), (3, 3, 4)] + ) @testing.requires.order_by_col_from_union @testing.requires.parens_in_union_contained_select_w_limit_offset @@ -296,7 +300,9 @@ class CompoundSelectTest(fixtures.TablesTest): ) u1 = union(s1, s2).limit(2) - self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + self._assert_result( + u1.order_by(u1.selected_columns.id), [(2, 2, 3), (3, 3, 4)] + ) @testing.requires.parens_in_union_contained_select_wo_limit_offset def test_order_by_selectable_in_unions(self): @@ -305,7 +311,9 @@ class CompoundSelectTest(fixtures.TablesTest): s2 = select([table]).where(table.c.id == 3).order_by(table.c.id) u1 = union(s1, s2).limit(2) - self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + self._assert_result( + u1.order_by(u1.selected_columns.id), [(2, 2, 3), (3, 3, 4)] + ) def test_distinct_selectable_in_unions(self): table = self.tables.some_table @@ -313,7 +321,9 @@ class CompoundSelectTest(fixtures.TablesTest): s2 = select([table]).where(table.c.id == 3).distinct() u1 = union(s1, s2).limit(2) - self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + self._assert_result( + u1.order_by(u1.selected_columns.id), [(2, 2, 3), (3, 3, 4)] + ) @testing.requires.parens_in_union_contained_select_w_limit_offset def test_limit_offset_in_unions_from_alias(self): @@ -357,7 +367,9 @@ class CompoundSelectTest(fixtures.TablesTest): ) u1 = union(s1, s2).limit(2) - self._assert_result(u1.order_by(u1.c.id), [(2, 2, 3), (3, 3, 4)]) + self._assert_result( + u1.order_by(u1.selected_columns.id), [(2, 2, 3), (3, 3, 4)] + ) class ExpandingBoundInTest(fixtures.TablesTest): diff --git a/test/aaa_profiling/test_memusage.py b/test/aaa_profiling/test_memusage.py index d172c4a87d..156d480441 100644 --- a/test/aaa_profiling/test_memusage.py +++ b/test/aaa_profiling/test_memusage.py @@ -963,7 +963,7 @@ class MemUsageWBackendTest(EnsureZeroed): # http://thread.gmane.org/gmane.comp.python.db.pysqlite.user/2290 @testing.crashes("mysql+cymysql", "blocking") - def test_join_cache(self): + def test_join_cache_deprecated_coercion(self): metadata = MetaData(self.engine) table1 = Table( "table1", @@ -999,7 +999,55 @@ class MemUsageWBackendTest(EnsureZeroed): def go(): s = table2.select() sess = session() - sess.query(Foo).join((s, Foo.bars)).all() + with testing.expect_deprecated( + "Implicit coercion of SELECT and " "textual SELECT constructs" + ): + sess.query(Foo).join(s, Foo.bars).all() + sess.rollback() + + try: + go() + finally: + metadata.drop_all() + + @testing.crashes("mysql+cymysql", "blocking") + def test_join_cache(self): + metadata = MetaData(self.engine) + table1 = Table( + "table1", + metadata, + Column( + "id", Integer, primary_key=True, test_needs_autoincrement=True + ), + Column("data", String(30)), + ) + table2 = Table( + "table2", + metadata, + Column( + "id", Integer, primary_key=True, test_needs_autoincrement=True + ), + Column("data", String(30)), + Column("t1id", Integer, ForeignKey("table1.id")), + ) + + class Foo(object): + pass + + class Bar(object): + pass + + mapper( + Foo, table1, properties={"bars": relationship(mapper(Bar, table2))} + ) + metadata.create_all() + session = sessionmaker() + + @profile_memory() + def go(): + s = table2.select().subquery() + sess = session() + sess.query(Foo).join(s, Foo.bars).all() sess.rollback() try: diff --git a/test/base/test_utils.py b/test/base/test_utils.py index fea34cf8da..d7e4deb289 100644 --- a/test/base/test_utils.py +++ b/test/base/test_utils.py @@ -9,6 +9,7 @@ from sqlalchemy import sql from sqlalchemy import testing from sqlalchemy import util from sqlalchemy.sql import column +from sqlalchemy.sql.base import SeparateKeyColumnCollection from sqlalchemy.testing import assert_raises from sqlalchemy.testing import assert_raises_message from sqlalchemy.testing import eq_ @@ -441,6 +442,29 @@ class ToListTest(fixtures.TestBase): ) +class SeparateKeysColumnCollectionTest( + testing.AssertsCompiledSQL, fixtures.TestBase +): + def test_in(self): + cc = SeparateKeyColumnCollection() + cc["kcol1"] = sql.column("col1") + cc["kcol2"] = sql.column("col2") + cc["kcol3"] = sql.column("col3") + assert "col1" not in cc + assert "kcol2" in cc + + def test_get(self): + c1, c2 = sql.column("col1"), sql.column("col2") + cc = SeparateKeyColumnCollection([("kcol1", c1), ("kcol2", c2)]) + is_(cc.kcol1, c1) + is_(cc.kcol2, c2) + + def test_all_cols(self): + c1, c2 = sql.column("col1"), sql.column("col2") + cc = SeparateKeyColumnCollection([("kcol1", c1), ("kcol2", c2)]) + eq_(cc._all_columns, [c1, c2]) + + class ColumnCollectionTest(testing.AssertsCompiledSQL, fixtures.TestBase): def test_in(self): cc = sql.ColumnCollection() diff --git a/test/dialect/oracle/test_compiler.py b/test/dialect/oracle/test_compiler.py index f11daebb46..f6d4895a5d 100644 --- a/test/dialect/oracle/test_compiler.py +++ b/test/dialect/oracle/test_compiler.py @@ -66,14 +66,14 @@ class CompileTest(fixtures.TestBase, AssertsCompiledSQL): def test_subquery(self): t = table("sometable", column("col1"), column("col2")) - s = select([t]) + s = select([t]).subquery() s = select([s.c.col1, s.c.col2]) self.assert_compile( s, - "SELECT col1, col2 FROM (SELECT " + "SELECT anon_1.col1, anon_1.col2 FROM (SELECT " "sometable.col1 AS col1, sometable.col2 " - "AS col2 FROM sometable)", + "AS col2 FROM sometable) anon_1", ) def test_bindparam_quote(self): @@ -176,33 +176,34 @@ class CompileTest(fixtures.TestBase, AssertsCompiledSQL): def test_limit_two(self): t = table("sometable", column("col1"), column("col2")) - s = select([t]).limit(10).offset(20) + s = select([t]).limit(10).offset(20).subquery() - # TODO: this will require a subquery s2 = select([s.c.col1, s.c.col2]) self.assert_compile( s2, - "SELECT col1, col2 FROM (SELECT anon_1.col1 AS col1, " - "anon_1.col2 AS col2 " - "FROM (SELECT anon_2.col1 AS col1, anon_2.col2 AS col2, " + "SELECT anon_1.col1, anon_1.col2 FROM " + "(SELECT anon_2.col1 AS col1, " + "anon_2.col2 AS col2 " + "FROM (SELECT anon_3.col1 AS col1, anon_3.col2 AS col2, " "ROWNUM AS ora_rn " "FROM (SELECT sometable.col1 AS col1, " - "sometable.col2 AS col2 FROM sometable) anon_2 " - "WHERE ROWNUM <= :param_1 + :param_2) anon_1 " - "WHERE ora_rn > :param_2)", + "sometable.col2 AS col2 FROM sometable) anon_3 " + "WHERE ROWNUM <= :param_1 + :param_2) anon_2 " + "WHERE ora_rn > :param_2) anon_1", checkparams={"param_1": 10, "param_2": 20}, ) self.assert_compile( s2, - "SELECT col1, col2 FROM (SELECT anon_1.col1 AS col1, " - "anon_1.col2 AS col2 " - "FROM (SELECT anon_2.col1 AS col1, anon_2.col2 AS col2, " + "SELECT anon_1.col1, anon_1.col2 FROM " + "(SELECT anon_2.col1 AS col1, " + "anon_2.col2 AS col2 " + "FROM (SELECT anon_3.col1 AS col1, anon_3.col2 AS col2, " "ROWNUM AS ora_rn " "FROM (SELECT sometable.col1 AS col1, " - "sometable.col2 AS col2 FROM sometable) anon_2 " - "WHERE ROWNUM <= :param_1 + :param_2) anon_1 " - "WHERE ora_rn > :param_2)", + "sometable.col2 AS col2 FROM sometable) anon_3 " + "WHERE ROWNUM <= :param_1 + :param_2) anon_2 " + "WHERE ora_rn > :param_2) anon_1", ) c = s2.compile(dialect=oracle.OracleDialect()) eq_(len(c._result_columns), 2) diff --git a/test/dialect/oracle/test_types.py b/test/dialect/oracle/test_types.py index ae673b1fae..af9719130a 100644 --- a/test/dialect/oracle/test_types.py +++ b/test/dialect/oracle/test_types.py @@ -251,20 +251,22 @@ class TypesTest(fixtures.TestBase): def test_rowid(self): metadata = self.metadata t = Table("t1", metadata, Column("x", Integer)) - t.create() - t.insert().execute(x=5) - s1 = select([t]) - s2 = select([column("rowid")]).select_from(s1) - rowid = s2.scalar() - - # the ROWID type is not really needed here, - # as cx_oracle just treats it as a string, - # but we want to make sure the ROWID works... - rowid_col = column("rowid", oracle.ROWID) - s3 = select([t.c.x, rowid_col]).where( - rowid_col == cast(rowid, oracle.ROWID) - ) - eq_(s3.select().execute().fetchall(), [(5, rowid)]) + + with testing.db.begin() as conn: + t.create(conn) + conn.execute(t.insert(), {"x": 5}) + s1 = select([t]).subquery() + s2 = select([column("rowid")]).select_from(s1) + rowid = conn.scalar(s2) + + # the ROWID type is not really needed here, + # as cx_oracle just treats it as a string, + # but we want to make sure the ROWID works... + rowid_col = column("rowid", oracle.ROWID) + s3 = select([t.c.x, rowid_col]).where( + rowid_col == cast(rowid, oracle.ROWID) + ) + eq_(conn.execute(s3).fetchall(), [(5, rowid)]) @testing.fails_on( "+zxjdbc", "Not yet known how to pass values of the " "INTERVAL type" diff --git a/test/engine/test_reflection.py b/test/engine/test_reflection.py index 8830033b33..3dc3a4461c 100644 --- a/test/engine/test_reflection.py +++ b/test/engine/test_reflection.py @@ -906,7 +906,7 @@ class ReflectionTest(fixtures.TestBase, ComparesTables): autoload=True, ) u2 = Table("users", meta2, autoload=True) - s = sa.select([a2]) + s = sa.select([a2]).subquery() assert s.c.user_id is not None assert len(a2.foreign_keys) == 1 @@ -931,7 +931,7 @@ class ReflectionTest(fixtures.TestBase, ComparesTables): Column("user_id", sa.Integer, sa.ForeignKey("users.id")), autoload=True, ) - s = sa.select([a2]) + s = sa.select([a2]).subquery() assert s.c.user_id is not None assert len(a2.foreign_keys) == 1 diff --git a/test/orm/inheritance/test_relationship.py b/test/orm/inheritance/test_relationship.py index 3c121e06be..93396c73b0 100644 --- a/test/orm/inheritance/test_relationship.py +++ b/test/orm/inheritance/test_relationship.py @@ -2278,8 +2278,8 @@ class MultipleAdaptUsesEntityOverTableTest( is_(c_adapted.element, c) ctod = q._from_obj[0].right - ad_adapted = ctod.left - d_adapted = ctod.right + ad_adapted = ctod.element.left + d_adapted = ctod.element.right is_(ad_adapted.element, a) is_(d_adapted.element, d) diff --git a/test/orm/test_deprecations.py b/test/orm/test_deprecations.py index 83fdf60449..8a858583b1 100644 --- a/test/orm/test_deprecations.py +++ b/test/orm/test_deprecations.py @@ -1,4 +1,5 @@ import sqlalchemy as sa +from sqlalchemy import and_ from sqlalchemy import event from sqlalchemy import exc from sqlalchemy import func @@ -938,6 +939,42 @@ class DeprecatedQueryTest(_fixtures.FixtureTest, AssertsCompiledSQL): self.assert_sql_count(testing.db, go, 1) + def test_onclause_conditional_adaption(self): + Item, Order, orders, order_items, User = ( + self.classes.Item, + self.classes.Order, + self.tables.orders, + self.tables.order_items, + self.classes.User, + ) + + sess = Session() + + oalias = orders.select() + + with self._expect_implicit_subquery(): + self.assert_compile( + sess.query(User) + .join(oalias, User.orders) + .join( + Item, + and_( + Order.id == order_items.c.order_id, + order_items.c.item_id == Item.id, + ), + from_joinpoint=True, + ), + "SELECT users.id AS users_id, users.name AS users_name " + "FROM users JOIN " + "(SELECT orders.id AS id, orders.user_id AS user_id, " + "orders.address_id AS address_id, orders.description " + "AS description, orders.isopen AS isopen FROM orders) " + "AS anon_1 ON users.id = anon_1.user_id JOIN items " + "ON anon_1.id = order_items.order_id " + "AND order_items.item_id = items.id", + use_default_dialect=True, + ) + class DeprecatedInhTest(_poly_fixtures._Polymorphic): def test_with_polymorphic(self): @@ -951,6 +988,43 @@ class DeprecatedInhTest(_poly_fixtures._Polymorphic): sa.inspect(p_poly).selectable.compare(select([Person]).subquery()) ) + def test_multiple_adaption(self): + """test that multiple filter() adapters get chained together " + and work correctly within a multiple-entry join().""" + + Company = _poly_fixtures.Company + Machine = _poly_fixtures.Machine + Engineer = _poly_fixtures.Engineer + + people = self.tables.people + engineers = self.tables.engineers + machines = self.tables.machines + + sess = create_session() + + mach_alias = machines.select() + with DeprecatedQueryTest._expect_implicit_subquery(): + self.assert_compile( + sess.query(Company) + .join(people.join(engineers), Company.employees) + .join(mach_alias, Engineer.machines, from_joinpoint=True) + .filter(Engineer.name == "dilbert") + .filter(Machine.name == "foo"), + "SELECT companies.company_id AS companies_company_id, " + "companies.name AS companies_name " + "FROM companies JOIN (people " + "JOIN engineers ON people.person_id = " + "engineers.person_id) ON companies.company_id = " + "people.company_id JOIN " + "(SELECT machines.machine_id AS machine_id, " + "machines.name AS name, " + "machines.engineer_id AS engineer_id " + "FROM machines) AS anon_1 " + "ON engineers.person_id = anon_1.engineer_id " + "WHERE people.name = :name_1 AND anon_1.name = :name_2", + use_default_dialect=True, + ) + class DeprecatedMapperTest(_fixtures.FixtureTest, AssertsCompiledSQL): __dialect__ = "default" diff --git a/test/orm/test_froms.py b/test/orm/test_froms.py index 51a7e9aa5c..774f335f70 100644 --- a/test/orm/test_froms.py +++ b/test/orm/test_froms.py @@ -2532,7 +2532,7 @@ class SelectFromTest(QueryTest, AssertsCompiledSQL): User(name="jack", addresses=[Address(id=1)]), ) - def test_select_from_aliased(self): + def test_select_from_aliased_one(self): User, users = self.classes.User, self.tables.users mapper(User, users) @@ -2550,6 +2550,40 @@ class SelectFromTest(QueryTest, AssertsCompiledSQL): ) eq_(q.all(), [("chuck",), ("ed",), ("fred",), ("jack",)]) + def test_select_from_aliased_two(self): + User, users = self.classes.User, self.tables.users + + mapper(User, users) + + sess = create_session() + + ua = aliased(User) + + q = sess.query(User.name).select_entity_from(ua).order_by(User.name) + self.assert_compile( + q, + "SELECT users_1.name AS users_1_name FROM users AS users_1 " + "ORDER BY users_1.name", + ) + eq_(q.all(), [("chuck",), ("ed",), ("fred",), ("jack",)]) + + def test_select_from_core_alias_one(self): + User, users = self.classes.User, self.tables.users + + mapper(User, users) + + sess = create_session() + + ua = users.alias() + + q = sess.query(User.name).select_entity_from(ua).order_by(User.name) + self.assert_compile( + q, + "SELECT users_1.name AS users_1_name FROM users AS users_1 " + "ORDER BY users_1.name", + ) + eq_(q.all(), [("chuck",), ("ed",), ("fred",), ("jack",)]) + def test_differentiate_self_external(self): """test some different combinations of joining a table to a subquery of itself.""" diff --git a/test/orm/test_joins.py b/test/orm/test_joins.py index 66b69f6acc..24c2b6d1b1 100644 --- a/test/orm/test_joins.py +++ b/test/orm/test_joins.py @@ -499,13 +499,13 @@ class InheritedJoinTest(fixtures.MappedTest, AssertsCompiledSQL): use_default_dialect=True, ) - mach_alias = machines.select() + mach_alias = aliased(Machine, machines.select().subquery()) self.assert_compile( sess.query(Company) .join(people.join(engineers), Company.employees) .join(mach_alias, Engineer.machines, from_joinpoint=True) .filter(Engineer.name == "dilbert") - .filter(Machine.name == "foo"), + .filter(mach_alias.name == "foo"), "SELECT companies.company_id AS companies_company_id, " "companies.name AS companies_name " "FROM companies JOIN (people " @@ -1687,17 +1687,16 @@ class JoinTest(QueryTest, AssertsCompiledSQL): use_default_dialect=True, ) - oalias = orders.select() + oalias = aliased(Order, orders.select().subquery()) self.assert_compile( sess.query(User) .join(oalias, User.orders) .join( Item, and_( - Order.id == order_items.c.order_id, + oalias.id == order_items.c.order_id, order_items.c.item_id == Item.id, ), - from_joinpoint=True, ), "SELECT users.id AS users_id, users.name AS users_name " "FROM users JOIN " @@ -4104,6 +4103,7 @@ class JoinLateralTest(fixtures.MappedTest, AssertsCompiledSQL): subq = ( select([Book.book_id]) .where(Person.people_id == Book.book_owner_id) + .subquery() .lateral() ) @@ -4137,6 +4137,7 @@ class JoinLateralTest(fixtures.MappedTest, AssertsCompiledSQL): select([Book.book_id]) .correlate(Person) .where(Person.people_id == Book.book_owner_id) + .subquery() .lateral() ) diff --git a/test/orm/test_of_type.py b/test/orm/test_of_type.py index ce1c9548cf..53a759c443 100644 --- a/test/orm/test_of_type.py +++ b/test/orm/test_of_type.py @@ -287,7 +287,7 @@ class PolymorphicUnionsTest(_PolymorphicTestBase, _PolymorphicUnions): "CAST(NULL AS VARCHAR(50)) AS primary_language, " "managers.manager_name AS manager_name FROM people " "JOIN managers ON people.person_id = managers.person_id) " - "AS anon_1 ON companies.company_id = anon_1.company_id" + "AS pjoin_1 ON companies.company_id = pjoin_1.company_id" ) @@ -309,8 +309,8 @@ class PolymorphicAliasedJoinsTest( "FROM people LEFT OUTER JOIN engineers " "ON people.person_id = engineers.person_id " "LEFT OUTER JOIN managers " - "ON people.person_id = managers.person_id) AS anon_1 " - "ON companies.company_id = anon_1.people_company_id" + "ON people.person_id = managers.person_id) AS pjoin_1 " + "ON companies.company_id = pjoin_1.people_company_id" ) diff --git a/test/orm/test_query.py b/test/orm/test_query.py index 7e015b73ec..cc1f79413e 100644 --- a/test/orm/test_query.py +++ b/test/orm/test_query.py @@ -1910,9 +1910,11 @@ class ExpressionTest(QueryTest, AssertsCompiledSQL): q1 = s.query(User).filter(User.name == "ed") self.assert_compile( - select([q1]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " - "users.name AS users_name FROM users WHERE users.name = :name_1)", + select([q1.with_labels().subquery()]), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " + "users.name AS users_name " + "FROM users WHERE users.name = :name_1) AS anon_1", ) def test_join(self): @@ -1942,9 +1944,11 @@ class ExpressionTest(QueryTest, AssertsCompiledSQL): q1 = s.query(User.id, User.name).group_by(User.name) self.assert_compile( - select([q1]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " - "users.name AS users_name FROM users GROUP BY users.name)", + select([q1.with_labels().subquery()]), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " + "users.name AS users_name FROM users GROUP BY users.name) " + "AS anon_1", ) def test_group_by_append(self): @@ -1955,10 +1959,11 @@ class ExpressionTest(QueryTest, AssertsCompiledSQL): # test append something to group_by self.assert_compile( - select([q1.group_by(User.id)]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " + select([q1.group_by(User.id).with_labels().subquery()]), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " "users.name AS users_name FROM users " - "GROUP BY users.name, users.id)", + "GROUP BY users.name, users.id) AS anon_1", ) def test_group_by_cancellation(self): @@ -1968,16 +1973,20 @@ class ExpressionTest(QueryTest, AssertsCompiledSQL): q1 = s.query(User.id, User.name).group_by(User.name) # test cancellation by using None, replacement with something else self.assert_compile( - select([q1.group_by(None).group_by(User.id)]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " - "users.name AS users_name FROM users GROUP BY users.id)", + select( + [q1.group_by(None).group_by(User.id).with_labels().subquery()] + ), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " + "users.name AS users_name FROM users GROUP BY users.id) AS anon_1", ) # test cancellation by using None, replacement with nothing self.assert_compile( - select([q1.group_by(None)]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " - "users.name AS users_name FROM users)", + select([q1.group_by(None).with_labels().subquery()]), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " + "users.name AS users_name FROM users) AS anon_1", ) def test_group_by_cancelled_still_present(self): @@ -1994,9 +2003,11 @@ class ExpressionTest(QueryTest, AssertsCompiledSQL): q1 = s.query(User.id, User.name).order_by(User.name) self.assert_compile( - select([q1]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " - "users.name AS users_name FROM users ORDER BY users.name)", + select([q1.with_labels().subquery()]), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " + "users.name AS users_name FROM users ORDER BY users.name) " + "AS anon_1", ) def test_order_by_append(self): @@ -2007,10 +2018,11 @@ class ExpressionTest(QueryTest, AssertsCompiledSQL): # test append something to order_by self.assert_compile( - select([q1.order_by(User.id)]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " + select([q1.order_by(User.id).with_labels().subquery()]), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " "users.name AS users_name FROM users " - "ORDER BY users.name, users.id)", + "ORDER BY users.name, users.id) AS anon_1", ) def test_order_by_cancellation(self): @@ -2020,16 +2032,20 @@ class ExpressionTest(QueryTest, AssertsCompiledSQL): q1 = s.query(User.id, User.name).order_by(User.name) # test cancellation by using None, replacement with something else self.assert_compile( - select([q1.order_by(None).order_by(User.id)]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " - "users.name AS users_name FROM users ORDER BY users.id)", + select( + [q1.order_by(None).order_by(User.id).with_labels().subquery()] + ), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " + "users.name AS users_name FROM users ORDER BY users.id) AS anon_1", ) # test cancellation by using None, replacement with nothing self.assert_compile( - select([q1.order_by(None)]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " - "users.name AS users_name FROM users)", + select([q1.order_by(None).with_labels().subquery()]), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " + "users.name AS users_name FROM users) AS anon_1", ) def test_order_by_cancellation_false(self): @@ -2039,16 +2055,20 @@ class ExpressionTest(QueryTest, AssertsCompiledSQL): q1 = s.query(User.id, User.name).order_by(User.name) # test cancellation by using None, replacement with something else self.assert_compile( - select([q1.order_by(False).order_by(User.id)]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " - "users.name AS users_name FROM users ORDER BY users.id)", + select( + [q1.order_by(False).order_by(User.id).with_labels().subquery()] + ), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " + "users.name AS users_name FROM users ORDER BY users.id) AS anon_1", ) # test cancellation by using None, replacement with nothing self.assert_compile( - select([q1.order_by(False)]), - "SELECT users_id, users_name FROM (SELECT users.id AS users_id, " - "users.name AS users_name FROM users)", + select([q1.order_by(False).with_labels().subquery()]), + "SELECT anon_1.users_id, anon_1.users_name FROM " + "(SELECT users.id AS users_id, " + "users.name AS users_name FROM users) AS anon_1", ) def test_order_by_cancelled_allows_assertions(self): @@ -3199,6 +3219,11 @@ class SetOpsTest(QueryTest, AssertsCompiledSQL): [User(name="ed"), User(name="fred"), User(name="jack")], ) + eq_( + fred.union(ed).union(jack).order_by(User.name).all(), + [User(name="ed"), User(name="fred"), User(name="jack")], + ) + def test_statement_labels(self): """test that label conflicts don't occur with joins etc.""" diff --git a/test/sql/test_compare.py b/test/sql/test_compare.py index f9decc58a4..d48a8ed338 100644 --- a/test/sql/test_compare.py +++ b/test/sql/test_compare.py @@ -48,6 +48,7 @@ from sqlalchemy.sql.functions import ReturnTypeFromArgs from sqlalchemy.sql.selectable import _OffsetLimitParam from sqlalchemy.sql.selectable import FromGrouping from sqlalchemy.sql.selectable import Selectable +from sqlalchemy.sql.selectable import SelectStatementGrouping from sqlalchemy.testing import assert_raises_message from sqlalchemy.testing import eq_ from sqlalchemy.testing import fixtures @@ -242,6 +243,10 @@ class CompareAndCopyTest(fixtures.TestBase): FromGrouping(table_a.alias("a")), FromGrouping(table_a.alias("b")), ), + lambda: ( + SelectStatementGrouping(select([table_a])), + SelectStatementGrouping(select([table_b])), + ), lambda: ( select([table_a.c.a]).scalar_subquery(), select([table_a.c.a]).where(table_a.c.b == 5).scalar_subquery(), diff --git a/test/sql/test_compiler.py b/test/sql/test_compiler.py index e012c2713e..ea1a9bd75a 100644 --- a/test/sql/test_compiler.py +++ b/test/sql/test_compiler.py @@ -46,8 +46,8 @@ from sqlalchemy import select from sqlalchemy import Sequence from sqlalchemy import sql from sqlalchemy import String -from sqlalchemy import subquery from sqlalchemy import Table +from sqlalchemy import testing from sqlalchemy import Text from sqlalchemy import text from sqlalchemy import TIMESTAMP @@ -148,43 +148,39 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): def test_attribute_sanity(self): assert hasattr(table1, "c") - assert hasattr(table1.select(), "c") + assert hasattr(table1.select().subquery(), "c") assert not hasattr(table1.c.myid.self_group(), "columns") - assert hasattr(table1.select().self_group(), "columns") assert not hasattr(table1.c.myid, "columns") assert not hasattr(table1.c.myid, "c") - assert not hasattr(table1.select().c.myid, "c") - assert not hasattr(table1.select().c.myid, "columns") + assert not hasattr(table1.select().subquery().c.myid, "c") + assert not hasattr(table1.select().subquery().c.myid, "columns") assert not hasattr(table1.alias().c.myid, "columns") assert not hasattr(table1.alias().c.myid, "c") - if util.compat.py32: - assert_raises_message( - exc.InvalidRequestError, - "Scalar Select expression has no " - "columns; use this object directly within a " - "column-level expression.", - lambda: hasattr( - select([table1.c.myid]).scalar_subquery().self_group(), - "columns", - ), - ) - assert_raises_message( - exc.InvalidRequestError, - "Scalar Select expression has no " - "columns; use this object directly within a " - "column-level expression.", - lambda: hasattr( - select([table1.c.myid]).scalar_subquery(), "columns" - ), - ) - else: - assert not hasattr( - select([table1.c.myid]).scalar_subquery().self_group(), - "columns", - ) - assert not hasattr( - select([table1.c.myid]).scalar_subquery(), "columns" - ) + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns attributes are " + "deprecated" + ): + assert hasattr(table1.select(), "c") + + assert_raises_message( + exc.InvalidRequestError, + "Scalar Select expression has no " + "columns; use this object directly within a " + "column-level expression.", + getattr, + select([table1.c.myid]).scalar_subquery().self_group(), + "columns", + ) + + assert_raises_message( + exc.InvalidRequestError, + "Scalar Select expression has no " + "columns; use this object directly within a " + "column-level expression.", + getattr, + select([table1.c.myid]).scalar_subquery(), + "columns", + ) def test_prefix_constructor(self): class Pref(HasPrefixes): @@ -370,23 +366,24 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): another select, for the purposes of selecting from the exported columns of that select.""" - s = select([table1], table1.c.name == "jack") + s = select([table1], table1.c.name == "jack").subquery() self.assert_compile( select([s], s.c.myid == 7), - "SELECT myid, name, description FROM " + "SELECT anon_1.myid, anon_1.name, anon_1.description FROM " "(SELECT mytable.myid AS myid, " "mytable.name AS name, mytable.description AS description " "FROM mytable " - "WHERE mytable.name = :name_1) WHERE myid = :myid_1", + "WHERE mytable.name = :name_1) AS anon_1 WHERE " + "anon_1.myid = :myid_1", ) sq = select([table1]) self.assert_compile( - sq.select(), - "SELECT myid, name, description FROM " + sq.subquery().select(), + "SELECT anon_1.myid, anon_1.name, anon_1.description FROM " "(SELECT mytable.myid AS myid, " "mytable.name AS name, mytable.description " - "AS description FROM mytable)", + "AS description FROM mytable) AS anon_1", ) sq = select([table1]).alias("sq") @@ -482,6 +479,15 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): "keyed_y, keyed.z AS keyed_z FROM keyed", ) + self.assert_compile( + select([select([keyed]).apply_labels().subquery()]).apply_labels(), + "SELECT anon_1.keyed_x AS anon_1_keyed_x, " + "anon_1.keyed_y AS anon_1_keyed_y, " + "anon_1.keyed_z AS anon_1_keyed_z " + "FROM (SELECT keyed.x AS keyed_x, keyed.y AS keyed_y, " + "keyed.z AS keyed_z FROM keyed) AS anon_1", + ) + def test_paramstyles(self): stmt = text("select :foo, :bar, :bat from sometable") @@ -687,36 +693,6 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): "EXISTS (SELECT * FROM myothertable WHERE " "myothertable.otherid = mytable.myid)", ) - self.assert_compile( - table1.select( - exists() - .where(table2.c.otherid == table1.c.myid) - .correlate(table1) - ).replace_selectable(table2, table2.alias()), - "SELECT mytable.myid, mytable.name, " - "mytable.description FROM mytable WHERE " - "EXISTS (SELECT * FROM myothertable AS " - "myothertable_1 WHERE myothertable_1.otheri" - "d = mytable.myid)", - ) - self.assert_compile( - table1.select( - exists() - .where(table2.c.otherid == table1.c.myid) - .correlate(table1) - ) - .select_from( - table1.join(table2, table1.c.myid == table2.c.otherid) - ) - .replace_selectable(table2, table2.alias()), - "SELECT mytable.myid, mytable.name, " - "mytable.description FROM mytable JOIN " - "myothertable AS myothertable_1 ON " - "mytable.myid = myothertable_1.otherid " - "WHERE EXISTS (SELECT * FROM myothertable " - "AS myothertable_1 WHERE " - "myothertable_1.otherid = mytable.myid)", - ) self.assert_compile( select( @@ -762,6 +738,18 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): "addresses, users WHERE addresses.user_id = " "users.user_id) AS s", ) + self.assert_compile( + table1.select( + table1.c.myid + == select( + [table1.c.myid], table1.c.name == "jack" + ).scalar_subquery() + ), + "SELECT mytable.myid, mytable.name, " + "mytable.description FROM mytable WHERE " + "mytable.myid = (SELECT mytable.myid FROM " + "mytable WHERE mytable.name = :name_1)", + ) self.assert_compile( table1.select( table1.c.myid @@ -784,9 +772,9 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): "myothertable.otherid = mytable.myid)", ) talias = table1.alias("ta") - s = subquery( - "sq2", [talias], exists([1], table2.c.otherid == talias.c.myid) - ) + s = select( + [talias], exists([1], table2.c.otherid == talias.c.myid) + ).subquery("sq2") self.assert_compile( select([s, table1]), "SELECT sq2.myid, sq2.name, " @@ -816,27 +804,33 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): def test_orderby_subquery(self): self.assert_compile( - table1.select( - order_by=[ + table1.select().order_by( + select( + [table2.c.otherid], table1.c.myid == table2.c.otherid + ).scalar_subquery() + ), + "SELECT mytable.myid, mytable.name, " + "mytable.description FROM mytable ORDER BY " + "(SELECT myothertable.otherid FROM " + "myothertable WHERE mytable.myid = " + "myothertable.otherid)", + ) + self.assert_compile( + table1.select().order_by( + desc( select( [table2.c.otherid], table1.c.myid == table2.c.otherid ).scalar_subquery() - ] + ) ), "SELECT mytable.myid, mytable.name, " "mytable.description FROM mytable ORDER BY " "(SELECT myothertable.otherid FROM " "myothertable WHERE mytable.myid = " - "myothertable.otherid)", + "myothertable.otherid) DESC", ) def test_scalar_select(self): - - self.assert_compile( - func.coalesce(select([table1.c.myid]).scalar_subquery()), - "coalesce((SELECT mytable.myid FROM mytable))", - ) - s = select([table1.c.myid], correlate=False).scalar_subquery() self.assert_compile( select([table1, s]), @@ -878,9 +872,10 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): "mytable)", ) self.assert_compile( - select([table1.c.myid]).where(s > table1.c.myid), + select([table1.c.myid]).where(table1.c.myid < s), "SELECT mytable.myid FROM mytable WHERE " - "(SELECT mytable.myid FROM mytable) > mytable.myid", + "mytable.myid < (SELECT mytable.myid FROM " + "mytable)", ) s = select([table1.c.myid]).scalar_subquery() self.assert_compile( @@ -917,22 +912,18 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): # 'columns' attribute s = select([table1.c.myid]).scalar_subquery() - try: - s.c.foo - except exc.InvalidRequestError as err: - assert ( - str(err) - == "Scalar Select expression has no columns; use this " - "object directly within a column-level expression." - ) - try: - s.columns.foo - except exc.InvalidRequestError as err: - assert ( - str(err) - == "Scalar Select expression has no columns; use this " - "object directly within a column-level expression." - ) + assert_raises_message( + exc.InvalidRequestError, + "Scalar Select expression has no columns; use this " + "object directly within a column-level expression.", + lambda: s.c.foo, + ) + assert_raises_message( + exc.InvalidRequestError, + "Scalar Select expression has no columns; use this " + "object directly within a column-level expression.", + lambda: s.columns.foo, + ) zips = table( "zips", column("zipcode"), column("latitude"), column("longitude") @@ -1542,7 +1533,7 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): # make an alias of the "selectable". column names # stay the same (i.e. the labels), table name "changes" to "t2view". - a = alias(q, "t2view") + a = q.alias("t2view") # select from that alias, also using labels. two levels of labels # should produce two underscores. @@ -1564,6 +1555,50 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): "WHERE t2view.mytable_myid = :mytable_myid_1", ) + def test_alias_nesting_table(self): + self.assert_compile( + select([table1.alias("foo").alias("bar").alias("bat")]), + "SELECT bat.myid, bat.name, bat.description FROM mytable AS bat", + ) + + self.assert_compile( + select([table1.alias(None).alias("bar").alias("bat")]), + "SELECT bat.myid, bat.name, bat.description FROM mytable AS bat", + ) + + self.assert_compile( + select([table1.alias("foo").alias(None).alias("bat")]), + "SELECT bat.myid, bat.name, bat.description FROM mytable AS bat", + ) + + self.assert_compile( + select([table1.alias("foo").alias("bar").alias(None)]), + "SELECT bar_1.myid, bar_1.name, bar_1.description " + "FROM mytable AS bar_1", + ) + + self.assert_compile( + select([table1.alias("foo").alias(None).alias(None)]), + "SELECT anon_1.myid, anon_1.name, anon_1.description " + "FROM mytable AS anon_1", + ) + + def test_alias_nesting_subquery(self): + stmt = select([table1]).subquery() + self.assert_compile( + select([stmt.alias("foo").alias("bar").alias("bat")]), + "SELECT bat.myid, bat.name, bat.description FROM " + "(SELECT mytable.myid AS myid, mytable.name AS name, " + "mytable.description AS description FROM mytable) AS bat", + ) + + self.assert_compile( + select([stmt.alias("foo").alias(None).alias(None)]), + "SELECT anon_1.myid, anon_1.name, anon_1.description FROM " + "(SELECT mytable.myid AS myid, mytable.name AS name, " + "mytable.description AS description FROM mytable) AS anon_1", + ) + def test_prefix(self): self.assert_compile( table1.select() @@ -1894,7 +1929,7 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): "mytable.myid = :myid_1 UNION " "SELECT mytable.myid, mytable.name, mytable.description " "FROM mytable WHERE mytable.myid = :myid_2 " - "ORDER BY mytable.myid", + "ORDER BY myid", ) x = union(select([table1]), select([table1])) @@ -1911,17 +1946,18 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): select([table1.c.myid, table1.c.name]), select([table2]), select([table3]), - ) + ).order_by("name") self.assert_compile( u1, "SELECT mytable.myid, mytable.name " "FROM mytable UNION SELECT myothertable.otherid, " "myothertable.othername FROM myothertable " "UNION SELECT thirdtable.userid, thirdtable.otherstuff " - "FROM thirdtable", + "FROM thirdtable ORDER BY name", ) - assert u1.corresponding_column(table2.c.otherid) is u1.c.myid + u1s = u1.subquery() + assert u1s.corresponding_column(table2.c.otherid) is u1s.c.myid self.assert_compile( union( @@ -1931,14 +1967,32 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): offset=10, limit=5, ), + # note table name is omitted here. The CompoundSelect, inside of + # _label_resolve_dict(), creates a subquery of itself and then + # turns "named_with_column" off, so that we can order by the + # "myid" name as relative to the CompoundSelect itself without it + # having a name. "SELECT mytable.myid, mytable.name " "FROM mytable UNION SELECT myothertable.otherid, " "myothertable.othername " - "FROM myothertable ORDER BY myid " # note table name is omitted + "FROM myothertable ORDER BY myid " "LIMIT :param_1 OFFSET :param_2", {"param_1": 5, "param_2": 10}, ) + # these tests are mostly in test_text, however adding one here + # to check the special thing CompoundSelect does with labels + assert_raises_message( + exc.CompileError, + "Can't resolve label reference for ORDER BY / GROUP BY. Textual " + "SQL expression 'noname'", + union( + select([table1.c.myid, table1.c.name]), + select([table2]), + order_by=["noname"], + ).compile, + ) + self.assert_compile( union( select( @@ -2098,7 +2152,9 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): ) self.assert_compile( - union(s.limit(1).alias("a"), s.limit(2).alias("b")), + union( + s.limit(1).alias("a").element, s.limit(2).alias("b").element + ), "(SELECT foo, bar FROM bat LIMIT :param_1) " "UNION (SELECT foo, bar FROM bat LIMIT :param_2)", ) @@ -2382,7 +2438,7 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): s = select([t, literal("lala").label("hoho")]) self.assert_compile(s, "SELECT foo.id, :param_1 AS hoho FROM foo") - assert [str(c) for c in s.c] == ["id", "hoho"] + assert [str(c) for c in s.subquery().c] == ["anon_1.id", "anon_1.hoho"] def test_bind_callable(self): expr = column("x") == bindparam("key", callable_=lambda: 12) @@ -2864,7 +2920,7 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): assert_raises_message( exc.InvalidRequestError, "Cannot initialize a sub-selectable with this Column", - lambda: sel1.c, + lambda: sel1.subquery().c, ) # calling label or scalar_subquery doesn't compile @@ -2910,7 +2966,7 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): ] ) - eq_(list(s1.c.keys()), ["myid", "foobar", str(f1), "gg"]) + eq_(list(s1.subquery().c.keys()), ["myid", "foobar", str(f1), "gg"]) meta = MetaData() t1 = Table("mytable", meta, Column("col1", Integer)) @@ -2946,7 +3002,7 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): t = table1 s1 = select([col], from_obj=t) - assert list(s1.c.keys()) == [key], list(s1.c.keys()) + assert list(s1.subquery().c.keys()) == [key], list(s1.c.keys()) if lbl: self.assert_compile( @@ -2955,24 +3011,26 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): else: self.assert_compile(s1, "SELECT %s FROM mytable" % (expr,)) - s1 = select([s1]) + s1 = select([s1.subquery()]) if lbl: + alias_ = "anon_2" if lbl == "anon_1" else "anon_1" self.assert_compile( s1, - "SELECT %s FROM (SELECT %s AS %s FROM mytable)" - % (lbl, expr, lbl), + "SELECT %s.%s FROM (SELECT %s AS %s FROM mytable) AS %s" + % (alias_, lbl, expr, lbl, alias_), ) elif col.table is not None: # sqlite rule labels subquery columns self.assert_compile( s1, - "SELECT %s FROM (SELECT %s AS %s FROM mytable)" - % (key, expr, key), + "SELECT anon_1.%s FROM (SELECT %s AS %s FROM mytable) " + "AS anon_1" % (key, expr, key), ) else: self.assert_compile( s1, - "SELECT %s FROM (SELECT %s FROM mytable)" % (expr, expr), + "SELECT anon_1.%s FROM (SELECT %s FROM mytable) AS anon_1" + % (expr, expr), ) def test_hints(self): @@ -2993,7 +3051,7 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): table1.join(table2, table1.c.myid == table2.c.otherid) ) .with_hint(table1, "hint1") - ) + ).subquery() s4 = ( select([table3]) @@ -3068,42 +3126,42 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): mysql_d, "SELECT thirdtable.userid, thirdtable.otherstuff " "FROM thirdtable " - "hint3 INNER JOIN (SELECT mytable.myid, mytable.name, " - "mytable.description, myothertable.otherid, " - "myothertable.othername FROM mytable hint1 INNER " - "JOIN myothertable ON mytable.myid = myothertable.otherid) " - "ON othername = thirdtable.otherstuff", + "hint3 INNER JOIN (SELECT mytable.myid AS myid, " + "mytable.name AS name, " + "mytable.description AS description, " + "myothertable.otherid AS otherid, " + "myothertable.othername AS othername FROM mytable hint1 INNER " + "JOIN myothertable ON " + "mytable.myid = myothertable.otherid) AS anon_1 " + "ON anon_1.othername = thirdtable.otherstuff", ), ( s4, sybase_d, "SELECT thirdtable.userid, thirdtable.otherstuff " "FROM thirdtable " - "hint3 JOIN (SELECT mytable.myid, mytable.name, " - "mytable.description, myothertable.otherid, " - "myothertable.othername FROM mytable hint1 " - "JOIN myothertable ON mytable.myid = myothertable.otherid) " - "ON othername = thirdtable.otherstuff", + "hint3 JOIN (SELECT mytable.myid AS myid, " + "mytable.name AS name, " + "mytable.description AS description, " + "myothertable.otherid AS otherid, " + "myothertable.othername AS othername FROM mytable hint1 " + "JOIN myothertable ON " + "mytable.myid = myothertable.otherid) AS anon_1 " + "ON anon_1.othername = thirdtable.otherstuff", ), ( s4, oracle_d, "SELECT /*+ hint3 */ thirdtable.userid, thirdtable.otherstuff " - "FROM thirdtable JOIN (SELECT /*+ hint1 */ mytable.myid," - " mytable.name, mytable.description, myothertable.otherid," - " myothertable.othername FROM mytable JOIN myothertable ON" - " mytable.myid = myothertable.otherid) ON othername =" - " thirdtable.otherstuff", + "FROM thirdtable JOIN (SELECT /*+ hint1 */ " + "mytable.myid AS myid," + " mytable.name AS name, mytable.description AS description, " + "myothertable.otherid AS otherid," + " myothertable.othername AS othername " + "FROM mytable JOIN myothertable ON " + "mytable.myid = myothertable.otherid) anon_1 ON " + "anon_1.othername = thirdtable.otherstuff", ), - # TODO: figure out dictionary ordering solution here - # (s5, oracle_d, - # "SELECT /*+ hint3 */ /*+ hint1 */ thirdtable.userid, " - # "thirdtable.otherstuff " - # "FROM thirdtable JOIN (SELECT mytable.myid," - # " mytable.name, mytable.description, myothertable.otherid," - # " myothertable.othername FROM mytable JOIN myothertable ON" - # " mytable.myid = myothertable.otherid) ON othername =" - # " thirdtable.otherstuff"), ( s6, oracle_d, @@ -3798,12 +3856,14 @@ class CorrelateTest(fixtures.TestBase, AssertsCompiledSQL): def test_dont_overcorrelate(self): self.assert_compile( - select([table1], from_obj=[table1, table1.select()]), + select([table1]) + .select_from(table1) + .select_from(table1.select().subquery()), "SELECT mytable.myid, mytable.name, " "mytable.description FROM mytable, (SELECT " "mytable.myid AS myid, mytable.name AS " "name, mytable.description AS description " - "FROM mytable)", + "FROM mytable) AS anon_1", ) def _fixture(self): @@ -4052,9 +4112,9 @@ class CorrelateTest(fixtures.TestBase, AssertsCompiledSQL): def test_correlate_from_all_ok(self): t1, t2, s1 = self._fixture() self.assert_compile( - select([t1, t2, s1]), - "SELECT t1.a, t2.a, a FROM t1, t2, " - "(SELECT t1.a AS a FROM t1, t2 WHERE t1.a = t2.a)", + select([t1, t2, s1.subquery()]), + "SELECT t1.a, t2.a, anon_1.a FROM t1, t2, " + "(SELECT t1.a AS a FROM t1, t2 WHERE t1.a = t2.a) AS anon_1", ) def test_correlate_auto_where_singlefrom(self): @@ -4118,7 +4178,8 @@ class CorrelateTest(fixtures.TestBase, AssertsCompiledSQL): .order_by(c.c.pos) .limit(1) ) - s = s.correlate(p) + s = s.correlate(p).subquery() + s = exists().select_from(s).where(s.c.id == 1) s = select([p]).where(s) self.assert_compile( @@ -4126,7 +4187,8 @@ class CorrelateTest(fixtures.TestBase, AssertsCompiledSQL): "SELECT parent.id FROM parent WHERE EXISTS (SELECT * " "FROM (SELECT child.id AS id, child.parent_id AS parent_id, " "child.pos AS pos FROM child WHERE child.parent_id = parent.id " - "ORDER BY child.pos LIMIT :param_1) WHERE id = :id_1)", + "ORDER BY child.pos LIMIT :param_1) AS anon_1 " + "WHERE anon_1.id = :id_1)", ) def test_no_contextless_correlate_except(self): @@ -4174,14 +4236,15 @@ class CorrelateTest(fixtures.TestBase, AssertsCompiledSQL): t2 = table("t2", column("y")) s = select([t1.c.x]).where(t1.c.x == t2.c.y) - s2 = select([t2, s]) - s3 = select([t1, s2]) + s2 = select([t2, s.subquery()]) + s3 = select([t1, s2.subquery()]) self.assert_compile( s3, - "SELECT t1.x, y, x FROM t1, " - "(SELECT t2.y AS y, x FROM t2, " - "(SELECT t1.x AS x FROM t1, t2 WHERE t1.x = t2.y))", + "SELECT t1.x, anon_1.y, anon_1.x FROM t1, " + "(SELECT t2.y AS y, anon_2.x AS x FROM t2, " + "(SELECT t1.x AS x FROM t1, t2 WHERE t1.x = t2.y) " + "AS anon_2) AS anon_1", ) @@ -4267,7 +4330,7 @@ class ResultMapTest(fixtures.TestBase): def test_compound_not_toplevel_doesnt_populate(self): t = Table("t", MetaData(), Column("a", Integer), Column("b", Integer)) - subq = select([t]).union(select([t])) + subq = select([t]).union(select([t])).subquery() stmt = select([t.c.a]).select_from(t.join(subq, t.c.a == subq.c.a)) comp = stmt.compile() eq_( @@ -4357,7 +4420,7 @@ class ResultMapTest(fixtures.TestBase): def test_nested_api(self): from sqlalchemy.engine.result import ResultMetaData - stmt2 = select([table2]) + stmt2 = select([table2]).subquery() stmt1 = select([table1]).select_from(stmt2) @@ -4368,10 +4431,12 @@ class ResultMapTest(fixtures.TestBase): class MyCompiler(compiler.SQLCompiler): def visit_select(self, stmt, *arg, **kw): - if stmt is stmt2: + if stmt is stmt2.element: with self._nested_result() as nested: - contexts[stmt2] = nested - text = super(MyCompiler, self).visit_select(stmt2) + contexts[stmt2.element] = nested + text = super(MyCompiler, self).visit_select( + stmt2.element + ) self._add_to_result_map("k1", "k1", (1, 2, 3), int_) else: text = super(MyCompiler, self).visit_select( @@ -4381,9 +4446,8 @@ class ResultMapTest(fixtures.TestBase): return text comp = MyCompiler(default.DefaultDialect(), stmt1) - eq_( - ResultMetaData._create_result_map(contexts[stmt2][0]), + ResultMetaData._create_result_map(contexts[stmt2.element][0]), { "otherid": ( "otherid", @@ -4457,7 +4521,7 @@ class ResultMapTest(fixtures.TestBase): eq_(len(list(stmt.inner_columns)), 7) # but only exposes 5 of them, the other two are dupes of x and y - eq_(len(stmt.c), 5) + eq_(len(stmt.subquery().c), 5) # and when it generates a SELECT it will also render only 5 eq_(len(stmt._columns_plus_names), 5) diff --git a/test/sql/test_deprecations.py b/test/sql/test_deprecations.py index 6bf0e8962e..d126912a5d 100644 --- a/test/sql/test_deprecations.py +++ b/test/sql/test_deprecations.py @@ -1,5 +1,6 @@ #! coding: utf-8 +from sqlalchemy import alias from sqlalchemy import bindparam from sqlalchemy import Column from sqlalchemy import column @@ -8,8 +9,12 @@ from sqlalchemy import exc from sqlalchemy import ForeignKey from sqlalchemy import func from sqlalchemy import Integer +from sqlalchemy import join +from sqlalchemy import literal_column from sqlalchemy import MetaData +from sqlalchemy import null from sqlalchemy import select +from sqlalchemy import sql from sqlalchemy import String from sqlalchemy import Table from sqlalchemy import table @@ -19,8 +24,11 @@ from sqlalchemy import util from sqlalchemy.engine import default from sqlalchemy.schema import DDL from sqlalchemy.sql import coercions +from sqlalchemy.sql import quoted_name from sqlalchemy.sql import roles from sqlalchemy.sql import util as sql_util +from sqlalchemy.sql import visitors +from sqlalchemy.sql.selectable import SelectStatementGrouping from sqlalchemy.testing import assert_raises from sqlalchemy.testing import assert_raises_message from sqlalchemy.testing import AssertsCompiledSQL @@ -377,8 +385,81 @@ class ForUpdateTest(fixtures.TestBase, AssertsCompiledSQL): class SubqueryCoercionsTest(fixtures.TestBase, AssertsCompiledSQL): + __dialect__ = "default" + + table1 = table( + "mytable", + column("myid", Integer), + column("name", String), + column("description", String), + ) + + table2 = table( + "myothertable", column("otherid", Integer), column("othername", String) + ) + + def test_select_of_select(self): + stmt = select([self.table1.c.myid]) + + with testing.expect_deprecated( + r"The SelectBase.select\(\) method is deprecated and will be " + "removed" + ): + self.assert_compile( + stmt.select(), + "SELECT anon_1.myid FROM (SELECT mytable.myid AS myid " + "FROM mytable) AS anon_1", + ) + + def test_join_of_select(self): + stmt = select([self.table1.c.myid]) + + with testing.expect_deprecated( + r"The SelectBase.join\(\) method is deprecated and will be " + "removed" + ): + self.assert_compile( + stmt.join( + self.table2, self.table2.c.otherid == self.table1.c.myid + ), + # note the SQL is wrong here as the subquery now has a name. + # however, even SQLite which accepts unnamed subqueries in a + # JOIN cannot actually join with how SQLAlchemy 1.3 and + # earlier would render: + # sqlite> select myid, otherid from (select myid from mytable) + # join myothertable on mytable.myid=myothertable.otherid; + # Error: no such column: mytable.myid + # if using stmt.c.col, that fails often as well if there are + # any naming overlaps: + # sqlalchemy.exc.OperationalError: (sqlite3.OperationalError) + # ambiguous column name: id + # [SQL: SELECT id, data + # FROM (SELECT a.id AS id, a.data AS data + # FROM a) JOIN b ON b.a_id = id] + # so that shows that nobody is using this anyway + "(SELECT mytable.myid AS myid FROM mytable) AS anon_1 " + "JOIN myothertable ON myothertable.otherid = mytable.myid", + ) + + def test_outerjoin_of_select(self): + stmt = select([self.table1.c.myid]) + + with testing.expect_deprecated( + r"The SelectBase.outerjoin\(\) method is deprecated and will be " + "removed" + ): + self.assert_compile( + stmt.outerjoin( + self.table2, self.table2.c.otherid == self.table1.c.myid + ), + # note the SQL is wrong here as the subquery now has a name + "(SELECT mytable.myid AS myid FROM mytable) AS anon_1 " + "LEFT OUTER JOIN myothertable " + "ON myothertable.otherid = mytable.myid", + ) + def test_column_roles(self): - stmt = select([table1.c.myid]) + stmt = select([self.table1.c.myid]) for role in [ roles.WhereHavingRole, @@ -402,7 +483,7 @@ class SubqueryCoercionsTest(fixtures.TestBase, AssertsCompiledSQL): is_true(coerced.compare(stmt.scalar_subquery())) def test_labeled_role(self): - stmt = select([table1.c.myid]) + stmt = select([self.table1.c.myid]) with testing.expect_deprecated( "coercing SELECT object to scalar " @@ -427,7 +508,7 @@ class SubqueryCoercionsTest(fixtures.TestBase, AssertsCompiledSQL): "subquery in a column-expression context is deprecated" ): self.assert_compile( - func.coalesce(select([table1.c.myid])), + func.coalesce(select([self.table1.c.myid])), "coalesce((SELECT mytable.myid FROM mytable))", ) @@ -435,9 +516,9 @@ class SubqueryCoercionsTest(fixtures.TestBase, AssertsCompiledSQL): "coercing SELECT object to scalar " "subquery in a column-expression context is deprecated" ): - s = select([table1.c.myid]).alias() + s = select([self.table1.c.myid]).alias() self.assert_compile( - select([table1.c.myid]).where(table1.c.myid == s), + select([self.table1.c.myid]).where(self.table1.c.myid == s), "SELECT mytable.myid FROM mytable WHERE " "mytable.myid = (SELECT mytable.myid FROM " "mytable)", @@ -448,7 +529,7 @@ class SubqueryCoercionsTest(fixtures.TestBase, AssertsCompiledSQL): "subquery in a column-expression context is deprecated" ): self.assert_compile( - select([table1.c.myid]).where(s > table1.c.myid), + select([self.table1.c.myid]).where(s > self.table1.c.myid), "SELECT mytable.myid FROM mytable WHERE " "mytable.myid < (SELECT mytable.myid FROM " "mytable)", @@ -458,9 +539,9 @@ class SubqueryCoercionsTest(fixtures.TestBase, AssertsCompiledSQL): "coercing SELECT object to scalar " "subquery in a column-expression context is deprecated" ): - s = select([table1.c.myid]).alias() + s = select([self.table1.c.myid]).alias() self.assert_compile( - select([table1.c.myid]).where(table1.c.myid == s), + select([self.table1.c.myid]).where(self.table1.c.myid == s), "SELECT mytable.myid FROM mytable WHERE " "mytable.myid = (SELECT mytable.myid FROM " "mytable)", @@ -471,23 +552,35 @@ class SubqueryCoercionsTest(fixtures.TestBase, AssertsCompiledSQL): "subquery in a column-expression context is deprecated" ): self.assert_compile( - select([table1.c.myid]).where(s > table1.c.myid), + select([self.table1.c.myid]).where(s > self.table1.c.myid), "SELECT mytable.myid FROM mytable WHERE " "mytable.myid < (SELECT mytable.myid FROM " "mytable)", ) + def test_standalone_alias(self): + with testing.expect_deprecated( + "Implicit coercion of SELECT and textual SELECT constructs" + ): + stmt = alias(select([self.table1.c.myid]), "foo") + + self.assert_compile(stmt, "SELECT mytable.myid FROM mytable") + + is_true( + stmt.compare(select([self.table1.c.myid]).subquery().alias("foo")) + ) + def test_as_scalar(self): with testing.expect_deprecated( r"The SelectBase.as_scalar\(\) method is deprecated and " "will be removed in a future release." ): - stmt = select([table1.c.myid]).as_scalar() + stmt = select([self.table1.c.myid]).as_scalar() - is_true(stmt.compare(select([table1.c.myid]).scalar_subquery())) + is_true(stmt.compare(select([self.table1.c.myid]).scalar_subquery())) def test_fromclause_subquery(self): - stmt = select([table1.c.myid]) + stmt = select([self.table1.c.myid]) with testing.expect_deprecated( "Implicit coercion of SELECT and textual SELECT constructs " "into FROM clauses is deprecated" @@ -498,6 +591,93 @@ class SubqueryCoercionsTest(fixtures.TestBase, AssertsCompiledSQL): is_true(coerced.compare(stmt.subquery())) + def test_plain_fromclause_select_to_subquery(self): + with testing.expect_deprecated( + "Implicit coercion of SELECT and textual SELECT " + "constructs into FROM clauses is deprecated;" + ): + element = coercions.expect( + roles.FromClauseRole, + SelectStatementGrouping(select([self.table1])), + ) + is_true( + element.compare( + SelectStatementGrouping(select([self.table1])).subquery() + ) + ) + + def test_functions_select_method_two(self): + expr = func.rows("foo") + with testing.expect_deprecated( + "Implicit coercion of SELECT and textual SELECT constructs " + "into FROM clauses is deprecated" + ): + stmt = select(["*"]).select_from(expr.select()) + self.assert_compile( + stmt, "SELECT * FROM (SELECT rows(:rows_2) AS rows_1) AS anon_1" + ) + + def test_functions_with_cols(self): + users = table( + "users", column("id"), column("name"), column("fullname") + ) + calculate = select( + [column("q"), column("z"), column("r")], + from_obj=[ + func.calculate(bindparam("x", None), bindparam("y", None)) + ], + ) + + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns attributes are " + "deprecated and will be removed" + ): + self.assert_compile( + select([users], users.c.id > calculate.c.z), + "SELECT users.id, users.name, users.fullname " + "FROM users, (SELECT q, z, r " + "FROM calculate(:x, :y)) AS anon_1 " + "WHERE users.id > anon_1.z", + ) + + +class LateralSubqueryCoercionsTest(fixtures.TablesTest, AssertsCompiledSQL): + __dialect__ = default.DefaultDialect(supports_native_boolean=True) + + run_setup_bind = None + + run_create_tables = None + + @classmethod + def define_tables(cls, metadata): + Table( + "people", + metadata, + Column("people_id", Integer, primary_key=True), + Column("age", Integer), + Column("name", String(30)), + ) + Table( + "bookcases", + metadata, + Column("bookcase_id", Integer, primary_key=True), + Column( + "bookcase_owner_id", Integer, ForeignKey("people.people_id") + ), + Column("bookcase_shelves", Integer), + Column("bookcase_width", Integer), + ) + Table( + "books", + metadata, + Column("book_id", Integer, primary_key=True), + Column( + "bookcase_id", Integer, ForeignKey("bookcases.bookcase_id") + ), + Column("book_owner_id", Integer, ForeignKey("people.people_id")), + Column("book_weight", Integer), + ) + class TextTest(fixtures.TestBase, AssertsCompiledSQL): __dialect__ = "default" @@ -530,7 +710,7 @@ class TextTest(fixtures.TestBase, AssertsCompiledSQL): t = text( "select id, name from user", typemap=dict(id=Integer, name=String), - ) + ).subquery() stmt = select([table1.c.myid]).select_from( table1.join(t, table1.c.myid == t.c.id) @@ -554,9 +734,283 @@ class TextTest(fixtures.TestBase, AssertsCompiledSQL): text("select id, name from user", autocommit=True) -table1 = table( - "mytable", - column("myid", Integer), - column("name", String), - column("description", String), -) +class SelectableTest(fixtures.TestBase, AssertsCompiledSQL): + __dialect__ = "default" + + metadata = MetaData() + table1 = Table( + "table1", + metadata, + Column("col1", Integer, primary_key=True), + Column("col2", String(20)), + Column("col3", Integer), + Column("colx", Integer), + ) + + table2 = Table( + "table2", + metadata, + Column("col1", Integer, primary_key=True), + Column("col2", Integer, ForeignKey("table1.col1")), + Column("col3", String(20)), + Column("coly", Integer), + ) + + def _c_deprecated(self): + return testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns attributes are " + "deprecated" + ) + + def test_deprecated_subquery_standalone(self): + from sqlalchemy import subquery + + with testing.expect_deprecated( + r"The standalone subquery\(\) function is deprecated" + ): + stmt = subquery( + None, + [literal_column("1").label("a")], + order_by=literal_column("1"), + ) + + self.assert_compile( + select([stmt]), + "SELECT anon_1.a FROM (SELECT 1 AS a ORDER BY 1) AS anon_1", + ) + + def test_append_column_after_replace_selectable(self): + basesel = select([literal_column("1").label("a")]) + tojoin = select( + [literal_column("1").label("a"), literal_column("2").label("b")] + ) + basefrom = basesel.alias("basefrom") + joinfrom = tojoin.alias("joinfrom") + sel = select([basefrom.c.a]) + + with testing.expect_deprecated( + r"The Selectable.replace_selectable\(\) " "method is deprecated" + ): + replaced = sel.replace_selectable( + basefrom, basefrom.join(joinfrom, basefrom.c.a == joinfrom.c.a) + ) + self.assert_compile( + replaced, + "SELECT basefrom.a FROM (SELECT 1 AS a) AS basefrom " + "JOIN (SELECT 1 AS a, 2 AS b) AS joinfrom " + "ON basefrom.a = joinfrom.a", + ) + replaced.append_column(joinfrom.c.b) + self.assert_compile( + replaced, + "SELECT basefrom.a, joinfrom.b FROM (SELECT 1 AS a) AS basefrom " + "JOIN (SELECT 1 AS a, 2 AS b) AS joinfrom " + "ON basefrom.a = joinfrom.a", + ) + + def test_against_cloned_non_table(self): + # test that corresponding column digs across + # clone boundaries with anonymous labeled elements + col = func.count().label("foo") + sel = select([col]) + + sel2 = visitors.ReplacingCloningVisitor().traverse(sel) + with testing.expect_deprecated("The SelectBase.c"): + assert ( + sel2._implicit_subquery.corresponding_column(col) is sel2.c.foo + ) + + sel3 = visitors.ReplacingCloningVisitor().traverse(sel2) + with testing.expect_deprecated("The SelectBase.c"): + assert ( + sel3._implicit_subquery.corresponding_column(col) is sel3.c.foo + ) + + def test_alias_union(self): + + # same as testunion, except its an alias of the union + + u = ( + select( + [ + self.table1.c.col1, + self.table1.c.col2, + self.table1.c.col3, + self.table1.c.colx, + null().label("coly"), + ] + ) + .union( + select( + [ + self.table2.c.col1, + self.table2.c.col2, + self.table2.c.col3, + null().label("colx"), + self.table2.c.coly, + ] + ) + ) + .alias("analias") + ) + s1 = self.table1.select(use_labels=True) + s2 = self.table2.select(use_labels=True) + with self._c_deprecated(): + assert u.corresponding_column(s1.c.table1_col2) is u.c.col2 + assert u.corresponding_column(s2.c.table2_col2) is u.c.col2 + assert u.corresponding_column(s2.c.table2_coly) is u.c.coly + assert s2.c.corresponding_column(u.c.coly) is s2.c.table2_coly + + def test_join_against_self_implicit_subquery(self): + jj = select([self.table1.c.col1.label("bar_col1")]) + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns attributes are " + "deprecated and will be removed", + "Implicit coercion of SELECT", + ): + jjj = join(self.table1, jj, self.table1.c.col1 == jj.c.bar_col1) + + jjj_bar_col1 = jjj.c["%s_bar_col1" % jj._implicit_subquery.name] + assert jjj_bar_col1 is not None + + # test column directly against itself + + assert jjj.corresponding_column(jjj.c.table1_col1) is jjj.c.table1_col1 + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns attributes are " + "deprecated and will be removed" + ): + assert jjj.corresponding_column(jj.c.bar_col1) is jjj_bar_col1 + + # test alias of the join + + j2 = jjj.alias("foo") + assert j2.corresponding_column(self.table1.c.col1) is j2.c.table1_col1 + + def test_select_labels(self): + a = self.table1.select(use_labels=True) + j = join(a._implicit_subquery, self.table2) + + criterion = a._implicit_subquery.c.table1_col1 == self.table2.c.col2 + self.assert_(criterion.compare(j.onclause)) + + +class QuoteTest(fixtures.TestBase, AssertsCompiledSQL): + __dialect__ = "default" + + def test_literal_column_label_embedded_select_samename_explcit_quote(self): + col = sql.literal_column("NEEDS QUOTES").label( + quoted_name("NEEDS QUOTES", True) + ) + + with testing.expect_deprecated( + r"The SelectBase.select\(\) method is deprecated" + ): + self.assert_compile( + select([col]).select(), + 'SELECT anon_1."NEEDS QUOTES" FROM ' + '(SELECT NEEDS QUOTES AS "NEEDS QUOTES") AS anon_1', + ) + + def test_literal_column_label_embedded_select_diffname_explcit_quote(self): + col = sql.literal_column("NEEDS QUOTES").label( + quoted_name("NEEDS QUOTES_", True) + ) + + with testing.expect_deprecated( + r"The SelectBase.select\(\) method is deprecated" + ): + self.assert_compile( + select([col]).select(), + 'SELECT anon_1."NEEDS QUOTES_" FROM ' + '(SELECT NEEDS QUOTES AS "NEEDS QUOTES_") AS anon_1', + ) + + def test_literal_column_label_embedded_select_diffname(self): + col = sql.literal_column("NEEDS QUOTES").label("NEEDS QUOTES_") + + with testing.expect_deprecated( + r"The SelectBase.select\(\) method is deprecated" + ): + self.assert_compile( + select([col]).select(), + 'SELECT anon_1."NEEDS QUOTES_" FROM (SELECT NEEDS QUOTES AS ' + '"NEEDS QUOTES_") AS anon_1', + ) + + def test_literal_column_label_embedded_select_samename(self): + col = sql.literal_column("NEEDS QUOTES").label("NEEDS QUOTES") + + with testing.expect_deprecated( + r"The SelectBase.select\(\) method is deprecated" + ): + self.assert_compile( + select([col]).select(), + 'SELECT anon_1."NEEDS QUOTES" FROM (SELECT NEEDS QUOTES AS ' + '"NEEDS QUOTES") AS anon_1', + ) + + +class TextualSelectTest(fixtures.TestBase, AssertsCompiledSQL): + __dialect__ = "default" + + table1 = table( + "mytable", + column("myid", Integer), + column("name", String), + column("description", String), + ) + + table2 = table( + "myothertable", column("otherid", Integer), column("othername", String) + ) + + def test_basic_subquery_resultmap(self): + table1 = self.table1 + t = text("select id, name from user").columns(id=Integer, name=String) + + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns", "Implicit coercion" + ): + stmt = select([table1.c.myid]).select_from( + table1.join(t, table1.c.myid == t.c.id) + ) + compiled = stmt.compile() + eq_( + compiled._create_result_map(), + { + "myid": ( + "myid", + (table1.c.myid, "myid", "myid"), + table1.c.myid.type, + ) + }, + ) + + def test_column_collection_ordered(self): + t = text("select a, b, c from foo").columns( + column("a"), column("b"), column("c") + ) + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns" + ): + eq_(t.c.keys(), ["a", "b", "c"]) + + def test_column_collection_pos_plus_bykey(self): + # overlapping positional names + type names + t = text("select a, b, c from foo").columns( + column("a"), column("b"), b=Integer, c=String + ) + + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns" + ): + eq_(t.c.keys(), ["a", "b", "c"]) + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns" + ): + eq_(t.c.b.type._type_affinity, Integer) + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns" + ): + eq_(t.c.c.type._type_affinity, String) diff --git a/test/sql/test_functions.py b/test/sql/test_functions.py index d2cf1d50af..e0efb10086 100644 --- a/test/sql/test_functions.py +++ b/test/sql/test_functions.py @@ -92,7 +92,7 @@ class CompileTest(fixtures.TestBase, AssertsCompiledSQL): dialect=dialect, ) - functions._registry['_default'].pop('fake_func') + functions._registry["_default"].pop("fake_func") def test_use_labels(self): self.assert_compile( @@ -238,7 +238,7 @@ class CompileTest(fixtures.TestBase, AssertsCompiledSQL): def test_replace_function(self): class replaceable_func(GenericFunction): type = Integer - identifier = 'replaceable_func' + identifier = "replaceable_func" assert isinstance(func.Replaceable_Func().type, Integer) assert isinstance(func.RePlAcEaBlE_fUnC().type, Integer) @@ -247,11 +247,12 @@ class CompileTest(fixtures.TestBase, AssertsCompiledSQL): with expect_warnings( "The GenericFunction 'replaceable_func' is already registered and " "is going to be overriden.", - regex=False + regex=False, ): + class replaceable_func_override(GenericFunction): type = DateTime - identifier = 'replaceable_func' + identifier = "replaceable_func" assert isinstance(func.Replaceable_Func().type, DateTime) assert isinstance(func.RePlAcEaBlE_fUnC().type, DateTime) @@ -260,7 +261,7 @@ class CompileTest(fixtures.TestBase, AssertsCompiledSQL): def test_replace_function_case_insensitive(self): class replaceable_func(GenericFunction): type = Integer - identifier = 'replaceable_func' + identifier = "replaceable_func" assert isinstance(func.Replaceable_Func().type, Integer) assert isinstance(func.RePlAcEaBlE_fUnC().type, Integer) @@ -269,11 +270,12 @@ class CompileTest(fixtures.TestBase, AssertsCompiledSQL): with expect_warnings( "The GenericFunction 'replaceable_func' is already registered and " "is going to be overriden.", - regex=False + regex=False, ): + class replaceable_func_override(GenericFunction): type = DateTime - identifier = 'REPLACEABLE_Func' + identifier = "REPLACEABLE_Func" assert isinstance(func.Replaceable_Func().type, DateTime) assert isinstance(func.RePlAcEaBlE_fUnC().type, DateTime) @@ -458,14 +460,14 @@ class CompileTest(fixtures.TestBase, AssertsCompiledSQL): from_obj=[ func.calculate(bindparam("x", None), bindparam("y", None)) ], - ) + ).subquery() self.assert_compile( select([users], users.c.id > calculate.c.z), "SELECT users.id, users.name, users.fullname " "FROM users, (SELECT q, z, r " - "FROM calculate(:x, :y)) " - "WHERE users.id > z", + "FROM calculate(:x, :y)) AS anon_1 " + "WHERE users.id > anon_1.z", ) s = select( @@ -504,8 +506,8 @@ class CompileTest(fixtures.TestBase, AssertsCompiledSQL): def test_select_method_two(self): expr = func.rows("foo") self.assert_compile( - select(["*"]).select_from(expr.select()), - "SELECT * FROM (SELECT rows(:rows_2) AS rows_1)", + select(["*"]).select_from(expr.select().subquery()), + "SELECT * FROM (SELECT rows(:rows_2) AS rows_1) AS anon_1", ) def test_select_method_three(self): @@ -1087,7 +1089,7 @@ class RegisterTest(fixtures.TestBase, AssertsCompiledSQL): functions._registry = self._registry def test_GenericFunction_is_registered(self): - assert 'GenericFunction' not in functions._registry['_default'] + assert "GenericFunction" not in functions._registry["_default"] def test_register_function(self): @@ -1101,7 +1103,7 @@ class RegisterTest(fixtures.TestBase, AssertsCompiledSQL): class registered_func_child(registered_func): type = sqltypes.Integer - assert 'registered_func' in functions._registry['_default'] + assert "registered_func" in functions._registry["_default"] assert isinstance(func.registered_func_child().type, Integer) class not_registered_func(GenericFunction): @@ -1113,5 +1115,5 @@ class RegisterTest(fixtures.TestBase, AssertsCompiledSQL): class not_registered_func_child(not_registered_func): type = sqltypes.Integer - assert 'not_registered_func' not in functions._registry['_default'] + assert "not_registered_func" not in functions._registry["_default"] assert isinstance(func.not_registered_func_child().type, Integer) diff --git a/test/sql/test_generative.py b/test/sql/test_generative.py index 3bf6c70562..93e4355174 100644 --- a/test/sql/test_generative.py +++ b/test/sql/test_generative.py @@ -390,10 +390,10 @@ class ClauseTest(fixtures.TestBase, AssertsCompiledSQL): def test_aliased_cloned_column_adapt_inner(self): clause = select([t1.c.col1, func.foo(t1.c.col2).label("foo")]) - - aliased1 = select([clause.c.col1, clause.c.foo]) + c_sub = clause.subquery() + aliased1 = select([c_sub.c.col1, c_sub.c.foo]).subquery() aliased2 = clause - aliased2.c.col1, aliased2.c.foo + aliased2.selected_columns.col1, aliased2.selected_columns.foo aliased3 = cloned_traverse(aliased2, {}, {}) # fixed by [ticket:2419]. the inside columns @@ -406,9 +406,11 @@ class ClauseTest(fixtures.TestBase, AssertsCompiledSQL): eq_(str(f1), str(f2)) def test_aliased_cloned_column_adapt_exported(self): - clause = select([t1.c.col1, func.foo(t1.c.col2).label("foo")]) + clause = select( + [t1.c.col1, func.foo(t1.c.col2).label("foo")] + ).subquery() - aliased1 = select([clause.c.col1, clause.c.foo]) + aliased1 = select([clause.c.col1, clause.c.foo]).subquery() aliased2 = clause aliased2.c.col1, aliased2.c.foo aliased3 = cloned_traverse(aliased2, {}, {}) @@ -424,10 +426,11 @@ class ClauseTest(fixtures.TestBase, AssertsCompiledSQL): def test_aliased_cloned_schema_column_adapt_exported(self): clause = select([t3.c.col1, func.foo(t3.c.col2).label("foo")]) + c_sub = clause.subquery() - aliased1 = select([clause.c.col1, clause.c.foo]) + aliased1 = select([c_sub.c.col1, c_sub.c.foo]).subquery() aliased2 = clause - aliased2.c.col1, aliased2.c.foo + aliased2.selected_columns.col1, aliased2.selected_columns.foo aliased3 = cloned_traverse(aliased2, {}, {}) # also fixed by [ticket:2419]. When we look at the @@ -435,8 +438,8 @@ class ClauseTest(fixtures.TestBase, AssertsCompiledSQL): # have an _is_clone_of pointer. But we now modified _make_proxy # to assign this. adapter = sql_util.ColumnAdapter(aliased1) - f1 = select([adapter.columns[c] for c in aliased2.c]) - f2 = select([adapter.columns[c] for c in aliased3.c]) + f1 = select([adapter.columns[c] for c in aliased2.selected_columns]) + f2 = select([adapter.columns[c] for c in aliased3.selected_columns]) eq_(str(f1), str(f2)) def test_labeled_expression_adapt(self): @@ -567,14 +570,18 @@ class ClauseTest(fixtures.TestBase, AssertsCompiledSQL): def test_union(self): u = union(t1.select(), t2.select()) u2 = CloningVisitor().traverse(u) - assert str(u) == str(u2) - assert [str(c) for c in u2.c] == [str(c) for c in u.c] + eq_(str(u), str(u2)) + + eq_( + [str(c) for c in u2.selected_columns], + [str(c) for c in u.selected_columns], + ) u = union(t1.select(), t2.select()) - cols = [str(c) for c in u.c] + cols = [str(c) for c in u.selected_columns] u2 = CloningVisitor().traverse(u) - assert str(u) == str(u2) - assert [str(c) for c in u2.c] == cols + eq_(str(u), str(u2)) + eq_([str(c) for c in u2.selected_columns], cols) s1 = select([t1], t1.c.col1 == bindparam("id_param")) s2 = select([t2]) @@ -582,9 +589,11 @@ class ClauseTest(fixtures.TestBase, AssertsCompiledSQL): u2 = u.params(id_param=7) u3 = u.params(id_param=10) - assert str(u) == str(u2) == str(u3) - assert u2.compile().params == {"id_param": 7} - assert u3.compile().params == {"id_param": 10} + + eq_(str(u), str(u2)) + eq_(str(u2), str(u3)) + eq_(u2.compile().params, {"id_param": 7}) + eq_(u3.compile().params, {"id_param": 10}) def test_in(self): expr = t1.c.col1.in_(["foo", "bar"]) @@ -677,16 +686,22 @@ class ClauseTest(fixtures.TestBase, AssertsCompiledSQL): ) orig = str(s) s2 = CloningVisitor().traverse(s) - assert orig == str(s) == str(s2) + eq_(orig, str(s)) + eq_(str(s), str(s2)) s4 = CloningVisitor().traverse(s2) - assert orig == str(s) == str(s2) == str(s4) + eq_(orig, str(s)) + eq_(str(s), str(s2)) + eq_(str(s), str(s4)) s3 = sql_util.ClauseAdapter(table("foo")).traverse(s) - assert orig == str(s) == str(s3) + eq_(orig, str(s)) + eq_(str(s), str(s3)) s4 = sql_util.ClauseAdapter(table("foo")).traverse(s3) - assert orig == str(s) == str(s3) == str(s4) + eq_(orig, str(s)) + eq_(str(s), str(s3)) + eq_(str(s), str(s4)) subq = subq.alias("subq") s = select( @@ -694,7 +709,7 @@ class ClauseTest(fixtures.TestBase, AssertsCompiledSQL): from_obj=[t1, subq, t1.join(subq, t1.c.col1 == subq.c.col2)], ) s5 = CloningVisitor().traverse(s) - assert orig == str(s) == str(s5) + eq_(str(s), str(s5)) def test_correlated_select(self): s = select( @@ -882,10 +897,10 @@ class ColumnAdapterTest(fixtures.TestBase, AssertsCompiledSQL): """ - stmt = select([t1.c.col1, t2.c.col1]).apply_labels() + stmt = select([t1.c.col1, t2.c.col1]).apply_labels().subquery() sa = stmt.alias() - stmt2 = select([t2, sa]) + stmt2 = select([t2, sa]).subquery() a1 = sql_util.ColumnAdapter(stmt) a2 = sql_util.ColumnAdapter(stmt2) @@ -902,6 +917,9 @@ class ColumnAdapterTest(fixtures.TestBase, AssertsCompiledSQL): # sends this to stmt2.c.table2_col1 is_(a1_to_a2.columns[t2.c.col1], stmt2.c.table2_col1) + # check that these aren't the same column + is_not_(stmt2.c.col1, stmt2.c.table2_col1) + # for mutually exclusive columns, order doesn't matter is_(a2_to_a1.columns[t1.c.col1], stmt2.c.table1_col1) is_(a1_to_a2.columns[t1.c.col1], stmt2.c.table1_col1) @@ -1491,7 +1509,7 @@ class ClauseAdapterTest(fixtures.TestBase, AssertsCompiledSQL): ) j1 = a.outerjoin(b) - j2 = select([j1], use_labels=True) + j2 = select([j1], use_labels=True).subquery() j3 = c.join(j2, j2.c.b_id == c.c.bid) @@ -1500,8 +1518,8 @@ class ClauseAdapterTest(fixtures.TestBase, AssertsCompiledSQL): j4, "c JOIN (SELECT a.id AS a_id, b.id AS " "b_id, b.aid AS b_aid FROM a LEFT OUTER " - "JOIN b ON a.id = b.aid) ON b_id = c.bid " - "LEFT OUTER JOIN d ON a_id = d.aid", + "JOIN b ON a.id = b.aid) AS anon_1 ON anon_1.b_id = c.bid " + "LEFT OUTER JOIN d ON anon_1.a_id = d.aid", ) j5 = j3.alias("foo") j6 = sql_util.ClauseAdapter(j5).copy_and_process([j4])[0] @@ -1514,12 +1532,13 @@ class ClauseAdapterTest(fixtures.TestBase, AssertsCompiledSQL): self.assert_compile( j6, "(SELECT c.id AS c_id, c.bid AS c_bid, " - "a_id AS a_id, b_id AS b_id, b_aid AS " - "b_aid FROM c JOIN (SELECT a.id AS a_id, " + "anon_1.a_id AS anon_1_a_id, anon_1.b_id AS anon_1_b_id, " + "anon_1.b_aid AS " + "anon_1_b_aid FROM c JOIN (SELECT a.id AS a_id, " "b.id AS b_id, b.aid AS b_aid FROM a LEFT " - "OUTER JOIN b ON a.id = b.aid) ON b_id = " + "OUTER JOIN b ON a.id = b.aid) AS anon_1 ON anon_1.b_id = " "c.bid) AS foo LEFT OUTER JOIN d ON " - "foo.a_id = d.aid", + "foo.anon_1_a_id = d.aid", ) def test_derived_from(self): @@ -1805,9 +1824,8 @@ class SelectTest(fixtures.TestBase, AssertsCompiledSQL): select_copy, "SELECT table1.col1, table1.col2, " "table1.col3, yyy FROM table1", ) - assert s.columns is not select_copy.columns - assert s._columns is not select_copy._columns - assert s._raw_columns is not select_copy._raw_columns + is_not_(s.selected_columns, select_copy.selected_columns) + is_not_(s._raw_columns, select_copy._raw_columns) self.assert_compile( s, "SELECT table1.col1, table1.col2, " "table1.col3 FROM table1" ) diff --git a/test/sql/test_join_rewriting.py b/test/sql/test_join_rewriting.py index b9bcfc16a1..a966af4e49 100644 --- a/test/sql/test_join_rewriting.py +++ b/test/sql/test_join_rewriting.py @@ -94,7 +94,7 @@ class _JoinRewriteTestBase(AssertsCompiledSQL): # column name should be in result map, as we never render # .key in SQL - for key, col in zip([c.name for c in s.c], s.inner_columns): + for key, col in zip([c.name for c in s.subquery().c], s.inner_columns): key = key % compiled.anon_map assert col in compiled._create_result_map()[key][1] @@ -226,9 +226,14 @@ class _JoinRewriteTestBase(AssertsCompiledSQL): b_j1 = b.join(j1) b_j2 = b.join(j2) - s = union( - select([b_j1], use_labels=True), select([b_j2], use_labels=True) - ).select(use_labels=True) + s = ( + union( + select([b_j1], use_labels=True), + select([b_j2], use_labels=True), + ) + .subquery() + .select(use_labels=True) + ) self._test(s, self._b_ab1_union_c_ab2) @@ -378,17 +383,19 @@ class JoinRewriteTest(_JoinRewriteTestBase, fixtures.TestBase): ) _b_ab1_union_c_ab2 = ( - "SELECT b_id AS b_id, b_a_id AS b_a_id, a_id AS a_id, b1_id AS b1_id, " - "b1_a_id AS b1_a_id FROM " - "(SELECT b.id AS b_id, b.a_id AS b_a_id, anon_1.a_id AS a_id, " - "anon_1.b1_id AS b1_id, anon_1.b1_a_id AS b1_a_id " + "SELECT anon_1.b_id AS anon_1_b_id, anon_1.b_a_id AS anon_1_b_a_id, " + "anon_1.a_id AS anon_1_a_id, anon_1.b1_id AS anon_1_b1_id, " + "anon_1.b1_a_id AS anon_1_b1_a_id FROM " + "(SELECT b.id AS b_id, b.a_id AS b_a_id, anon_2.a_id AS a_id, " + "anon_2.b1_id AS b1_id, anon_2.b1_a_id AS b1_a_id " "FROM b JOIN (SELECT a.id AS a_id, b1.id AS b1_id, b1.a_id AS b1_a_id " - "FROM a JOIN b1 ON a.id = b1.a_id) AS anon_1 ON anon_1.a_id = b.a_id " + "FROM a JOIN b1 ON a.id = b1.a_id) AS anon_2 ON anon_2.a_id = b.a_id " "UNION " - "SELECT b.id AS b_id, b.a_id AS b_a_id, anon_2.a_id AS a_id, " - "anon_2.b2_id AS b2_id, anon_2.b2_a_id AS b2_a_id " + "SELECT b.id AS b_id, b.a_id AS b_a_id, anon_3.a_id AS a_id, " + "anon_3.b2_id AS b2_id, anon_3.b2_a_id AS b2_a_id " "FROM b JOIN (SELECT a.id AS a_id, b2.id AS b2_id, b2.a_id AS b2_a_id " - "FROM a JOIN b2 ON a.id = b2.a_id) AS anon_2 ON anon_2.a_id = b.a_id)" + "FROM a JOIN b2 ON a.id = b2.a_id) AS anon_3 ON anon_3.a_id = b.a_id) " + "AS anon_1" ) _b_a_id_double_overlap_annotated = ( @@ -495,8 +502,9 @@ class JoinPlainTest(_JoinRewriteTestBase, fixtures.TestBase): ) _b_ab1_union_c_ab2 = ( - "SELECT b_id AS b_id, b_a_id AS b_a_id, a_id AS a_id, b1_id AS b1_id, " - "b1_a_id AS b1_a_id FROM " + "SELECT anon_1.b_id AS anon_1_b_id, anon_1.b_a_id AS anon_1_b_a_id, " + "anon_1.a_id AS anon_1_a_id, anon_1.b1_id AS anon_1_b1_id, " + "anon_1.b1_a_id AS anon_1_b1_a_id FROM " "(SELECT b.id AS b_id, b.a_id AS b_a_id, a.id AS a_id, " "b1.id AS b1_id, " "b1.a_id AS b1_a_id FROM b " @@ -504,7 +512,7 @@ class JoinPlainTest(_JoinRewriteTestBase, fixtures.TestBase): "UNION " "SELECT b.id AS b_id, b.a_id AS b_a_id, a.id AS a_id, b2.id AS b2_id, " "b2.a_id AS b2_a_id FROM b " - "JOIN (a JOIN b2 ON a.id = b2.a_id) ON a.id = b.a_id)" + "JOIN (a JOIN b2 ON a.id = b2.a_id) ON a.id = b.a_id) AS anon_1" ) _b_a_id_double_overlap_annotated = ( @@ -606,14 +614,15 @@ class JoinNoUseLabelsTest(_JoinRewriteTestBase, fixtures.TestBase): ) _b_ab1_union_c_ab2 = ( - "SELECT b_id, b_a_id, a_id, b1_id, b1_a_id " + "SELECT anon_1.b_id, anon_1.b_a_id, anon_1.a_id, anon_1.b1_id, " + "anon_1.b1_a_id " "FROM (SELECT b.id AS b_id, b.a_id AS b_a_id, a.id AS a_id, " "b1.id AS b1_id, b1.a_id AS b1_a_id " "FROM b JOIN (a JOIN b1 ON a.id = b1.a_id) ON a.id = b.a_id " "UNION " "SELECT b.id AS b_id, b.a_id AS b_a_id, a.id AS a_id, b2.id AS b2_id, " "b2.a_id AS b2_a_id " - "FROM b JOIN (a JOIN b2 ON a.id = b2.a_id) ON a.id = b.a_id)" + "FROM b JOIN (a JOIN b2 ON a.id = b2.a_id) ON a.id = b.a_id) AS anon_1" ) _b_a_id_double_overlap_annotated = ( diff --git a/test/sql/test_lateral.py b/test/sql/test_lateral.py index ee9b13d1df..b18dde57bc 100644 --- a/test/sql/test_lateral.py +++ b/test/sql/test_lateral.py @@ -7,6 +7,7 @@ from sqlalchemy import lateral from sqlalchemy import String from sqlalchemy import Table from sqlalchemy import table +from sqlalchemy import text from sqlalchemy import true from sqlalchemy.engine import default from sqlalchemy.sql import func @@ -55,6 +56,22 @@ class LateralTest(fixtures.TablesTest, AssertsCompiledSQL): ) def test_standalone(self): + table1 = self.tables.people + subq = select([table1.c.people_id]).subquery() + + # alias name is not rendered because subquery is not + # in the context of a FROM clause + self.assert_compile( + lateral(subq, name="alias"), + "LATERAL (SELECT people.people_id FROM people)", + ) + + self.assert_compile( + subq.lateral(name="alias"), + "LATERAL (SELECT people.people_id FROM people)", + ) + + def test_standalone_implicit_subquery(self): table1 = self.tables.people subq = select([table1.c.people_id]) @@ -71,6 +88,17 @@ class LateralTest(fixtures.TablesTest, AssertsCompiledSQL): ) def test_select_from(self): + table1 = self.tables.people + subq = select([table1.c.people_id]).subquery() + + # in a FROM context, now you get "AS alias" and column labeling + self.assert_compile( + select([subq.lateral(name="alias")]), + "SELECT alias.people_id FROM LATERAL " + "(SELECT people.people_id AS people_id FROM people) AS alias", + ) + + def test_select_from_implicit_subquery(self): table1 = self.tables.people subq = select([table1.c.people_id]) @@ -81,6 +109,17 @@ class LateralTest(fixtures.TablesTest, AssertsCompiledSQL): "(SELECT people.people_id AS people_id FROM people) AS alias", ) + def test_select_from_text_implicit_subquery(self): + table1 = self.tables.people + subq = text("SELECT people_id FROM people").columns(table1.c.people_id) + + # in a FROM context, now you get "AS alias" and column labeling + self.assert_compile( + select([subq.lateral(name="alias")]), + "SELECT alias.people_id FROM LATERAL " + "(SELECT people_id FROM people) AS alias", + ) + def test_plain_join(self): table1 = self.tables.people table2 = self.tables.books @@ -88,6 +127,45 @@ class LateralTest(fixtures.TablesTest, AssertsCompiledSQL): table2.c.book_owner_id == table1.c.people_id ) + # FROM books, people? isn't this wrong? No! Because + # this is only a fragment, books isn't in any other FROM clause + self.assert_compile( + join(table1, lateral(subq.subquery(), name="alias"), true()), + "people JOIN LATERAL (SELECT books.book_id AS book_id " + "FROM books, people WHERE books.book_owner_id = people.people_id) " + "AS alias ON true", + ) + + # put it in correct context, implicit correlation works fine + self.assert_compile( + select([table1]).select_from( + join(table1, lateral(subq.subquery(), name="alias"), true()) + ), + "SELECT people.people_id, people.age, people.name " + "FROM people JOIN LATERAL (SELECT books.book_id AS book_id " + "FROM books WHERE books.book_owner_id = people.people_id) " + "AS alias ON true", + ) + + # explicit correlation + subq = subq.correlate(table1) + self.assert_compile( + select([table1]).select_from( + join(table1, lateral(subq.subquery(), name="alias"), true()) + ), + "SELECT people.people_id, people.age, people.name " + "FROM people JOIN LATERAL (SELECT books.book_id AS book_id " + "FROM books WHERE books.book_owner_id = people.people_id) " + "AS alias ON true", + ) + + def test_plain_join_implicit_subquery(self): + table1 = self.tables.people + table2 = self.tables.books + subq = select([table2.c.book_id]).where( + table2.c.book_owner_id == table1.c.people_id + ) + # FROM books, people? isn't this wrong? No! Because # this is only a fragment, books isn't in any other FROM clause self.assert_compile( @@ -128,6 +206,7 @@ class LateralTest(fixtures.TablesTest, AssertsCompiledSQL): select([table2.c.book_id]) .correlate(table1) .where(table1.c.people_id == table2.c.book_owner_id) + .subquery() .lateral() ) stmt = select([table1, subq.c.book_id]).select_from( @@ -142,6 +221,30 @@ class LateralTest(fixtures.TablesTest, AssertsCompiledSQL): "WHERE people.people_id = books.book_owner_id) AS anon_1 ON true", ) + def test_join_lateral_w_select_implicit_subquery(self): + table1 = self.tables.people + table2 = self.tables.books + + subq = ( + select([table2.c.book_id]) + .correlate(table1) + .where(table1.c.people_id == table2.c.book_owner_id) + .lateral() + ) + stmt = select([table1, subq.c.book_id]).select_from( + table1.join(subq, true()) + ) + + self.assert_compile( + stmt, + "SELECT people.people_id, people.age, people.name, " + "anon_1.book_id " + "FROM people JOIN LATERAL (SELECT books.book_id AS book_id " + "FROM books " + "WHERE people.people_id = books.book_owner_id) " + "AS anon_1 ON true", + ) + def test_from_function(self): bookcases = self.tables.bookcases srf = lateral(func.generate_series(1, bookcases.c.bookcase_shelves)) diff --git a/test/sql/test_metadata.py b/test/sql/test_metadata.py index 0d3a044303..5081cfbd2a 100644 --- a/test/sql/test_metadata.py +++ b/test/sql/test_metadata.py @@ -2634,7 +2634,7 @@ class ConstraintTest(fixtures.TestBase): def test_column_references_derived(self): t1, t2, t3 = self._single_fixture() - s1 = tsa.select([tsa.select([t1]).alias()]) + s1 = tsa.select([tsa.select([t1]).alias()]).subquery() assert t2.c.a.references(s1.c.a) assert not t2.c.a.references(s1.c.b) @@ -2646,7 +2646,7 @@ class ConstraintTest(fixtures.TestBase): def test_derived_column_references(self): t1, t2, t3 = self._single_fixture() - s1 = tsa.select([tsa.select([t2]).alias()]) + s1 = tsa.select([tsa.select([t2]).alias()]).subquery() assert s1.c.a.references(t1.c.a) assert not s1.c.a.references(t1.c.b) @@ -2765,7 +2765,7 @@ class ConstraintTest(fixtures.TestBase): Column("id", Integer, ForeignKey("t1.id"), primary_key=True), ) - s = tsa.select([t2]) + s = tsa.select([t2]).subquery() t2fk = list(t2.c.id.foreign_keys)[0] sfk = list(s.c.id.foreign_keys)[0] @@ -3532,7 +3532,7 @@ class ColumnDefinitionTest(AssertsCompiledSQL, fixtures.TestBase): s = select([t1.select().alias()]) # proxy has goofy thing - eq_(s.c.name.my_goofy_thing(), "hi") + eq_(s.subquery().c.name.my_goofy_thing(), "hi") # compile works self.assert_compile( @@ -3561,7 +3561,7 @@ class ColumnDefinitionTest(AssertsCompiledSQL, fixtures.TestBase): "'test.sql.test_metadata..*MyColumn'> " "object. Ensure the class includes a _constructor()", getattr, - select([t1.select().alias()]), + select([t1.select().alias()]).subquery(), "c", ) diff --git a/test/sql/test_operators.py b/test/sql/test_operators.py index f85a601bab..66fe185983 100644 --- a/test/sql/test_operators.py +++ b/test/sql/test_operators.py @@ -402,7 +402,17 @@ class _CustomComparatorTests(object): def test_column_proxy(self): t = Table("t", MetaData(), Column("foo", self._add_override_factory())) - proxied = t.select().c.foo + with testing.expect_deprecated( + "The SelectBase.c and SelectBase.columns attributes " + "are deprecated" + ): + proxied = t.select().c.foo + self._assert_add_override(proxied) + self._assert_and_override(proxied) + + def test_subquery_proxy(self): + t = Table("t", MetaData(), Column("foo", self._add_override_factory())) + proxied = t.select().subquery().c.foo self._assert_add_override(proxied) self._assert_and_override(proxied) diff --git a/test/sql/test_query.py b/test/sql/test_query.py index 2726b3f3c2..ba878cfbc7 100644 --- a/test/sql/test_query.py +++ b/test/sql/test_query.py @@ -704,7 +704,7 @@ class QueryTest(fixtures.TestBase): ) ) .where(users.c.user_id == 9) - ).order_by(stmt.c.user_id) + ).order_by("user_id") eq_( conn.execute( diff --git a/test/sql/test_quote.py b/test/sql/test_quote.py index 8ef4728781..47206d3f63 100644 --- a/test/sql/test_quote.py +++ b/test/sql/test_quote.py @@ -695,17 +695,6 @@ class QuoteTest(fixtures.TestBase, AssertsCompiledSQL): ') AS "Alias1"', ) - def test_literal_column_label_embedded_select_samename(self): - col = sql.literal_column("NEEDS QUOTES").label("NEEDS QUOTES") - - # embedded SELECT use case, going away in 1.4 however use a - # SelectStatementGrouping here when that merges - self.assert_compile( - select([col]).select(), - 'SELECT "NEEDS QUOTES" FROM (SELECT NEEDS QUOTES AS ' - '"NEEDS QUOTES")', - ) - def test_literal_column_label_alias_samename(self): col = sql.literal_column("NEEDS QUOTES").label("NEEDS QUOTES") @@ -715,17 +704,6 @@ class QuoteTest(fixtures.TestBase, AssertsCompiledSQL): '"NEEDS QUOTES") AS anon_1', ) - def test_literal_column_label_embedded_select_diffname(self): - col = sql.literal_column("NEEDS QUOTES").label("NEEDS QUOTES_") - - # embedded SELECT use case, going away in 1.4 however use a - # SelectStatementGrouping here when that merges - self.assert_compile( - select([col]).select(), - 'SELECT "NEEDS QUOTES_" FROM (SELECT NEEDS QUOTES AS ' - '"NEEDS QUOTES_")', - ) - def test_literal_column_label_alias_diffname(self): col = sql.literal_column("NEEDS QUOTES").label("NEEDS QUOTES_") @@ -735,19 +713,6 @@ class QuoteTest(fixtures.TestBase, AssertsCompiledSQL): '"NEEDS QUOTES_") AS anon_1', ) - def test_literal_column_label_embedded_select_samename_explcit_quote(self): - col = sql.literal_column("NEEDS QUOTES").label( - quoted_name("NEEDS QUOTES", True) - ) - - # embedded SELECT use case, going away in 1.4 however use a - # SelectStatementGrouping here when that merges - self.assert_compile( - select([col]).select(), - 'SELECT "NEEDS QUOTES" FROM ' - '(SELECT NEEDS QUOTES AS "NEEDS QUOTES")', - ) - def test_literal_column_label_alias_samename_explcit_quote(self): col = sql.literal_column("NEEDS QUOTES").label( quoted_name("NEEDS QUOTES", True) @@ -759,19 +724,6 @@ class QuoteTest(fixtures.TestBase, AssertsCompiledSQL): '(SELECT NEEDS QUOTES AS "NEEDS QUOTES") AS anon_1', ) - def test_literal_column_label_embedded_select_diffname_explcit_quote(self): - col = sql.literal_column("NEEDS QUOTES").label( - quoted_name("NEEDS QUOTES_", True) - ) - - # embedded SELECT use case, going away in 1.4 however use a - # SelectStatementGrouping here when that merges - self.assert_compile( - select([col]).select(), - 'SELECT "NEEDS QUOTES_" FROM ' - '(SELECT NEEDS QUOTES AS "NEEDS QUOTES_")', - ) - def test_literal_column_label_alias_diffname_explcit_quote(self): col = sql.literal_column("NEEDS QUOTES").label( quoted_name("NEEDS QUOTES_", True) diff --git a/test/sql/test_resultset.py b/test/sql/test_resultset.py index 6e48374ca3..5289ddb77a 100644 --- a/test/sql/test_resultset.py +++ b/test/sql/test_resultset.py @@ -1418,8 +1418,8 @@ class KeyTargetingTest(fixtures.TablesTest): in_(keyed2.c.b, row) in_(a, row) in_(b, row) - in_(stmt.c.a, row) - in_(stmt.c.b, row) + in_(stmt.selected_columns.a, row) + in_(stmt.selected_columns.b, row) def test_columnclause_schema_column_four(self): keyed2 = self.tables.keyed2 @@ -1436,8 +1436,8 @@ class KeyTargetingTest(fixtures.TablesTest): in_(keyed2.c.b, row) in_(a, row) in_(b, row) - in_(stmt.c.keyed2_a, row) - in_(stmt.c.keyed2_b, row) + in_(stmt.selected_columns.keyed2_a, row) + in_(stmt.selected_columns.keyed2_b, row) def test_columnclause_schema_column_five(self): keyed2 = self.tables.keyed2 @@ -1451,8 +1451,8 @@ class KeyTargetingTest(fixtures.TablesTest): in_(keyed2.c.a, row) in_(keyed2.c.b, row) - in_(stmt.c.keyed2_a, row) - in_(stmt.c.keyed2_b, row) + in_(stmt.selected_columns.keyed2_a, row) + in_(stmt.selected_columns.keyed2_b, row) class PositionalTextTest(fixtures.TablesTest): diff --git a/test/sql/test_roles.py b/test/sql/test_roles.py index 0d819c13d9..de5951d203 100644 --- a/test/sql/test_roles.py +++ b/test/sql/test_roles.py @@ -4,6 +4,7 @@ from sqlalchemy import Integer from sqlalchemy import MetaData from sqlalchemy import select from sqlalchemy import Table +from sqlalchemy import testing from sqlalchemy import text from sqlalchemy.schema import DDL from sqlalchemy.schema import Sequence @@ -19,6 +20,8 @@ from sqlalchemy.sql import True_ from sqlalchemy.sql.coercions import expect from sqlalchemy.sql.elements import _truncated_label from sqlalchemy.sql.elements import Null +from sqlalchemy.sql.selectable import FromGrouping +from sqlalchemy.sql.selectable import SelectStatementGrouping from sqlalchemy.testing import assert_raises_message from sqlalchemy.testing import fixtures from sqlalchemy.testing import is_ @@ -184,6 +187,37 @@ class RoleTest(fixtures.TestBase): "select * from table", ) + def test_select_is_coerced_into_fromclause_w_deprecation(self): + with testing.expect_deprecated( + "Implicit coercion of SELECT and textual SELECT " + "constructs into FROM clauses is deprecated;" + ): + element = expect( + roles.FromClauseRole, SelectStatementGrouping(select([t])) + ) + is_true( + element.compare( + SelectStatementGrouping(select([t])).subquery() + ) + ) + + def test_from_clause_is_not_a_select(self): + assert_raises_message( + exc.ArgumentError, + r"SELECT construct or equivalent text\(\) construct expected,", + expect, + roles.SelectStatementRole, + FromGrouping(t), + ) + + def test_text_as_from_select_statement(self): + is_true( + expect( + roles.SelectStatementRole, + text("select * from table").columns(t.c.q), + ).compare(text("select * from table").columns(t.c.q)) + ) + def test_statement_coercion_select(self): is_true( expect(roles.CoerceTextStatementRole, select([t])).compare( diff --git a/test/sql/test_selectable.py b/test/sql/test_selectable.py index 230b5423ba..485a0e4281 100644 --- a/test/sql/test_selectable.py +++ b/test/sql/test_selectable.py @@ -29,6 +29,7 @@ from sqlalchemy import TypeDecorator from sqlalchemy import union from sqlalchemy import util from sqlalchemy.sql import Alias +from sqlalchemy.sql import base from sqlalchemy.sql import column from sqlalchemy.sql import elements from sqlalchemy.sql import table @@ -88,7 +89,7 @@ class SelectableTest( table1.c.col1, table1.c.col1.label("c1"), ] - ) + ).subquery() # this tests the same thing as # test_direct_correspondence_on_labels below - @@ -98,12 +99,38 @@ class SelectableTest( assert s.corresponding_column(s.c.col1) is s.c.col1 assert s.corresponding_column(s.c.c1) is s.c.c1 - def test_labeled_subquery_twice(self): + def test_labeled_select_twice(self): scalar_select = select([table1.c.col1]).label("foo") s1 = select([scalar_select]) s2 = select([scalar_select, scalar_select]) + eq_( + s1.selected_columns.foo.proxy_set, + set( + [s1.selected_columns.foo, scalar_select, scalar_select.element] + ), + ) + eq_( + s2.selected_columns.foo.proxy_set, + set( + [s2.selected_columns.foo, scalar_select, scalar_select.element] + ), + ) + + assert ( + s1.corresponding_column(scalar_select) is s1.selected_columns.foo + ) + assert ( + s2.corresponding_column(scalar_select) is s2.selected_columns.foo + ) + + def test_labeled_subquery_twice(self): + scalar_select = select([table1.c.col1]).label("foo") + + s1 = select([scalar_select]).subquery() + s2 = select([scalar_select, scalar_select]).subquery() + eq_( s1.c.foo.proxy_set, set([s1.c.foo, scalar_select, scalar_select.element]), @@ -116,12 +143,21 @@ class SelectableTest( assert s1.corresponding_column(scalar_select) is s1.c.foo assert s2.corresponding_column(scalar_select) is s2.c.foo - def test_label_grouped_still_corresponds(self): + def test_select_label_grouped_still_corresponds(self): label = select([table1.c.col1]).label("foo") label2 = label.self_group() s1 = select([label]) s2 = select([label2]) + assert s1.corresponding_column(label) is s1.selected_columns.foo + assert s2.corresponding_column(label) is s2.selected_columns.foo + + def test_subquery_label_grouped_still_corresponds(self): + label = select([table1.c.col1]).label("foo") + label2 = label.self_group() + + s1 = select([label]).subquery() + s2 = select([label2]).subquery() assert s1.corresponding_column(label) is s1.c.foo assert s2.corresponding_column(label) is s2.c.foo @@ -144,13 +180,22 @@ class SelectableTest( def test_keyed_gen(self): s = select([keyed]) - eq_(s.c.colx.key, "colx") + eq_(s.selected_columns.colx.key, "colx") - eq_(s.c.colx.name, "x") + eq_(s.selected_columns.colx.name, "x") - assert s.corresponding_column(keyed.c.colx) is s.c.colx - assert s.corresponding_column(keyed.c.coly) is s.c.coly - assert s.corresponding_column(keyed.c.z) is s.c.z + assert ( + s.selected_columns.corresponding_column(keyed.c.colx) + is s.selected_columns.colx + ) + assert ( + s.selected_columns.corresponding_column(keyed.c.coly) + is s.selected_columns.coly + ) + assert ( + s.selected_columns.corresponding_column(keyed.c.z) + is s.selected_columns.z + ) sel2 = s.alias() assert sel2.corresponding_column(keyed.c.colx) is sel2.c.colx @@ -160,9 +205,18 @@ class SelectableTest( def test_keyed_label_gen(self): s = select([keyed]).apply_labels() - assert s.corresponding_column(keyed.c.colx) is s.c.keyed_colx - assert s.corresponding_column(keyed.c.coly) is s.c.keyed_coly - assert s.corresponding_column(keyed.c.z) is s.c.keyed_z + assert ( + s.selected_columns.corresponding_column(keyed.c.colx) + is s.selected_columns.keyed_colx + ) + assert ( + s.selected_columns.corresponding_column(keyed.c.coly) + is s.selected_columns.keyed_coly + ) + assert ( + s.selected_columns.corresponding_column(keyed.c.z) + is s.selected_columns.keyed_z + ) sel2 = s.alias() assert sel2.corresponding_column(keyed.c.colx) is sel2.c.keyed_colx @@ -184,6 +238,9 @@ class SelectableTest( c = Column("foo", Integer, key="bar") t = Table("t", MetaData(), c) s = select([t])._clone() + assert c in s.selected_columns.bar.proxy_set + + s = select([t]).subquery()._clone() assert c in s.c.bar.proxy_set def test_clone_c_proxy_key_lower(self): @@ -191,6 +248,9 @@ class SelectableTest( c.key = "bar" t = table("t", c) s = select([t])._clone() + assert c in s.selected_columns.bar.proxy_set + + s = select([t]).subquery()._clone() assert c in s.c.bar.proxy_set def test_no_error_on_unsupported_expr_key(self): @@ -204,6 +264,13 @@ class SelectableTest( expr = BinaryExpression(t.c.x, t.c.y, myop) s = select([t, expr]) + + # anon_label, e.g. a truncated_label, is used here becuase + # the expr has no name, no key, and myop() can't create a + # string, so this is the last resort + eq_(s.selected_columns.keys(), ["x", "y", expr.anon_label]) + + s = select([t, expr]).subquery() eq_(s.c.keys(), ["x", "y", expr.anon_label]) def test_cloned_intersection(self): @@ -219,10 +286,7 @@ class SelectableTest( s2c1 = s2._clone() s3c1 = s3._clone() - eq_( - elements._cloned_intersection([s1c1, s3c1], [s2c1, s1c2]), - set([s1c1]), - ) + eq_(base._cloned_intersection([s1c1, s3c1], [s2c1, s1c2]), set([s1c1])) def test_cloned_difference(self): t1 = table("t1", column("x")) @@ -238,27 +302,36 @@ class SelectableTest( s3c1 = s3._clone() eq_( - elements._cloned_difference([s1c1, s2c1, s3c1], [s2c1, s1c2]), + base._cloned_difference([s1c1, s2c1, s3c1], [s2c1, s1c2]), set([s3c1]), ) def test_distance_on_aliases(self): a1 = table1.alias("a1") for s in ( - select([a1, table1], use_labels=True), - select([table1, a1], use_labels=True), + select([a1, table1], use_labels=True).subquery(), + select([table1, a1], use_labels=True).subquery(), ): assert s.corresponding_column(table1.c.col1) is s.c.table1_col1 assert s.corresponding_column(a1.c.col1) is s.c.a1_col1 def test_join_against_self(self): - jj = select([table1.c.col1.label("bar_col1")]) + jj = select([table1.c.col1.label("bar_col1")]).subquery() jjj = join(table1, jj, table1.c.col1 == jj.c.bar_col1) # test column directly against itself + # joins necessarily have to prefix column names with the name + # of the selectable, else the same-named columns will overwrite + # one another. In this case, we unfortunately have this unfriendly + # "anonymous" name, whereas before when select() could be a FROM + # the "bar_col1" label would be directly in the join() object. However + # this was a useless join() object because PG and MySQL don't accept + # unnamed subqueries in joins in any case. + name = "%s_bar_col1" % (jj.name,) + assert jjj.corresponding_column(jjj.c.table1_col1) is jjj.c.table1_col1 - assert jjj.corresponding_column(jj.c.bar_col1) is jjj.c.bar_col1 + assert jjj.corresponding_column(jj.c.bar_col1) is jjj.c[name] # test alias of the join @@ -267,13 +340,15 @@ class SelectableTest( def test_clone_append_column(self): sel = select([literal_column("1").label("a")]) - eq_(list(sel.c.keys()), ["a"]) + eq_(list(sel.selected_columns.keys()), ["a"]) cloned = visitors.ReplacingCloningVisitor().traverse(sel) cloned.append_column(literal_column("2").label("b")) cloned.append_column(func.foo()) - eq_(list(cloned.c.keys()), ["a", "b", "foo()"]) + eq_(list(cloned.selected_columns.keys()), ["a", "b", "foo()"]) - def test_append_column_after_replace_selectable(self): + def test_append_column_after_visitor_replace(self): + # test for a supported idiom that matches the deprecated / removed + # replace_selectable method basesel = select([literal_column("1").label("a")]) tojoin = select( [literal_column("1").label("a"), literal_column("2").label("b")] @@ -281,9 +356,14 @@ class SelectableTest( basefrom = basesel.alias("basefrom") joinfrom = tojoin.alias("joinfrom") sel = select([basefrom.c.a]) - replaced = sel.replace_selectable( - basefrom, basefrom.join(joinfrom, basefrom.c.a == joinfrom.c.a) - ) + + replace_from = basefrom.join(joinfrom, basefrom.c.a == joinfrom.c.a) + + def replace(elem): + if elem is basefrom: + return replace_from + + replaced = visitors.replacement_traverse(sel, {}, replace) self.assert_compile( replaced, "SELECT basefrom.a FROM (SELECT 1 AS a) AS basefrom " @@ -302,7 +382,7 @@ class SelectableTest( # test that corresponding column digs across # clone boundaries with anonymous labeled elements col = func.count().label("foo") - sel = select([col]) + sel = select([col]).subquery() sel2 = visitors.ReplacingCloningVisitor().traverse(sel) assert sel2.corresponding_column(col) is sel2.c.foo @@ -323,13 +403,16 @@ class SelectableTest( impl = Integer stmt = select([type_coerce(column("x"), MyType).label("foo")]) - stmt2 = stmt.select() + subq = stmt.subquery() + stmt2 = subq.select() + subq2 = stmt2.subquery() assert isinstance(stmt._raw_columns[0].type, MyType) - assert isinstance(stmt.c.foo.type, MyType) - assert isinstance(stmt2.c.foo.type, MyType) + assert isinstance(subq.c.foo.type, MyType) + assert isinstance(stmt2.selected_columns.foo.type, MyType) + assert isinstance(subq2.c.foo.type, MyType) - def test_select_on_table(self): - sel = select([table1, table2], use_labels=True) + def test_subquery_on_table(self): + sel = select([table1, table2], use_labels=True).subquery() assert sel.corresponding_column(table1.c.col1) is sel.c.table1_col1 assert ( @@ -383,7 +466,7 @@ class SelectableTest( "AS a", ) - def test_union(self): + def test_union_correspondence(self): # tests that we can correspond a column in a Select statement # with a certain Table, against a column in a Union where one of @@ -411,8 +494,34 @@ class SelectableTest( s1 = table1.select(use_labels=True) s2 = table2.select(use_labels=True) - assert u.corresponding_column(s1.c.table1_col2) is u.c.col2 - assert u.corresponding_column(s2.c.table2_col2) is u.c.col2 + assert ( + u.corresponding_column(s1.selected_columns.table1_col2) + is u.selected_columns.col2 + ) + + # right now, the "selected_columns" of a union are those of the + # first selectable. so without using a subquery that represents + # all the SELECTs in the union, we can't do corresponding column + # like this. perhaps compoundselect shouldn't even implement + # .corresponding_column directly + assert ( + u.corresponding_column(s2.selected_columns.table2_col2) is None + ) # really? u.selected_columns.col2 + + usub = u.subquery() + assert ( + usub.corresponding_column(s1.selected_columns.table1_col2) + is usub.c.col2 + ) + assert ( + usub.corresponding_column(s2.selected_columns.table2_col2) + is usub.c.col2 + ) + + s1sub = s1.subquery() + s2sub = s2.subquery() + assert usub.corresponding_column(s1sub.c.table1_col2) is usub.c.col2 + assert usub.corresponding_column(s2sub.c.table2_col2) is usub.c.col2 def test_union_precedence(self): # conflicting column correspondence should be resolved based on @@ -423,11 +532,11 @@ class SelectableTest( s3 = select([table1.c.col3, table1.c.colx]) s4 = select([table1.c.colx, table1.c.col3]) - u1 = union(s1, s2) + u1 = union(s1, s2).subquery() assert u1.corresponding_column(table1.c.col1) is u1.c.col1 assert u1.corresponding_column(table1.c.col2) is u1.c.col2 - u1 = union(s1, s2, s3, s4) + u1 = union(s1, s2, s3, s4).subquery() assert u1.corresponding_column(table1.c.col1) is u1.c.col1 assert u1.corresponding_column(table1.c.col2) is u1.c.col2 assert u1.corresponding_column(table1.c.colx) is u1.c.col2 @@ -437,12 +546,12 @@ class SelectableTest( s1 = select([table1.c.col1, table1.c.col2]) s2 = select([table1.c.col2, table1.c.col1]) - for c in s1.c: + for c in s1.selected_columns: c.proxy_set - for c in s2.c: + for c in s2.selected_columns: c.proxy_set - u1 = union(s1, s2) + u1 = union(s1, s2).subquery() assert u1.corresponding_column(table1.c.col2) is u1.c.col2 def test_singular_union(self): @@ -451,9 +560,9 @@ class SelectableTest( select([table1.c.col1, table1.c.col2, table1.c.col3]), ) u = union(select([table1.c.col1, table1.c.col2, table1.c.col3])) - assert u.c.col1 is not None - assert u.c.col2 is not None - assert u.c.col3 is not None + assert u.selected_columns.col1 is not None + assert u.selected_columns.col2 is not None + assert u.selected_columns.col3 is not None def test_alias_union(self): @@ -482,8 +591,8 @@ class SelectableTest( ) .alias("analias") ) - s1 = table1.select(use_labels=True) - s2 = table2.select(use_labels=True) + s1 = table1.select(use_labels=True).subquery() + s2 = table2.select(use_labels=True).subquery() assert u.corresponding_column(s1.c.table1_col2) is u.c.col2 assert u.corresponding_column(s2.c.table2_col2) is u.c.col2 assert u.corresponding_column(s2.c.table2_coly) is u.c.coly @@ -493,13 +602,15 @@ class SelectableTest( s1 = select([table1.c.col1, table1.c.col2]) s2 = select([table1.c.col1, table1.c.col2]).alias() - u1 = union(s1, s2) - assert u1.corresponding_column(s1.c.col1) is u1.c.col1 - assert u1.corresponding_column(s2.c.col1) is u1.c.col1 - - u2 = union(s2, s1) - assert u2.corresponding_column(s1.c.col1) is u2.c.col1 - assert u2.corresponding_column(s2.c.col1) is u2.c.col1 + # previously this worked + assert_raises_message( + exc.ArgumentError, + "SELECT construct for inclusion in a UNION or " + "other set construct expected", + union, + s1, + s2, + ) def test_union_of_text(self): s1 = select([table1.c.col1, table1.c.col2]) @@ -507,77 +618,130 @@ class SelectableTest( column("col1"), column("col2") ) - u1 = union(s1, s2) - assert u1.corresponding_column(s1.c.col1) is u1.c.col1 - assert u1.corresponding_column(s2.c.col1) is u1.c.col1 + u1 = union(s1, s2).subquery() + assert u1.corresponding_column(s1.selected_columns.col1) is u1.c.col1 + assert u1.corresponding_column(s2.selected_columns.col1) is u1.c.col1 + + u2 = union(s2, s1).subquery() + assert u2.corresponding_column(s1.selected_columns.col1) is u2.c.col1 + assert u2.corresponding_column(s2.selected_columns.col1) is u2.c.col1 + + def test_foo(self): + s1 = select([table1.c.col1, table1.c.col2]) + s2 = select([table1.c.col2, table1.c.col1]) + + u1 = union(s1, s2).subquery() + assert u1.corresponding_column(table1.c.col2) is u1.c.col2 + + metadata = MetaData() + table1_new = Table( + "table1", + metadata, + Column("col1", Integer, primary_key=True), + Column("col2", String(20)), + Column("col3", Integer), + Column("colx", Integer), + ) + # table1_new = table1 + + s1 = select([table1_new.c.col1, table1_new.c.col2]) + s2 = select([table1_new.c.col2, table1_new.c.col1]) + u1 = union(s1, s2).subquery() - u2 = union(s2, s1) - assert u2.corresponding_column(s1.c.col1) is u2.c.col1 - assert u2.corresponding_column(s2.c.col1) is u2.c.col1 + # TODO: failing due to proxy_set not correct + assert u1.corresponding_column(table1_new.c.col2) is u1.c.col2 - @testing.emits_warning("Column 'col1'") - def test_union_dupe_keys(self): + def test_union_alias_dupe_keys(self): s1 = select([table1.c.col1, table1.c.col2, table2.c.col1]) s2 = select([table2.c.col1, table2.c.col2, table2.c.col3]) - u1 = union(s1, s2) + u1 = union(s1, s2).subquery() + + with testing.expect_warnings("Column 'col1'"): + u1.c assert ( - u1.corresponding_column(s1.c._all_columns[0]) + u1.corresponding_column(s1.selected_columns._all_columns[0]) is u1.c._all_columns[0] ) - assert u1.corresponding_column(s2.c.col1) is u1.c._all_columns[0] - assert u1.corresponding_column(s1.c.col2) is u1.c.col2 - assert u1.corresponding_column(s2.c.col2) is u1.c.col2 - assert u1.corresponding_column(s2.c.col3) is u1.c._all_columns[2] + # due to the duplicate key, "col1" is now the column at the end + # of the list and the first column is not accessible by key + assert u1.c.col1 is u1.c._all_columns[2] + # table2.c.col1 is in two positions in this union, so...currently + # it is the replaced one at position 2. assert u1.corresponding_column(table2.c.col1) is u1.c._all_columns[2] - assert u1.corresponding_column(table2.c.col3) is u1.c._all_columns[2] - @testing.emits_warning("Column 'col1'") - def test_union_alias_dupe_keys(self): - s1 = select([table1.c.col1, table1.c.col2, table2.c.col1]).alias() - s2 = select([table2.c.col1, table2.c.col2, table2.c.col3]) - u1 = union(s1, s2) + # this is table2.c.col1 in both cases, so this is "right" + assert u1.corresponding_column(s2.selected_columns.col1) is u1.c.col1 + # same + assert u1.corresponding_column(s2.subquery().c.col1) is u1.c.col1 + + # col2 is working OK + assert u1.corresponding_column(s1.selected_columns.col2) is u1.c.col2 assert ( - u1.corresponding_column(s1.c._all_columns[0]) - is u1.c._all_columns[0] + u1.corresponding_column(s1.selected_columns.col2) + is u1.c._all_columns[1] ) - assert u1.corresponding_column(s2.c.col1) is u1.c._all_columns[0] - assert u1.corresponding_column(s1.c.col2) is u1.c.col2 - assert u1.corresponding_column(s2.c.col2) is u1.c.col2 + assert u1.corresponding_column(s2.selected_columns.col2) is u1.c.col2 + assert ( + u1.corresponding_column(s2.selected_columns.col2) + is u1.c._all_columns[1] + ) + assert u1.corresponding_column(s2.subquery().c.col2) is u1.c.col2 - assert u1.corresponding_column(s2.c.col3) is u1.c._all_columns[2] + # col3 is also "correct" , though confusing + assert u1.corresponding_column(s2.selected_columns.col3) is u1.c.col1 - # this differs from the non-alias test because table2.c.col1 is - # more directly at s2.c.col1 than it is s1.c.col1. - assert u1.corresponding_column(table2.c.col1) is u1.c._all_columns[0] + assert u1.corresponding_column(table1.c.col1) is u1.c._all_columns[0] + assert u1.corresponding_column(table1.c.col2) is u1.c._all_columns[1] + assert u1.corresponding_column(table2.c.col1) is u1.c._all_columns[2] + assert u1.corresponding_column(table2.c.col2) is u1.c._all_columns[1] assert u1.corresponding_column(table2.c.col3) is u1.c._all_columns[2] - @testing.emits_warning("Column 'col1'") def test_union_alias_dupe_keys_grouped(self): - s1 = ( - select([table1.c.col1, table1.c.col2, table2.c.col1]) - .limit(1) - .alias() - ) + s1 = select([table1.c.col1, table1.c.col2, table2.c.col1]).limit(1) s2 = select([table2.c.col1, table2.c.col2, table2.c.col3]).limit(1) - u1 = union(s1, s2) + u1 = union(s1, s2).subquery() + + with testing.expect_warnings("Column 'col1'"): + u1.c + + # due to the duplicate key, "col1" is now the column at the end + # of the list and the first column is not accessible by key + assert u1.c.col1 is u1.c._all_columns[2] + + # table2.c.col1 is in two positions in this union, so...currently + # it is the replaced one at position 2. + assert u1.corresponding_column(table2.c.col1) is u1.c._all_columns[2] + + # this is table2.c.col1 in both cases, so this is "right" + assert u1.corresponding_column(s2.selected_columns.col1) is u1.c.col1 + # same + assert u1.corresponding_column(s2.subquery().c.col1) is u1.c.col1 + + # col2 is working OK + assert u1.corresponding_column(s1.selected_columns.col2) is u1.c.col2 assert ( - u1.corresponding_column(s1.c._all_columns[0]) - is u1.c._all_columns[0] + u1.corresponding_column(s1.selected_columns.col2) + is u1.c._all_columns[1] + ) + assert u1.corresponding_column(s2.selected_columns.col2) is u1.c.col2 + assert ( + u1.corresponding_column(s2.selected_columns.col2) + is u1.c._all_columns[1] ) - assert u1.corresponding_column(s2.c.col1) is u1.c._all_columns[0] - assert u1.corresponding_column(s1.c.col2) is u1.c.col2 - assert u1.corresponding_column(s2.c.col2) is u1.c.col2 + assert u1.corresponding_column(s2.subquery().c.col2) is u1.c.col2 - assert u1.corresponding_column(s2.c.col3) is u1.c._all_columns[2] + # col3 is also "correct" , though confusing + assert u1.corresponding_column(s2.selected_columns.col3) is u1.c.col1 - # this differs from the non-alias test because table2.c.col1 is - # more directly at s2.c.col1 than it is s1.c.col1. - assert u1.corresponding_column(table2.c.col1) is u1.c._all_columns[0] + assert u1.corresponding_column(table1.c.col1) is u1.c._all_columns[0] + assert u1.corresponding_column(table1.c.col2) is u1.c._all_columns[1] + assert u1.corresponding_column(table2.c.col1) is u1.c._all_columns[2] + assert u1.corresponding_column(table2.c.col2) is u1.c._all_columns[1] assert u1.corresponding_column(table2.c.col3) is u1.c._all_columns[2] def test_select_union(self): @@ -607,9 +771,9 @@ class SelectableTest( ) .alias("analias") ) - s = select([u]) - s1 = table1.select(use_labels=True) - s2 = table2.select(use_labels=True) + s = select([u]).subquery() + s1 = table1.select(use_labels=True).subquery() + s2 = table2.select(use_labels=True).subquery() assert s.corresponding_column(s1.c.table1_col2) is s.c.col2 assert s.corresponding_column(s2.c.table2_col2) is s.c.col2 @@ -653,15 +817,15 @@ class SelectableTest( criterion = a.c.table1_col1 == b.c.col2 self.assert_(criterion.compare(j.onclause)) - def test_select_alias(self): + def test_select_subquery_join(self): a = table1.select().alias("a") j = join(a, table2) criterion = a.c.col1 == table2.c.col2 self.assert_(criterion.compare(j.onclause)) - def test_select_labels(self): - a = table1.select(use_labels=True) + def test_subquery_labels_join(self): + a = table1.select(use_labels=True).subquery() j = join(a, table2) criterion = a.c.table1_col1 == table2.c.col2 @@ -683,18 +847,27 @@ class SelectableTest( table1.c.col2.label("acol2"), table1.c.col3.label("acol3"), ] - ) + ).subquery() j = join(a, table2) criterion = a.c.acol1 == table2.c.col2 self.assert_(criterion.compare(j.onclause)) - def test_labeled_select_correspoinding(self): + def test_labeled_select_corresponding(self): l1 = select([func.max(table1.c.col1)]).label("foo") s = select([l1]) - eq_(s.corresponding_column(l1), s.c.foo) + eq_(s.corresponding_column(l1), s.selected_columns.foo) s = select([table1.c.col1, l1]) + eq_(s.corresponding_column(l1), s.selected_columns.foo) + + def test_labeled_subquery_corresponding(self): + l1 = select([func.max(table1.c.col1)]).label("foo") + s = select([l1]).subquery() + + eq_(s.corresponding_column(l1), s.c.foo) + + s = select([table1.c.col1, l1]).subquery() eq_(s.corresponding_column(l1), s.c.foo) def test_select_alias_labels(self): @@ -724,7 +897,7 @@ class SelectableTest( t2 = Table("t2", m, Column("id", Integer, ForeignKey("t1.id"))) t3 = Table("t3", m2, Column("id", Integer, ForeignKey("t1.id2"))) - s = select([t2, t3], use_labels=True) + s = select([t2, t3], use_labels=True).subquery() assert_raises(exc.NoReferencedTableError, s.join, t1) @@ -732,19 +905,19 @@ class SelectableTest( # See [ticket:2167] for this one. l1 = table1.c.col1.label("a") l2 = select([l1]).label("b") - s = select([l2]) + s = select([l2]).subquery() assert s.c.b is not None self.assert_compile( s.select(), - "SELECT b FROM " - "(SELECT (SELECT table1.col1 AS a FROM table1) AS b)", + "SELECT anon_1.b FROM " + "(SELECT (SELECT table1.col1 AS a FROM table1) AS b) AS anon_1", ) - s2 = select([s.label("c")]) + s2 = select([s.element.label("c")]).subquery() self.assert_compile( s2.select(), - "SELECT c FROM (SELECT (SELECT (" - "SELECT table1.col1 AS a FROM table1) AS b) AS c)", + "SELECT anon_1.c FROM (SELECT (SELECT (" + "SELECT table1.col1 AS a FROM table1) AS b) AS c) AS anon_1", ) def test_self_referential_select_raises(self): @@ -752,7 +925,8 @@ class SelectableTest( s = select([t]) - s.append_whereclause(s.c.x > 5) + with testing.expect_deprecated("The SelectBase.c"): + s.append_whereclause(s.c.x > 5) assert_raises_message( exc.InvalidRequestError, r"select\(\) construct refers to itself as a FROM", @@ -762,7 +936,7 @@ class SelectableTest( def test_unusual_column_elements_text(self): """test that .c excludes text().""" - s = select([table1.c.col1, text("foo")]) + s = select([table1.c.col1, text("foo")]).subquery() eq_(list(s.c), [s.c.col1]) def test_unusual_column_elements_clauselist(self): @@ -770,14 +944,16 @@ class SelectableTest( from sqlalchemy.sql.expression import ClauseList - s = select([table1.c.col1, ClauseList(table1.c.col2, table1.c.col3)]) + s = select( + [table1.c.col1, ClauseList(table1.c.col2, table1.c.col3)] + ).subquery() eq_(list(s.c), [s.c.col1, s.c.col2, s.c.col3]) def test_unusual_column_elements_boolean_clauselist(self): """test that BooleanClauseList is placed as single element in .c.""" c2 = and_(table1.c.col2 == 5, table1.c.col3 == 4) - s = select([table1.c.col1, c2]) + s = select([table1.c.col1, c2]).subquery() eq_(list(s.c), [s.c.col1, s.corresponding_column(c2)]) def test_from_list_deferred_constructor(self): @@ -936,11 +1112,11 @@ class RefreshForNewColTest(fixtures.TestBase): a = table("a", column("x")) b = table("b", column("y")) s = select([a, b]).apply_labels() - s.c + s.selected_columns q = column("x") b.append_column(q) s._refresh_for_new_column(q) - assert q in s.c.b_x.proxy_set + assert q in s.selected_columns.b_x.proxy_set def test_alias_alias_samename_init(self): a = table("a", column("x")) @@ -954,8 +1130,11 @@ class RefreshForNewColTest(fixtures.TestBase): q = column("x") b.append_column(q) + assert "_columns" in s2.__dict__ + s2._refresh_for_new_column(q) + assert "_columns" not in s2.__dict__ is_(s1.corresponding_column(s2.c.b_x), s1.c.b_x) def test_aliased_select_samename_uninit(self): @@ -1005,23 +1184,18 @@ class RefreshForNewColTest(fixtures.TestBase): q = column("q") a.append_column(q) s3._refresh_for_new_column(q) - assert a.c.q in s3.c.q.proxy_set + assert a.c.q in s3.selected_columns.q.proxy_set - def test_union_init_raises(self): + def test_union_init(self): a = table("a", column("x")) s1 = select([a]) s2 = select([a]) s3 = s1.union(s2) - s3.c + s3.selected_columns q = column("q") a.append_column(q) - assert_raises_message( - NotImplementedError, - "CompoundSelect constructs don't support addition of " - "columns to underlying selectables", - s3._refresh_for_new_column, - q, - ) + s3._refresh_for_new_column(q) + assert a.c.q in s3.selected_columns.q.proxy_set def test_nested_join_uninit(self): a = table("a", column("x")) @@ -1195,7 +1369,7 @@ class JoinAliasingTest(fixtures.TestBase, AssertsCompiledSQL): class JoinConditionTest(fixtures.TestBase, AssertsCompiledSQL): __dialect__ = "default" - def test_join_condition(self): + def test_join_condition_one(self): m = MetaData() t1 = Table("t1", m, Column("id", Integer)) t2 = Table( @@ -1211,13 +1385,6 @@ class JoinConditionTest(fixtures.TestBase, AssertsCompiledSQL): t4 = Table( "t4", m, Column("id", Integer), Column("t2id", ForeignKey("t2.id")) ) - t5 = Table( - "t5", - m, - Column("t1id1", ForeignKey("t1.id")), - Column("t1id2", ForeignKey("t1.id")), - ) - t1t2 = t1.join(t2) t2t3 = t2.join(t3) @@ -1235,6 +1402,32 @@ class JoinConditionTest(fixtures.TestBase, AssertsCompiledSQL): sql_util.join_condition(left, right, a_subset=a_subset) ) + def test_join_condition_two(self): + m = MetaData() + t1 = Table("t1", m, Column("id", Integer)) + t2 = Table( + "t2", m, Column("id", Integer), Column("t1id", ForeignKey("t1.id")) + ) + t3 = Table( + "t3", + m, + Column("id", Integer), + Column("t1id", ForeignKey("t1.id")), + Column("t2id", ForeignKey("t2.id")), + ) + t4 = Table( + "t4", m, Column("id", Integer), Column("t2id", ForeignKey("t2.id")) + ) + t5 = Table( + "t5", + m, + Column("t1id1", ForeignKey("t1.id")), + Column("t1id2", ForeignKey("t1.id")), + ) + + t1t2 = t1.join(t2) + t2t3 = t2.join(t3) + # these are ambiguous, or have no joins for left, right, a_subset in [ (t1t2, t3, None), @@ -1242,7 +1435,7 @@ class JoinConditionTest(fixtures.TestBase, AssertsCompiledSQL): (t1, t4, None), (t1t2, t2t3, None), (t5, t1, None), - (t5.select(use_labels=True), t1, None), + (t5.select(use_labels=True).subquery(), t1, None), ]: assert_raises( exc.ArgumentError, @@ -1252,6 +1445,24 @@ class JoinConditionTest(fixtures.TestBase, AssertsCompiledSQL): a_subset=a_subset, ) + def test_join_condition_three(self): + m = MetaData() + t1 = Table("t1", m, Column("id", Integer)) + t2 = Table( + "t2", m, Column("id", Integer), Column("t1id", ForeignKey("t1.id")) + ) + t3 = Table( + "t3", + m, + Column("id", Integer), + Column("t1id", ForeignKey("t1.id")), + Column("t2id", ForeignKey("t2.id")), + ) + t4 = Table( + "t4", m, Column("id", Integer), Column("t2id", ForeignKey("t2.id")) + ) + t1t2 = t1.join(t2) + t2t3 = t2.join(t3) als = t2t3.alias() # test join's behavior, including natural for left, right, expected in [ @@ -1266,6 +1477,22 @@ class JoinConditionTest(fixtures.TestBase, AssertsCompiledSQL): ]: assert expected.compare(left.join(right).onclause) + def test_join_condition_four(self): + m = MetaData() + t1 = Table("t1", m, Column("id", Integer)) + t2 = Table( + "t2", m, Column("id", Integer), Column("t1id", ForeignKey("t1.id")) + ) + t3 = Table( + "t3", + m, + Column("id", Integer), + Column("t1id", ForeignKey("t1.id")), + Column("t2id", ForeignKey("t2.id")), + ) + t1t2 = t1.join(t2) + t2t3 = t2.join(t3) + # these are right-nested joins j = t1t2.join(t2t3) assert j.onclause.compare(t2.c.id == t3.c.t2id) @@ -1275,7 +1502,23 @@ class JoinConditionTest(fixtures.TestBase, AssertsCompiledSQL): "(t2 JOIN t3 ON t2.id = t3.t2id) ON t2.id = t3.t2id", ) - st2t3 = t2t3.select(use_labels=True) + def test_join_condition_five(self): + m = MetaData() + t1 = Table("t1", m, Column("id", Integer)) + t2 = Table( + "t2", m, Column("id", Integer), Column("t1id", ForeignKey("t1.id")) + ) + t3 = Table( + "t3", + m, + Column("id", Integer), + Column("t1id", ForeignKey("t1.id")), + Column("t2id", ForeignKey("t2.id")), + ) + t1t2 = t1.join(t2) + t2t3 = t2.join(t3) + + st2t3 = t2t3.select(use_labels=True).subquery() j = t1t2.join(st2t3) assert j.onclause.compare(t2.c.id == st2t3.c.t3_t2id) self.assert_compile( @@ -1283,7 +1526,8 @@ class JoinConditionTest(fixtures.TestBase, AssertsCompiledSQL): "t1 JOIN t2 ON t1.id = t2.t1id JOIN " "(SELECT t2.id AS t2_id, t2.t1id AS t2_t1id, " "t3.id AS t3_id, t3.t1id AS t3_t1id, t3.t2id AS t3_t2id " - "FROM t2 JOIN t3 ON t2.id = t3.t2id) ON t2.id = t3_t2id", + "FROM t2 JOIN t3 ON t2.id = t3.t2id) AS anon_1 " + "ON t2.id = anon_1.t3_t2id", ) def test_join_multiple_equiv_fks(self): @@ -1582,8 +1826,10 @@ class ReduceTest(fixtures.TestBase, AssertsExecutionResults): Column("manager_id", Integer, primary_key=True), Column("manager_name", String(50)), ) - s = select([engineers, managers]).where( - engineers.c.engineer_name == managers.c.manager_name + s = ( + select([engineers, managers]) + .where(engineers.c.engineer_name == managers.c.manager_name) + .subquery() ) eq_( util.column_set(sql_util.reduce_columns(list(s.c), s)), @@ -1629,7 +1875,16 @@ class ReduceTest(fixtures.TestBase, AssertsExecutionResults): ) s1 = select([t1, t2]) s1 = s1.reduce_columns(only_synonyms=True) - eq_(set(s1.c), set([s1.c.x, s1.c.y, s1.c.q])) + eq_( + set(s1.selected_columns), + set( + [ + s1.selected_columns.x, + s1.selected_columns.y, + s1.selected_columns.q, + ] + ), + ) def test_reduce_only_synonym_lineage(self): m = MetaData() @@ -1642,7 +1897,7 @@ class ReduceTest(fixtures.TestBase, AssertsExecutionResults): ) # test that the first appearance in the columns clause # wins - t1 is first, t1.c.x wins - s1 = select([t1]) + s1 = select([t1]).subquery() s2 = select([t1, s1]).where(t1.c.x == s1.c.x).where(s1.c.y == t1.c.z) eq_( set(s2.reduce_columns().inner_columns), @@ -1650,7 +1905,7 @@ class ReduceTest(fixtures.TestBase, AssertsExecutionResults): ) # reverse order, s1.c.x wins - s1 = select([t1]) + s1 = select([t1]).subquery() s2 = select([s1, t1]).where(t1.c.x == s1.c.x).where(s1.c.y == t1.c.z) eq_( set(s2.reduce_columns().inner_columns), @@ -1939,7 +2194,7 @@ class AnnotationsTest(fixtures.TestBase): assert (t.c.x == 5).compare(x_a == 5) assert not (t.c.y == 5).compare(x_a == 5) - s = select([t]) + s = select([t]).subquery() x_p = s.c.x assert not x_a.compare(x_p) assert not t.c.x.compare(x_p) @@ -2025,12 +2280,12 @@ class AnnotationsTest(fixtures.TestBase): _constructor = Column t1 = Table("t1", MetaData(), MyColumn()) - s1 = t1.select() + s1 = t1.select().subquery() assert isinstance(t1.c.foo, MyColumn) assert isinstance(s1.c.foo, Column) annot_1 = t1.c.foo._annotate({}) - s2 = select([annot_1]) + s2 = select([annot_1]).subquery() assert isinstance(s2.c.foo, Column) annot_2 = s1._annotate({}) assert isinstance(annot_2.c.foo, Column) @@ -2064,7 +2319,7 @@ class AnnotationsTest(fixtures.TestBase): def test_annotated_corresponding_column(self): table1 = table("table1", column("col1")) - s1 = select([table1.c.col1]) + s1 = select([table1.c.col1]).subquery() t1 = s1._annotate({}) t2 = s1 @@ -2075,7 +2330,7 @@ class AnnotationsTest(fixtures.TestBase): assert t1.c is t2.c assert t1.c.col1 is t2.c.col1 - inner = select([s1]) + inner = select([s1]).subquery() assert ( inner.corresponding_column(t2.c.col1, require_embedded=False) @@ -2278,7 +2533,7 @@ class AnnotationsTest(fixtures.TestBase): a1 = table1.alias() s = select([a1.c.x]).select_from(a1.join(table2, a1.c.x == table2.c.y)) - assert_s = select([select([s])]) + assert_s = select([select([s.subquery()]).subquery()]) for fn in ( sql_util._deep_deannotate, lambda s: sql_util._deep_annotate(s, {"foo": "bar"}), @@ -2286,7 +2541,7 @@ class AnnotationsTest(fixtures.TestBase): lambda s: visitors.replacement_traverse(s, {}, lambda x: None), ): - sel = fn(select([fn(select([fn(s)]))])) + sel = fn(select([fn(select([fn(s.subquery())]).subquery())])) eq_(str(assert_s), str(sel)) def test_bind_unique_test(self): @@ -2359,7 +2614,7 @@ class WithLabelsTest(fixtures.TestBase): assert_raises_message( exc.SAWarning, r"replaced by Column.*, which has the same key", - lambda: s.c, + lambda: s.subquery().c, ) def _assert_result_keys(self, s, keys): @@ -2367,7 +2622,7 @@ class WithLabelsTest(fixtures.TestBase): eq_(set(compiled._create_result_map()), set(keys)) def _assert_subq_result_keys(self, s, keys): - compiled = s.select().compile() + compiled = s.subquery().select().compile() eq_(set(compiled._create_result_map()), set(keys)) def _names_overlap(self): @@ -2383,7 +2638,8 @@ class WithLabelsTest(fixtures.TestBase): def test_names_overlap_label(self): sel = self._names_overlap().apply_labels() - eq_(list(sel.c.keys()), ["t1_x", "t2_x"]) + eq_(list(sel.selected_columns.keys()), ["t1_x", "t2_x"]) + eq_(list(sel.subquery().c.keys()), ["t1_x", "t2_x"]) self._assert_result_keys(sel, ["t1_x", "t2_x"]) def _names_overlap_keys_dont(self): @@ -2394,12 +2650,14 @@ class WithLabelsTest(fixtures.TestBase): def test_names_overlap_keys_dont_nolabel(self): sel = self._names_overlap_keys_dont() - eq_(list(sel.c.keys()), ["a", "b"]) + eq_(list(sel.selected_columns.keys()), ["a", "b"]) + eq_(list(sel.subquery().c.keys()), ["a", "b"]) self._assert_result_keys(sel, ["x"]) def test_names_overlap_keys_dont_label(self): sel = self._names_overlap_keys_dont().apply_labels() - eq_(list(sel.c.keys()), ["t1_a", "t2_b"]) + eq_(list(sel.selected_columns.keys()), ["t1_a", "t2_b"]) + eq_(list(sel.subquery().c.keys()), ["t1_a", "t2_b"]) self._assert_result_keys(sel, ["t1_x", "t2_x"]) def _labels_overlap(self): @@ -2410,13 +2668,15 @@ class WithLabelsTest(fixtures.TestBase): def test_labels_overlap_nolabel(self): sel = self._labels_overlap() - eq_(list(sel.c.keys()), ["x_id", "id"]) + eq_(list(sel.selected_columns.keys()), ["x_id", "id"]) + eq_(list(sel.subquery().c.keys()), ["x_id", "id"]) self._assert_result_keys(sel, ["x_id", "id"]) def test_labels_overlap_label(self): sel = self._labels_overlap().apply_labels() t2 = sel.froms[1] - eq_(list(sel.c.keys()), ["t_x_id", t2.c.id.anon_label]) + eq_(list(sel.selected_columns.keys()), ["t_x_id", t2.c.id.anon_label]) + eq_(list(sel.subquery().c.keys()), ["t_x_id", t2.c.id.anon_label]) self._assert_result_keys(sel, ["t_x_id", "id_1"]) self._assert_subq_result_keys(sel, ["t_x_id", "id_1"]) @@ -2428,12 +2688,14 @@ class WithLabelsTest(fixtures.TestBase): def test_labels_overlap_keylabels_dont_nolabel(self): sel = self._labels_overlap_keylabels_dont() - eq_(list(sel.c.keys()), ["a", "b"]) + eq_(list(sel.selected_columns.keys()), ["a", "b"]) + eq_(list(sel.subquery().c.keys()), ["a", "b"]) self._assert_result_keys(sel, ["x_id", "id"]) def test_labels_overlap_keylabels_dont_label(self): sel = self._labels_overlap_keylabels_dont().apply_labels() - eq_(list(sel.c.keys()), ["t_a", "t_x_b"]) + eq_(list(sel.selected_columns.keys()), ["t_a", "t_x_b"]) + eq_(list(sel.subquery().c.keys()), ["t_a", "t_x_b"]) self._assert_result_keys(sel, ["t_x_id", "id_1"]) def _keylabels_overlap_labels_dont(self): @@ -2444,13 +2706,15 @@ class WithLabelsTest(fixtures.TestBase): def test_keylabels_overlap_labels_dont_nolabel(self): sel = self._keylabels_overlap_labels_dont() - eq_(list(sel.c.keys()), ["x_id", "id"]) + eq_(list(sel.selected_columns.keys()), ["x_id", "id"]) + eq_(list(sel.subquery().c.keys()), ["x_id", "id"]) self._assert_result_keys(sel, ["a", "b"]) def test_keylabels_overlap_labels_dont_label(self): sel = self._keylabels_overlap_labels_dont().apply_labels() t2 = sel.froms[1] - eq_(list(sel.c.keys()), ["t_x_id", t2.c.id.anon_label]) + eq_(list(sel.selected_columns.keys()), ["t_x_id", t2.c.id.anon_label]) + eq_(list(sel.subquery().c.keys()), ["t_x_id", t2.c.id.anon_label]) self._assert_result_keys(sel, ["t_a", "t_x_b"]) self._assert_subq_result_keys(sel, ["t_a", "t_x_b"]) @@ -2462,14 +2726,16 @@ class WithLabelsTest(fixtures.TestBase): def test_keylabels_overlap_labels_overlap_nolabel(self): sel = self._keylabels_overlap_labels_overlap() - eq_(list(sel.c.keys()), ["x_a", "a"]) + eq_(list(sel.selected_columns.keys()), ["x_a", "a"]) + eq_(list(sel.subquery().c.keys()), ["x_a", "a"]) self._assert_result_keys(sel, ["x_id", "id"]) self._assert_subq_result_keys(sel, ["x_id", "id"]) def test_keylabels_overlap_labels_overlap_label(self): sel = self._keylabels_overlap_labels_overlap().apply_labels() t2 = sel.froms[1] - eq_(list(sel.c.keys()), ["t_x_a", t2.c.a.anon_label]) + eq_(list(sel.selected_columns.keys()), ["t_x_a", t2.c.a.anon_label]) + eq_(list(sel.subquery().c.keys()), ["t_x_a", t2.c.a.anon_label]) self._assert_result_keys(sel, ["t_x_id", "id_1"]) self._assert_subq_result_keys(sel, ["t_x_id", "id_1"]) @@ -2486,7 +2752,8 @@ class WithLabelsTest(fixtures.TestBase): def test_keys_overlap_names_dont_label(self): sel = self._keys_overlap_names_dont().apply_labels() - eq_(list(sel.c.keys()), ["t1_x", "t2_x"]) + eq_(list(sel.selected_columns.keys()), ["t1_x", "t2_x"]) + eq_(list(sel.subquery().c.keys()), ["t1_x", "t2_x"]) self._assert_result_keys(sel, ["t1_a", "t2_b"]) @@ -2653,38 +2920,24 @@ class ForUpdateTest(fixtures.TestBase, AssertsCompiledSQL): class AliasTest(fixtures.TestBase, AssertsCompiledSQL): __dialect__ = "default" - def test_legacy_original_accessor(self): - t = table("t", column("c")) - a1 = t.alias() - a2 = a1.alias() - a3 = a2.alias() - - is_(a1.original, t) - is_(a2.original, t) - is_(a3.original, t) - - def test_wrapped(self): + def test_direct_element_hierarchy(self): t = table("t", column("c")) a1 = t.alias() a2 = a1.alias() a3 = a2.alias() is_(a1.element, t) - is_(a2.element, t) - is_(a3.element, t) - - is_(a3.wrapped, a2) - is_(a2.wrapped, a1) - is_(a1.wrapped, t) + is_(a2.element, a1) + is_(a3.element, a2) - def test_get_children_preserves_wrapped(self): + def test_get_children_preserves_multiple_nesting(self): t = table("t", column("c")) stmt = select([t]) a1 = stmt.alias() a2 = a1.alias() eq_(set(a2.get_children(column_collections=False)), {a1}) - def test_wrapped_correspondence(self): + def test_correspondence_multiple_nesting(self): t = table("t", column("c")) stmt = select([t]) a1 = stmt.alias() @@ -2692,15 +2945,12 @@ class AliasTest(fixtures.TestBase, AssertsCompiledSQL): is_(a1.corresponding_column(a2.c.c), a1.c.c) - def test_copy_internals_preserves_wrapped(self): + def test_copy_internals_multiple_nesting(self): t = table("t", column("c")) stmt = select([t]) a1 = stmt.alias() a2 = a1.alias() - is_(a2.element, a2.wrapped.element) - a3 = a2._clone() a3._copy_internals() is_(a1.corresponding_column(a3.c.c), a1.c.c) - is_(a3.element, a3.wrapped.element) diff --git a/test/sql/test_text.py b/test/sql/test_text.py index 4be53b2ebb..35d909ef8a 100644 --- a/test/sql/test_text.py +++ b/test/sql/test_text.py @@ -147,10 +147,13 @@ class SelectCompositionTest(fixtures.TestBase, AssertsCompiledSQL): table1.c.myid, ], from_obj=[table1], - ).select(), - "SELECT column1 AS foobar, column2 AS hoho, myid FROM " + ) + .subquery() + .select(), + "SELECT anon_1.column1 AS foobar, anon_1.column2 AS hoho, " + "anon_1.myid FROM " "(SELECT column1 AS foobar, column2 AS hoho, " - "mytable.myid AS myid FROM mytable)", + "mytable.myid AS myid FROM mytable) AS anon_1", ) def test_select_composition_seven(self): @@ -409,11 +412,15 @@ class AsFromTest(fixtures.TestBase, AssertsCompiledSQL): eq_( compiled._create_result_map(), { - "id": ("id", (t.c.id._proxies[0], "id", "id"), t.c.id.type), + "id": ( + "id", + (t.selected_columns.id, "id", "id"), + t.selected_columns.id.type, + ), "name": ( "name", - (t.c.name._proxies[0], "name", "name"), - t.c.name.type, + (t.selected_columns.name, "name", "name"), + t.selected_columns.name.type, ), }, ) @@ -425,17 +432,25 @@ class AsFromTest(fixtures.TestBase, AssertsCompiledSQL): eq_( compiled._create_result_map(), { - "id": ("id", (t.c.id._proxies[0], "id", "id"), t.c.id.type), + "id": ( + "id", + (t.selected_columns.id, "id", "id"), + t.selected_columns.id.type, + ), "name": ( "name", - (t.c.name._proxies[0], "name", "name"), - t.c.name.type, + (t.selected_columns.name, "name", "name"), + t.selected_columns.name.type, ), }, ) def test_basic_subquery_resultmap(self): - t = text("select id, name from user").columns(id=Integer, name=String) + t = ( + text("select id, name from user") + .columns(id=Integer, name=String) + .subquery() + ) stmt = select([table1.c.myid]).select_from( table1.join(t, table1.c.myid == t.c.id) @@ -456,16 +471,16 @@ class AsFromTest(fixtures.TestBase, AssertsCompiledSQL): t = text("select a, b, c from foo").columns( column("a"), column("b"), column("c") ) - eq_(t.c.keys(), ["a", "b", "c"]) + eq_(t.selected_columns.keys(), ["a", "b", "c"]) def test_column_collection_pos_plus_bykey(self): # overlapping positional names + type names t = text("select a, b, c from foo").columns( column("a"), column("b"), b=Integer, c=String ) - eq_(t.c.keys(), ["a", "b", "c"]) - eq_(t.c.b.type._type_affinity, Integer) - eq_(t.c.c.type._type_affinity, String) + eq_(t.selected_columns.keys(), ["a", "b", "c"]) + eq_(t.selected_columns.b.type._type_affinity, Integer) + eq_(t.selected_columns.c.type._type_affinity, String) def _xy_table_fixture(self): m = MetaData() @@ -549,6 +564,26 @@ class AsFromTest(fixtures.TestBase, AssertsCompiledSQL): "FROM mytable, t WHERE mytable.myid = t.id", ) + def test_subquery(self): + t = ( + text("select id, name from user") + .columns(id=Integer, name=String) + .subquery() + ) + + stmt = ( + select([table1.c.myid]) + .select_from(table1.join(t, table1.c.myid == t.c.id)) + .order_by(t.c.name) + ) + + self.assert_compile( + stmt, + "SELECT mytable.myid FROM mytable JOIN " + "(select id, name from user) AS anon_1 " + "ON mytable.myid = anon_1.id ORDER BY anon_1.name", + ) + def test_alias(self): t = ( text("select id, name from user") @@ -769,15 +804,16 @@ class OrderByLabelResolutionTest(fixtures.TestBase, AssertsCompiledSQL): ) def test_over(self): - stmt = select([column("foo"), column("bar")]) + stmt = select([column("foo"), column("bar")]).subquery() stmt = select( [func.row_number().over(order_by="foo", partition_by="bar")] ).select_from(stmt) self.assert_compile( stmt, - "SELECT row_number() OVER (PARTITION BY bar ORDER BY foo) " - "AS anon_1 FROM (SELECT foo, bar)", + "SELECT row_number() OVER " + "(PARTITION BY anon_2.bar ORDER BY anon_2.foo) " + "AS anon_1 FROM (SELECT foo, bar) AS anon_2", ) def test_union_column(self): diff --git a/test/sql/test_type_expressions.py b/test/sql/test_type_expressions.py index 18877c338f..f913ab6389 100644 --- a/test/sql/test_type_expressions.py +++ b/test/sql/test_type_expressions.py @@ -279,18 +279,10 @@ class DerivedTest(_ExprFixture, fixtures.TestBase, AssertsCompiledSQL): def test_select_from_select(self): table = self._fixture() self.assert_compile( - table.select().select(), - "SELECT x, lower(y) AS y FROM (SELECT test_table.x " - "AS x, test_table.y AS y FROM test_table)", - ) - - def test_select_from_alias(self): - table = self._fixture() - self.assert_compile( - table.select().alias().select(), - "SELECT anon_1.x, lower(anon_1.y) AS y FROM (SELECT " - "test_table.x AS x, test_table.y AS y " - "FROM test_table) AS anon_1", + table.select().subquery().select(), + "SELECT anon_1.x, lower(anon_1.y) AS y FROM " + "(SELECT test_table.x " + "AS x, test_table.y AS y FROM test_table) AS anon_1", ) def test_select_from_aliased_join(self):