From: Mike Bayer Date: Sun, 14 Dec 2014 22:10:44 +0000 (-0500) Subject: - rework the migration doc sections X-Git-Tag: rel_1_0_0b1~166 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=d5f88ee9e51ceeaf4705d3b456b33b779cf25a5c;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git - rework the migration doc sections - small fixes in bulk docs --- diff --git a/doc/build/changelog/migration_10.rst b/doc/build/changelog/migration_10.rst index e1fb13662b..db0d270a1d 100644 --- a/doc/build/changelog/migration_10.rst +++ b/doc/build/changelog/migration_10.rst @@ -8,7 +8,7 @@ What's New in SQLAlchemy 1.0? undergoing maintenance releases as of May, 2014, and SQLAlchemy version 1.0, as of yet unreleased. - Document last updated: December 8, 2014 + Document last updated: December 14, 2014 Introduction ============ @@ -17,13 +17,44 @@ This guide introduces what's new in SQLAlchemy version 1.0, and also documents changes which affect users migrating their applications from the 0.9 series of SQLAlchemy to 1.0. -Please carefully review -:ref:`behavioral_changes_orm_10` and :ref:`behavioral_changes_core_10` for -potentially backwards-incompatible changes. +Please carefully review the sections on behavioral changes for +potentially backwards-incompatible changes in behavior. -New Features -============ +New Features and Improvements - ORM +=================================== + +New Session Bulk INSERT/UPDATE API +---------------------------------- + +A new series of :class:`.Session` methods which provide hooks directly +into the unit of work's facility for emitting INSERT and UPDATE +statements has been created. When used correctly, this expert-oriented system +can allow ORM-mappings to be used to generate bulk insert and update +statements batched into executemany groups, allowing the statements +to proceed at speeds that rival direct use of the Core. + +.. seealso:: + + :ref:`bulk_operations` - introduction and full documentation + +:ticket:`3100` + +New Performance Example Suite +------------------------------ + +Inspired by the benchmarking done for the :ref:`bulk_operations` feature +as well as for the :ref:`faq_how_to_profile` section of the FAQ, a new +example section has been added which features several scripts designed +to illustrate the relative performance profile of various Core and ORM +techniques. The scripts are organized into use cases, and are packaged +under a single console interface such that any combination of demonstrations +can be run, dumping out timings, Python profile results and/or RunSnake profile +displays. + +.. seealso:: + + :ref:`examples_performance` .. _feature_3150: @@ -160,502 +191,169 @@ the polymorphic union of the base. :ticket:`3150` :ticket:`2670` :ticket:`3149` :ticket:`2952` :ticket:`3050` -.. _feature_3034: - -Select/Query LIMIT / OFFSET may be specified as an arbitrary SQL expression ----------------------------------------------------------------------------- - -The :meth:`.Select.limit` and :meth:`.Select.offset` methods now accept -any SQL expression, in addition to integer values, as arguments. The ORM -:class:`.Query` object also passes through any expression to the underlying -:class:`.Select` object. Typically -this is used to allow a bound parameter to be passed, which can be substituted -with a value later:: +ORM full object fetches 25% faster +---------------------------------- - sel = select([table]).limit(bindparam('mylimit')).offset(bindparam('myoffset')) +The mechanics of the ``loading.py`` module as well as the identity map +have undergone several passes of inlining, refactoring, and pruning, so +that a raw load of rows now populates ORM-based objects around 25% faster. +Assuming a 1M row table, a script like the following illustrates the type +of load that's improved the most:: -Dialects which don't support non-integer LIMIT or OFFSET expressions may continue -to not support this behavior; third party dialects may also need modification -in order to take advantage of the new behavior. A dialect which currently -uses the ``._limit`` or ``._offset`` attributes will continue to function -for those cases where the limit/offset was specified as a simple integer value. -However, when a SQL expression is specified, these two attributes will -instead raise a :class:`.CompileError` on access. A third-party dialect which -wishes to support the new feature should now call upon the ``._limit_clause`` -and ``._offset_clause`` attributes to receive the full SQL expression, rather -than the integer value. + import time + from sqlalchemy import Integer, Column, create_engine, Table + from sqlalchemy.orm import Session + from sqlalchemy.ext.declarative import declarative_base -.. _change_2051: + Base = declarative_base() -.. _feature_insert_from_select_defaults: + class Foo(Base): + __table__ = Table( + 'foo', Base.metadata, + Column('id', Integer, primary_key=True), + Column('a', Integer(), nullable=False), + Column('b', Integer(), nullable=False), + Column('c', Integer(), nullable=False), + ) -INSERT FROM SELECT now includes Python and SQL-expression defaults -------------------------------------------------------------------- + engine = create_engine( + 'mysql+mysqldb://scott:tiger@localhost/test', echo=True) -:meth:`.Insert.from_select` now includes Python and SQL-expression defaults if -otherwise unspecified; the limitation where non-server column defaults -aren't included in an INSERT FROM SELECT is now lifted and these -expressions are rendered as constants into the SELECT statement:: + sess = Session(engine) - from sqlalchemy import Table, Column, MetaData, Integer, select, func + now = time.time() - m = MetaData() + # avoid using all() so that we don't have the overhead of building + # a large list of full objects in memory + for obj in sess.query(Foo).yield_per(100).limit(1000000): + pass - t = Table( - 't', m, - Column('x', Integer), - Column('y', Integer, default=func.somefunction())) + print("Total time: %d" % (time.time() - now)) - stmt = select([t.c.x]) - print t.insert().from_select(['x'], stmt) +Local MacBookPro results bench from 19 seconds for 0.9 down to 14 seconds for +1.0. The :meth:`.Query.yield_per` call is always a good idea when batching +huge numbers of rows, as it prevents the Python interpreter from having +to allocate a huge amount of memory for all objects and their instrumentation +at once. Without the :meth:`.Query.yield_per`, the above script on the +MacBookPro is 31 seconds on 0.9 and 26 seconds on 1.0, the extra time spent +setting up very large memory buffers. -Will render:: +.. _feature_3176: - INSERT INTO t (x, y) SELECT t.x, somefunction() AS somefunction_1 - FROM t +New KeyedTuple implementation dramatically faster +------------------------------------------------- -The feature can be disabled using -:paramref:`.Insert.from_select.include_defaults`. +We took a look into the :class:`.KeyedTuple` implementation in the hopes +of improving queries like this:: -New Postgresql Table options ------------------------------ + rows = sess.query(Foo.a, Foo.b, Foo.c).all() -Added support for PG table options TABLESPACE, ON COMMIT, -WITH(OUT) OIDS, and INHERITS, when rendering DDL via -the :class:`.Table` construct. +The :class:`.KeyedTuple` class is used rather than Python's +``collections.namedtuple()``, because the latter has a very complex +type-creation routine that benchmarks much slower than :class:`.KeyedTuple`. +However, when fetching hundreds of thousands of rows, +``collections.namedtuple()`` quickly overtakes :class:`.KeyedTuple` which +becomes dramatically slower as instance invocation goes up. What to do? +A new type that hedges between the approaches of both. Benching +all three types for "size" (number of rows returned) and "num" +(number of distinct queries), the new "lightweight keyed tuple" either +outperforms both, or lags very slightly behind the faster object, based on +which scenario. In the "sweet spot", where we are both creating a good number +of new types as well as fetching a good number of rows, the lightweight +object totally smokes both namedtuple and KeyedTuple:: -.. seealso:: + ----------------- + size=10 num=10000 # few rows, lots of queries + namedtuple: 3.60302400589 # namedtuple falls over + keyedtuple: 0.255059957504 # KeyedTuple very fast + lw keyed tuple: 0.582715034485 # lw keyed trails right on KeyedTuple + ----------------- + size=100 num=1000 # <--- sweet spot + namedtuple: 0.365247011185 + keyedtuple: 0.24896979332 + lw keyed tuple: 0.0889317989349 # lw keyed blows both away! + ----------------- + size=10000 num=100 + namedtuple: 0.572599887848 + keyedtuple: 2.54251694679 + lw keyed tuple: 0.613876104355 + ----------------- + size=1000000 num=10 # few queries, lots of rows + namedtuple: 5.79669594765 # namedtuple very fast + keyedtuple: 28.856498003 # KeyedTuple falls over + lw keyed tuple: 6.74346804619 # lw keyed trails right on namedtuple - :ref:`postgresql_table_options` -:ticket:`2051` +:ticket:`3176` -New Session Bulk INSERT/UPDATE API ----------------------------------- +.. _feature_updatemany: -A new series of :class:`.Session` methods which provide hooks directly -into the unit of work's facility for emitting INSERT and UPDATE -statements has been created. When used correctly, this expert-oriented system -can allow ORM-mappings to be used to generate bulk insert and update -statements batched into executemany groups, allowing the statements -to proceed at speeds that rival direct use of the Core. +UPDATE statements are now batched with executemany() in a flush +---------------------------------------------------------------- -.. seealso:: +UPDATE statements can now be batched within an ORM flush +into more performant executemany() call, similarly to how INSERT +statements can be batched; this will be invoked within flush +based on the following criteria: - :ref:`bulk_operations` - introduction and full documentation +* two or more UPDATE statements in sequence involve the identical set of + columns to be modified. -:ticket:`3100` +* The statement has no embedded SQL expressions in the SET clause. -New Performance Example Suite ------------------------------- +* The mapping does not use a :paramref:`~.orm.mapper.version_id_col`, or + the backend dialect supports a "sane" rowcount for an executemany() + operation; most DBAPIs support this correctly now. -Inspired by the benchmarking done for the :ref:`bulk_operations` feature -as well as for the :ref:`faq_how_to_profile` section of the FAQ, a new -example section has been added which features several scripts designed -to illustrate the relative performance profile of various Core and ORM -techniques. The scripts are organized into use cases, and are packaged -under a single console interface such that any combination of demonstrations -can be run, dumping out timings, Python profile results and/or RunSnake profile -displays. +.. _feature_3178: -.. seealso:: - :ref:`examples_performance` +.. _bug_3035: +Session.get_bind() handles a wider variety of inheritance scenarios +------------------------------------------------------------------- -.. _feature_get_enums: +The :meth:`.Session.get_bind` method is invoked whenever a query or unit +of work flush process seeks to locate the database engine that corresponds +to a particular class. The method has been improved to handle a variety +of inheritance-oriented scenarios, including: -New get_enums() method with Postgresql Dialect ----------------------------------------------- +* Binding to a Mixin or Abstract Class:: -The :func:`.inspect` method returns a :class:`.PGInspector` object in the -case of Postgresql, which includes a new :meth:`.PGInspector.get_enums` -method that returns information on all available ``ENUM`` types:: + class MyClass(SomeMixin, Base): + __tablename__ = 'my_table' + # ... - from sqlalchemy import inspect, create_engine + session = Session(binds={SomeMixin: some_engine}) - engine = create_engine("postgresql+psycopg2://host/dbname") - insp = inspect(engine) - print(insp.get_enums()) -.. seealso:: +* Binding to inherited concrete subclasses individually based on table:: - :meth:`.PGInspector.get_enums` + class BaseClass(Base): + __tablename__ = 'base' -.. _feature_2891: + # ... -Postgresql Dialect reflects Materialized Views, Foreign Tables --------------------------------------------------------------- + class ConcreteSubClass(BaseClass): + __tablename__ = 'concrete' -Changes are as follows: + # ... -* the :class:`Table` construct with ``autoload=True`` will now match a name - that exists in the database as a materialized view or foriegn table. + __mapper_args__ = {'concrete': True} -* :meth:`.Inspector.get_view_names` will return plain and materialized view - names. -* :meth:`.Inspector.get_table_names` does **not** change for Postgresql, it - continues to return only the names of plain tables. + session = Session(binds={ + base_table: some_engine, + concrete_table: some_other_engine + }) -* A new method :meth:`.PGInspector.get_foreign_table_names` is added which - will return the names of tables that are specifically marked as "foreign" - in the Postgresql schema tables. -The change to reflection involves adding ``'m'`` and ``'f'`` to the list -of qualifiers we use when querying ``pg_class.relkind``, but this change -is new in 1.0.0 to avoid any backwards-incompatible surprises for those -running 0.9 in production. +:ticket:`3035` -:ticket:`2891` +.. _feature_2963: -.. _change_3264: - -Postgresql ``has_table()`` now works for temporary tables ---------------------------------------------------------- - -This is a simple fix such that "has table" for temporary tables now works, -so that code like the following may proceed:: - - from sqlalchemy import * - - metadata = MetaData() - user_tmp = Table( - "user_tmp", metadata, - Column("id", INT, primary_key=True), - Column('name', VARCHAR(50)), - prefixes=['TEMPORARY'] - ) - - e = create_engine("postgresql://scott:tiger@localhost/test", echo='debug') - with e.begin() as conn: - user_tmp.create(conn, checkfirst=True) - - # checkfirst will succeed - user_tmp.create(conn, checkfirst=True) - -The very unlikely case that this behavior will cause a non-failing application -to behave differently, is because Postgresql allows a non-temporary table -to silently overwrite a temporary table. So code like the following will -now act completely differently, no longer creating the real table following -the temporary table:: - - from sqlalchemy import * - - metadata = MetaData() - user_tmp = Table( - "user_tmp", metadata, - Column("id", INT, primary_key=True), - Column('name', VARCHAR(50)), - prefixes=['TEMPORARY'] - ) - - e = create_engine("postgresql://scott:tiger@localhost/test", echo='debug') - with e.begin() as conn: - user_tmp.create(conn, checkfirst=True) - - m2 = MetaData() - user = Table( - "user_tmp", m2, - Column("id", INT, primary_key=True), - Column('name', VARCHAR(50)), - ) - - # in 0.9, *will create* the new table, overwriting the old one. - # in 1.0, *will not create* the new table - user.create(conn, checkfirst=True) - -:ticket:`3264` - -.. _feature_gh134: - -Postgresql FILTER keyword -------------------------- - -The SQL standard FILTER keyword for aggregate functions is now supported -by Postgresql as of 9.4. SQLAlchemy allows this using -:meth:`.FunctionElement.filter`:: - - func.count(1).filter(True) - -.. seealso:: - - :meth:`.FunctionElement.filter` - - :class:`.FunctionFilter` - -.. _feature_3184: - -UniqueConstraint is now part of the Table reflection process ------------------------------------------------------------- - -A :class:`.Table` object populated using ``autoload=True`` will now -include :class:`.UniqueConstraint` constructs as well as -:class:`.Index` constructs. This logic has a few caveats for -Postgresql and Mysql: - -Postgresql -^^^^^^^^^^ - -Postgresql has the behavior such that when a UNIQUE constraint is -created, it implicitly creates a UNIQUE INDEX corresponding to that -constraint as well. The :meth:`.Inspector.get_indexes` and the -:meth:`.Inspector.get_unique_constraints` methods will continue to -**both** return these entries distinctly, where -:meth:`.Inspector.get_indexes` now features a token -``duplicates_constraint`` within the index entry indicating the -corresponding constraint when detected. However, when performing -full table reflection using ``Table(..., autoload=True)``, the -:class:`.Index` construct is detected as being linked to the -:class:`.UniqueConstraint`, and is **not** present within the -:attr:`.Table.indexes` collection; only the :class:`.UniqueConstraint` -will be present in the :attr:`.Table.constraints` collection. This -deduplication logic works by joining to the ``pg_constraint`` table -when querying ``pg_index`` to see if the two constructs are linked. - -MySQL -^^^^^ - -MySQL does not have separate concepts for a UNIQUE INDEX and a UNIQUE -constraint. While it supports both syntaxes when creating tables and indexes, -it does not store them any differently. The -:meth:`.Inspector.get_indexes` -and the :meth:`.Inspector.get_unique_constraints` methods will continue to -**both** return an entry for a UNIQUE index in MySQL, -where :meth:`.Inspector.get_unique_constraints` features a new token -``duplicates_index`` within the constraint entry indicating that this is a -dupe entry corresponding to that index. However, when performing -full table reflection using ``Table(..., autoload=True)``, -the :class:`.UniqueConstraint` construct is -**not** part of the fully reflected :class:`.Table` construct under any -circumstances; this construct is always represented by a :class:`.Index` -with the ``unique=True`` setting present in the :attr:`.Table.indexes` -collection. - -.. seealso:: - - :ref:`postgresql_index_reflection` - - :ref:`mysql_unique_constraints` - -:ticket:`3184` - - -Behavioral Improvements -======================= - -.. _feature_updatemany: - -UPDATE statements are now batched with executemany() in a flush ----------------------------------------------------------------- - -UPDATE statements can now be batched within an ORM flush -into more performant executemany() call, similarly to how INSERT -statements can be batched; this will be invoked within flush -based on the following criteria: - -* two or more UPDATE statements in sequence involve the identical set of - columns to be modified. - -* The statement has no embedded SQL expressions in the SET clause. - -* The mapping does not use a :paramref:`~.orm.mapper.version_id_col`, or - the backend dialect supports a "sane" rowcount for an executemany() - operation; most DBAPIs support this correctly now. - -ORM full object fetches 25% faster ----------------------------------- - -The mechanics of the ``loading.py`` module as well as the identity map -have undergone several passes of inlining, refactoring, and pruning, so -that a raw load of rows now populates ORM-based objects around 25% faster. -Assuming a 1M row table, a script like the following illustrates the type -of load that's improved the most:: - - import time - from sqlalchemy import Integer, Column, create_engine, Table - from sqlalchemy.orm import Session - from sqlalchemy.ext.declarative import declarative_base - - Base = declarative_base() - - class Foo(Base): - __table__ = Table( - 'foo', Base.metadata, - Column('id', Integer, primary_key=True), - Column('a', Integer(), nullable=False), - Column('b', Integer(), nullable=False), - Column('c', Integer(), nullable=False), - ) - - engine = create_engine( - 'mysql+mysqldb://scott:tiger@localhost/test', echo=True) - - sess = Session(engine) - - now = time.time() - - # avoid using all() so that we don't have the overhead of building - # a large list of full objects in memory - for obj in sess.query(Foo).yield_per(100).limit(1000000): - pass - - print("Total time: %d" % (time.time() - now)) - -Local MacBookPro results bench from 19 seconds for 0.9 down to 14 seconds for -1.0. The :meth:`.Query.yield_per` call is always a good idea when batching -huge numbers of rows, as it prevents the Python interpreter from having -to allocate a huge amount of memory for all objects and their instrumentation -at once. Without the :meth:`.Query.yield_per`, the above script on the -MacBookPro is 31 seconds on 0.9 and 26 seconds on 1.0, the extra time spent -setting up very large memory buffers. - - -.. _feature_3176: - -New KeyedTuple implementation dramatically faster -------------------------------------------------- - -We took a look into the :class:`.KeyedTuple` implementation in the hopes -of improving queries like this:: - - rows = sess.query(Foo.a, Foo.b, Foo.c).all() - -The :class:`.KeyedTuple` class is used rather than Python's -``collections.namedtuple()``, because the latter has a very complex -type-creation routine that benchmarks much slower than :class:`.KeyedTuple`. -However, when fetching hundreds of thousands of rows, -``collections.namedtuple()`` quickly overtakes :class:`.KeyedTuple` which -becomes dramatically slower as instance invocation goes up. What to do? -A new type that hedges between the approaches of both. Benching -all three types for "size" (number of rows returned) and "num" -(number of distinct queries), the new "lightweight keyed tuple" either -outperforms both, or lags very slightly behind the faster object, based on -which scenario. In the "sweet spot", where we are both creating a good number -of new types as well as fetching a good number of rows, the lightweight -object totally smokes both namedtuple and KeyedTuple:: - - ----------------- - size=10 num=10000 # few rows, lots of queries - namedtuple: 3.60302400589 # namedtuple falls over - keyedtuple: 0.255059957504 # KeyedTuple very fast - lw keyed tuple: 0.582715034485 # lw keyed trails right on KeyedTuple - ----------------- - size=100 num=1000 # <--- sweet spot - namedtuple: 0.365247011185 - keyedtuple: 0.24896979332 - lw keyed tuple: 0.0889317989349 # lw keyed blows both away! - ----------------- - size=10000 num=100 - namedtuple: 0.572599887848 - keyedtuple: 2.54251694679 - lw keyed tuple: 0.613876104355 - ----------------- - size=1000000 num=10 # few queries, lots of rows - namedtuple: 5.79669594765 # namedtuple very fast - keyedtuple: 28.856498003 # KeyedTuple falls over - lw keyed tuple: 6.74346804619 # lw keyed trails right on namedtuple - - -:ticket:`3176` - -.. _bug_3035: - -Session.get_bind() handles a wider variety of inheritance scenarios -------------------------------------------------------------------- - -The :meth:`.Session.get_bind` method is invoked whenever a query or unit -of work flush process seeks to locate the database engine that corresponds -to a particular class. The method has been improved to handle a variety -of inheritance-oriented scenarios, including: - -* Binding to a Mixin or Abstract Class:: - - class MyClass(SomeMixin, Base): - __tablename__ = 'my_table' - # ... - - session = Session(binds={SomeMixin: some_engine}) - - -* Binding to inherited concrete subclasses individually based on table:: - - class BaseClass(Base): - __tablename__ = 'base' - - # ... - - class ConcreteSubClass(BaseClass): - __tablename__ = 'concrete' - - # ... - - __mapper_args__ = {'concrete': True} - - - session = Session(binds={ - base_table: some_engine, - concrete_table: some_other_engine - }) - - -:ticket:`3035` - - -.. _feature_3178: - -New systems to safely emit parameterized warnings -------------------------------------------------- - -For a long time, there has been a restriction that warning messages could not -refer to data elements, such that a particular function might emit an -infinite number of unique warnings. The key place this occurs is in the -``Unicode type received non-unicode bind param value`` warning. Placing -the data value in this message would mean that the Python ``__warningregistry__`` -for that module, or in some cases the Python-global ``warnings.onceregistry``, -would grow unbounded, as in most warning scenarios, one of these two collections -is populated with every distinct warning message. - -The change here is that by using a special ``string`` type that purposely -changes how the string is hashed, we can control that a large number of -parameterized messages are hashed only on a small set of possible hash -values, such that a warning such as ``Unicode type received non-unicode -bind param value`` can be tailored to be emitted only a specific number -of times; beyond that, the Python warnings registry will begin recording -them as duplicates. - -To illustrate, the following test script will show only ten warnings being -emitted for ten of the parameter sets, out of a total of 1000:: - - from sqlalchemy import create_engine, Unicode, select, cast - import random - import warnings - - e = create_engine("sqlite://") - - # Use the "once" filter (which is also the default for Python - # warnings). Exactly ten of these warnings will - # be emitted; beyond that, the Python warnings registry will accumulate - # new values as dupes of one of the ten existing. - warnings.filterwarnings("once") - - for i in range(1000): - e.execute(select([cast( - ('foo_%d' % random.randint(0, 1000000)).encode('ascii'), Unicode)])) - -The format of the warning here is:: - - /path/lib/sqlalchemy/sql/sqltypes.py:186: SAWarning: Unicode type received - non-unicode bind param value 'foo_4852'. (this warning may be - suppressed after 10 occurrences) - - -:ticket:`3178` - -.. _feature_2963: - -.info dictionary improvements ------------------------------ +.info dictionary improvements +----------------------------- The :attr:`.InspectionAttr.info` collection is now available on every kind of object that one would retrieve from the :attr:`.Mapper.all_orm_descriptors` @@ -683,128 +381,6 @@ as remaining ORM constructs such as :func:`.orm.synonym`. :ticket:`2963` -.. _migration_3177: - -Change to single-table-inheritance criteria when using from_self(), count() ---------------------------------------------------------------------------- - -Given a single-table inheritance mapping, such as:: - - class Widget(Base): - __table__ = 'widget_table' - - class FooWidget(Widget): - pass - -Using :meth:`.Query.from_self` or :meth:`.Query.count` against a subclass -would produce a subquery, but then add the "WHERE" criteria for subtypes -to the outside:: - - sess.query(FooWidget).from_self().all() - -rendering:: - - SELECT - anon_1.widgets_id AS anon_1_widgets_id, - anon_1.widgets_type AS anon_1_widgets_type - FROM (SELECT widgets.id AS widgets_id, widgets.type AS widgets_type, - FROM widgets) AS anon_1 - WHERE anon_1.widgets_type IN (?) - -The issue with this is that if the inner query does not specify all -columns, then we can't add the WHERE clause on the outside (it actually tries, -and produces a bad query). This decision -apparently goes way back to 0.6.5 with the note "may need to make more -adjustments to this". Well, those adjustments have arrived! So now the -above query will render:: - - SELECT - anon_1.widgets_id AS anon_1_widgets_id, - anon_1.widgets_type AS anon_1_widgets_type - FROM (SELECT widgets.id AS widgets_id, widgets.type AS widgets_type, - FROM widgets - WHERE widgets.type IN (?)) AS anon_1 - -So that queries that don't include "type" will still work!:: - - sess.query(FooWidget.id).count() - -Renders:: - - SELECT count(*) AS count_1 - FROM (SELECT widgets.id AS widgets_id - FROM widgets - WHERE widgets.type IN (?)) AS anon_1 - - -:ticket:`3177` - - -.. _migration_3222: - - -single-table-inheritance criteria added to all ON clauses unconditionally -------------------------------------------------------------------------- - -When joining to a single-table inheritance subclass target, the ORM always adds -the "single table criteria" when joining on a relationship. Given a -mapping as:: - - class Widget(Base): - __tablename__ = 'widget' - id = Column(Integer, primary_key=True) - type = Column(String) - related_id = Column(ForeignKey('related.id')) - related = relationship("Related", backref="widget") - __mapper_args__ = {'polymorphic_on': type} - - - class FooWidget(Widget): - __mapper_args__ = {'polymorphic_identity': 'foo'} - - - class Related(Base): - __tablename__ = 'related' - id = Column(Integer, primary_key=True) - -It's been the behavior for quite some time that a JOIN on the relationship -will render a "single inheritance" clause for the type:: - - s.query(Related).join(FooWidget, Related.widget).all() - -SQL output:: - - SELECT related.id AS related_id - FROM related JOIN widget ON related.id = widget.related_id AND widget.type IN (:type_1) - -Above, because we joined to a subclass ``FooWidget``, :meth:`.Query.join` -knew to add the ``AND widget.type IN ('foo')`` criteria to the ON clause. - -The change here is that the ``AND widget.type IN()`` criteria is now appended -to *any* ON clause, not just those generated from a relationship, -including one that is explicitly stated:: - - # ON clause will now render as - # related.id = widget.related_id AND widget.type IN (:type_1) - s.query(Related).join(FooWidget, FooWidget.related_id == Related.id).all() - -As well as the "implicit" join when no ON clause of any kind is stated:: - - # ON clause will now render as - # related.id = widget.related_id AND widget.type IN (:type_1) - s.query(Related).join(FooWidget).all() - -Previously, the ON clause for these would not include the single-inheritance -criteria. Applications that are already adding this criteria to work around -this will want to remove its explicit use, though it should continue to work -fine if the criteria happens to be rendered twice in the meantime. - -.. seealso:: - - :ref:`bug_3233` - -:ticket:`3222` - .. _bug_3188: ColumnProperty constructs work a lot better with aliases, order_by @@ -853,85 +429,204 @@ New output:: FROM b WHERE b.a_id = a_1.id) AS anon_2 FROM a, a AS a_1 ORDER BY anon_2 -There were also many scenarios where the "order by" logic would fail -to order by label, for example if the mapping were "polymorphic":: +There were also many scenarios where the "order by" logic would fail +to order by label, for example if the mapping were "polymorphic":: + + class A(Base): + __tablename__ = 'a' + + id = Column(Integer, primary_key=True) + type = Column(String) + + __mapper_args__ = {'polymorphic_on': type, 'with_polymorphic': '*'} + +The order_by would fail to use the label, as it would be anonymized due +to the polymorphic loading:: + + SELECT a.id AS a_id, a.type AS a_type, (SELECT max(b.id) AS max_1 + FROM b WHERE b.a_id = a.id) AS anon_1 + FROM a ORDER BY (SELECT max(b.id) AS max_2 + FROM b WHERE b.a_id = a.id) + +Now that the order by label tracks the anonymized label, this now works:: + + SELECT a.id AS a_id, a.type AS a_type, (SELECT max(b.id) AS max_1 + FROM b WHERE b.a_id = a.id) AS anon_1 + FROM a ORDER BY anon_1 + +Included in these fixes are a variety of heisenbugs that could corrupt +the state of an ``aliased()`` construct such that the labeling logic +would again fail; these have also been fixed. + +:ticket:`3148` :ticket:`3188` + +New Features and Improvements - Core +==================================== + +.. _feature_3034: + +Select/Query LIMIT / OFFSET may be specified as an arbitrary SQL expression +---------------------------------------------------------------------------- + +The :meth:`.Select.limit` and :meth:`.Select.offset` methods now accept +any SQL expression, in addition to integer values, as arguments. The ORM +:class:`.Query` object also passes through any expression to the underlying +:class:`.Select` object. Typically +this is used to allow a bound parameter to be passed, which can be substituted +with a value later:: + + sel = select([table]).limit(bindparam('mylimit')).offset(bindparam('myoffset')) + +Dialects which don't support non-integer LIMIT or OFFSET expressions may continue +to not support this behavior; third party dialects may also need modification +in order to take advantage of the new behavior. A dialect which currently +uses the ``._limit`` or ``._offset`` attributes will continue to function +for those cases where the limit/offset was specified as a simple integer value. +However, when a SQL expression is specified, these two attributes will +instead raise a :class:`.CompileError` on access. A third-party dialect which +wishes to support the new feature should now call upon the ``._limit_clause`` +and ``._offset_clause`` attributes to receive the full SQL expression, rather +than the integer value. + +.. _change_2051: + +.. _feature_insert_from_select_defaults: + +INSERT FROM SELECT now includes Python and SQL-expression defaults +------------------------------------------------------------------- + +:meth:`.Insert.from_select` now includes Python and SQL-expression defaults if +otherwise unspecified; the limitation where non-server column defaults +aren't included in an INSERT FROM SELECT is now lifted and these +expressions are rendered as constants into the SELECT statement:: + + from sqlalchemy import Table, Column, MetaData, Integer, select, func + + m = MetaData() + + t = Table( + 't', m, + Column('x', Integer), + Column('y', Integer, default=func.somefunction())) + + stmt = select([t.c.x]) + print t.insert().from_select(['x'], stmt) + +Will render:: + + INSERT INTO t (x, y) SELECT t.x, somefunction() AS somefunction_1 + FROM t - class A(Base): - __tablename__ = 'a' +The feature can be disabled using +:paramref:`.Insert.from_select.include_defaults`. - id = Column(Integer, primary_key=True) - type = Column(String) +.. _feature_3184: - __mapper_args__ = {'polymorphic_on': type, 'with_polymorphic': '*'} +UniqueConstraint is now part of the Table reflection process +------------------------------------------------------------ -The order_by would fail to use the label, as it would be anonymized due -to the polymorphic loading:: +A :class:`.Table` object populated using ``autoload=True`` will now +include :class:`.UniqueConstraint` constructs as well as +:class:`.Index` constructs. This logic has a few caveats for +Postgresql and Mysql: - SELECT a.id AS a_id, a.type AS a_type, (SELECT max(b.id) AS max_1 - FROM b WHERE b.a_id = a.id) AS anon_1 - FROM a ORDER BY (SELECT max(b.id) AS max_2 - FROM b WHERE b.a_id = a.id) +Postgresql +^^^^^^^^^^ -Now that the order by label tracks the anonymized label, this now works:: +Postgresql has the behavior such that when a UNIQUE constraint is +created, it implicitly creates a UNIQUE INDEX corresponding to that +constraint as well. The :meth:`.Inspector.get_indexes` and the +:meth:`.Inspector.get_unique_constraints` methods will continue to +**both** return these entries distinctly, where +:meth:`.Inspector.get_indexes` now features a token +``duplicates_constraint`` within the index entry indicating the +corresponding constraint when detected. However, when performing +full table reflection using ``Table(..., autoload=True)``, the +:class:`.Index` construct is detected as being linked to the +:class:`.UniqueConstraint`, and is **not** present within the +:attr:`.Table.indexes` collection; only the :class:`.UniqueConstraint` +will be present in the :attr:`.Table.constraints` collection. This +deduplication logic works by joining to the ``pg_constraint`` table +when querying ``pg_index`` to see if the two constructs are linked. - SELECT a.id AS a_id, a.type AS a_type, (SELECT max(b.id) AS max_1 - FROM b WHERE b.a_id = a.id) AS anon_1 - FROM a ORDER BY anon_1 +MySQL +^^^^^ -Included in these fixes are a variety of heisenbugs that could corrupt -the state of an ``aliased()`` construct such that the labeling logic -would again fail; these have also been fixed. +MySQL does not have separate concepts for a UNIQUE INDEX and a UNIQUE +constraint. While it supports both syntaxes when creating tables and indexes, +it does not store them any differently. The +:meth:`.Inspector.get_indexes` +and the :meth:`.Inspector.get_unique_constraints` methods will continue to +**both** return an entry for a UNIQUE index in MySQL, +where :meth:`.Inspector.get_unique_constraints` features a new token +``duplicates_index`` within the constraint entry indicating that this is a +dupe entry corresponding to that index. However, when performing +full table reflection using ``Table(..., autoload=True)``, +the :class:`.UniqueConstraint` construct is +**not** part of the fully reflected :class:`.Table` construct under any +circumstances; this construct is always represented by a :class:`.Index` +with the ``unique=True`` setting present in the :attr:`.Table.indexes` +collection. -:ticket:`3148` :ticket:`3188` +.. seealso:: -.. _bug_3170: + :ref:`postgresql_index_reflection` -null(), false() and true() constants are no longer singletons -------------------------------------------------------------- + :ref:`mysql_unique_constraints` -These three constants were changed to return a "singleton" value -in 0.9; unfortunately, that would lead to a query like the following -to not render as expected:: +:ticket:`3184` - select([null(), null()]) -rendering only ``SELECT NULL AS anon_1``, because the two :func:`.null` -constructs would come out as the same ``NULL`` object, and -SQLAlchemy's Core model is based on object identity in order to -determine lexical significance. The change in 0.9 had no -importance other than the desire to save on object overhead; in general, -an unnamed construct needs to stay lexically unique so that it gets -labeled uniquely. +New systems to safely emit parameterized warnings +------------------------------------------------- -:ticket:`3170` +For a long time, there has been a restriction that warning messages could not +refer to data elements, such that a particular function might emit an +infinite number of unique warnings. The key place this occurs is in the +``Unicode type received non-unicode bind param value`` warning. Placing +the data value in this message would mean that the Python ``__warningregistry__`` +for that module, or in some cases the Python-global ``warnings.onceregistry``, +would grow unbounded, as in most warning scenarios, one of these two collections +is populated with every distinct warning message. -.. _change_3266: +The change here is that by using a special ``string`` type that purposely +changes how the string is hashed, we can control that a large number of +parameterized messages are hashed only on a small set of possible hash +values, such that a warning such as ``Unicode type received non-unicode +bind param value`` can be tailored to be emitted only a specific number +of times; beyond that, the Python warnings registry will begin recording +them as duplicates. -DBAPI exception wrapping and handle_error() event improvements --------------------------------------------------------------- +To illustrate, the following test script will show only ten warnings being +emitted for ten of the parameter sets, out of a total of 1000:: -SQLAlchemy's wrapping of DBAPI exceptions was not taking place in the -case where a :class:`.Connection` object was invalidated, and then tried -to reconnect and encountered an error; this has been resolved. + from sqlalchemy import create_engine, Unicode, select, cast + import random + import warnings -Additionally, the recently added :meth:`.ConnectionEvents.handle_error` -event is now invoked for errors that occur upon initial connect, upon -reconnect, and when :func:`.create_engine` is used given a custom connection -function via :paramref:`.create_engine.creator`. + e = create_engine("sqlite://") -The :class:`.ExceptionContext` object has a new datamember -:attr:`.ExceptionContext.engine` that will always refer to the :class:`.Engine` -in use, in those cases when the :class:`.Connection` object is not available -(e.g. on initial connect). + # Use the "once" filter (which is also the default for Python + # warnings). Exactly ten of these warnings will + # be emitted; beyond that, the Python warnings registry will accumulate + # new values as dupes of one of the ten existing. + warnings.filterwarnings("once") + + for i in range(1000): + e.execute(select([cast( + ('foo_%d' % random.randint(0, 1000000)).encode('ascii'), Unicode)])) +The format of the warning here is:: -:ticket:`3266` + /path/lib/sqlalchemy/sql/sqltypes.py:186: SAWarning: Unicode type received + non-unicode bind param value 'foo_4852'. (this warning may be + suppressed after 10 occurrences) -.. _behavioral_changes_orm_10: +:ticket:`3178` -Behavioral Changes - ORM -======================== +Key Behavioral Changes - ORM +============================ .. _bug_3228: @@ -1302,40 +997,160 @@ OUTER joins despite the innerjoin directive:: FROM users LEFT OUTER JOIN orders ON LEFT OUTER JOIN items ON -As noted in the 0.9 notes, the only database backend that has difficulty -with right-nested joins is SQLite; SQLAlchemy as of 0.9 converts a right-nested -join into a subquery as a join target on SQLite. +As noted in the 0.9 notes, the only database backend that has difficulty +with right-nested joins is SQLite; SQLAlchemy as of 0.9 converts a right-nested +join into a subquery as a join target on SQLite. + +.. seealso:: + + :ref:`feature_2976` - description of the feature as introduced in 0.9.4. + +:ticket:`3008` + +query.update() with ``synchronize_session='evaluate'`` raises on multi-table update +----------------------------------------------------------------------------------- + +The "evaulator" for :meth:`.Query.update` won't work with multi-table +updates, and needs to be set to ``synchronize_session=False`` or +``synchronize_session='fetch'`` when multiple tables are present. +The new behavior is that an explicit exception is now raised, with a message +to change the synchronize setting. +This is upgraded from a warning emitted as of 0.9.7. + +:ticket:`3117` + +Resurrect Event has been Removed +-------------------------------- + +The "resurrect" ORM event has been removed entirely. This event ceased to +have any function since version 0.8 removed the older "mutable" system +from the unit of work. + + +.. _migration_3177: + +Change to single-table-inheritance criteria when using from_self(), count() +--------------------------------------------------------------------------- + +Given a single-table inheritance mapping, such as:: + + class Widget(Base): + __table__ = 'widget_table' + + class FooWidget(Widget): + pass + +Using :meth:`.Query.from_self` or :meth:`.Query.count` against a subclass +would produce a subquery, but then add the "WHERE" criteria for subtypes +to the outside:: + + sess.query(FooWidget).from_self().all() + +rendering:: + + SELECT + anon_1.widgets_id AS anon_1_widgets_id, + anon_1.widgets_type AS anon_1_widgets_type + FROM (SELECT widgets.id AS widgets_id, widgets.type AS widgets_type, + FROM widgets) AS anon_1 + WHERE anon_1.widgets_type IN (?) + +The issue with this is that if the inner query does not specify all +columns, then we can't add the WHERE clause on the outside (it actually tries, +and produces a bad query). This decision +apparently goes way back to 0.6.5 with the note "may need to make more +adjustments to this". Well, those adjustments have arrived! So now the +above query will render:: + + SELECT + anon_1.widgets_id AS anon_1_widgets_id, + anon_1.widgets_type AS anon_1_widgets_type + FROM (SELECT widgets.id AS widgets_id, widgets.type AS widgets_type, + FROM widgets + WHERE widgets.type IN (?)) AS anon_1 + +So that queries that don't include "type" will still work!:: + + sess.query(FooWidget.id).count() + +Renders:: + + SELECT count(*) AS count_1 + FROM (SELECT widgets.id AS widgets_id + FROM widgets + WHERE widgets.type IN (?)) AS anon_1 + + +:ticket:`3177` + + +.. _migration_3222: + + +single-table-inheritance criteria added to all ON clauses unconditionally +------------------------------------------------------------------------- + +When joining to a single-table inheritance subclass target, the ORM always adds +the "single table criteria" when joining on a relationship. Given a +mapping as:: + + class Widget(Base): + __tablename__ = 'widget' + id = Column(Integer, primary_key=True) + type = Column(String) + related_id = Column(ForeignKey('related.id')) + related = relationship("Related", backref="widget") + __mapper_args__ = {'polymorphic_on': type} + + + class FooWidget(Widget): + __mapper_args__ = {'polymorphic_identity': 'foo'} + + + class Related(Base): + __tablename__ = 'related' + id = Column(Integer, primary_key=True) + +It's been the behavior for quite some time that a JOIN on the relationship +will render a "single inheritance" clause for the type:: + + s.query(Related).join(FooWidget, Related.widget).all() + +SQL output:: -.. seealso:: + SELECT related.id AS related_id + FROM related JOIN widget ON related.id = widget.related_id AND widget.type IN (:type_1) - :ref:`feature_2976` - description of the feature as introduced in 0.9.4. +Above, because we joined to a subclass ``FooWidget``, :meth:`.Query.join` +knew to add the ``AND widget.type IN ('foo')`` criteria to the ON clause. -:ticket:`3008` +The change here is that the ``AND widget.type IN()`` criteria is now appended +to *any* ON clause, not just those generated from a relationship, +including one that is explicitly stated:: -query.update() with ``synchronize_session='evaluate'`` raises on multi-table update ------------------------------------------------------------------------------------ + # ON clause will now render as + # related.id = widget.related_id AND widget.type IN (:type_1) + s.query(Related).join(FooWidget, FooWidget.related_id == Related.id).all() -The "evaulator" for :meth:`.Query.update` won't work with multi-table -updates, and needs to be set to ``synchronize_session=False`` or -``synchronize_session='fetch'`` when multiple tables are present. -The new behavior is that an explicit exception is now raised, with a message -to change the synchronize setting. -This is upgraded from a warning emitted as of 0.9.7. +As well as the "implicit" join when no ON clause of any kind is stated:: -:ticket:`3117` + # ON clause will now render as + # related.id = widget.related_id AND widget.type IN (:type_1) + s.query(Related).join(FooWidget).all() -Resurrect Event has been Removed --------------------------------- +Previously, the ON clause for these would not include the single-inheritance +criteria. Applications that are already adding this criteria to work around +this will want to remove its explicit use, though it should continue to work +fine if the criteria happens to be rendered twice in the meantime. -The "resurrect" ORM event has been removed entirely. This event ceased to -have any function since version 0.8 removed the older "mutable" system -from the unit of work. +.. seealso:: + :ref:`bug_3233` -.. _behavioral_changes_core_10: +:ticket:`3222` -Behavioral Changes - Core -========================= +Key Behavioral Changes - Core +============================= .. _migration_2992: @@ -1438,128 +1253,329 @@ of this change we have enhanced its functionality. When we have a :func:`.select` or :class:`.Query` that refers to some column name or named label, we might want to GROUP BY and/or ORDER BY known columns or labels:: - stmt = select([ - user.c.name, - func.count(user.c.id).label("id_count") - ]).group_by("name").order_by("id_count") + stmt = select([ + user.c.name, + func.count(user.c.id).label("id_count") + ]).group_by("name").order_by("id_count") + +In the above statement we expect to see "ORDER BY id_count", as opposed to a +re-statement of the function. The string argument given is actively +matched to an entry in the columns clause during compilation, so the above +statement would produce as we expect, without warnings (though note that +the ``"name"`` expression has been resolved to ``users.name``!):: + + SELECT users.name, count(users.id) AS id_count + FROM users GROUP BY users.name ORDER BY id_count + +However, if we refer to a name that cannot be located, then we get +the warning again, as below:: + + stmt = select([ + user.c.name, + func.count(user.c.id).label("id_count") + ]).order_by("some_label") + +The output does what we say, but again it warns us:: + + SAWarning: Can't resolve label reference 'some_label'; converting to + text() (this warning may be suppressed after 10 occurrences) + + SELECT users.name, count(users.id) AS id_count + FROM users ORDER BY some_label + +The above behavior applies to all those places where we might want to refer +to a so-called "label reference"; ORDER BY and GROUP BY, but also within an +OVER clause as well as a DISTINCT ON clause that refers to columns (e.g. the +Postgresql syntax). + +We can still specify any arbitrary expression for ORDER BY or others using +:func:`.text`:: + + stmt = select([users]).order_by(text("some special expression")) + +The upshot of the whole change is that SQLAlchemy now would like us +to tell it when a string is sent that this string is explicitly +a :func:`.text` construct, or a column, table, etc., and if we use it as a +label name in an order by, group by, or other expression, SQLAlchemy expects +that the string resolves to something known, else it should again +be qualified with :func:`.text` or similar. + +:ticket:`2992` + +.. _change_3163: + +Event listeners can not be added or removed from within that event's runner +--------------------------------------------------------------------------- + +Removal of an event listener from inside that same event itself would +modify the elements of a list during iteration, which would cause +still-attached event listeners to silently fail to fire. To prevent +this while still maintaining performance, the lists have been replaced +with ``collections.deque()``, which does not allow any additions or +removals during iteration, and instead raises ``RuntimeError``. + +:ticket:`3163` + +.. _change_3169: + +The INSERT...FROM SELECT construct now implies ``inline=True`` +-------------------------------------------------------------- + +Using :meth:`.Insert.from_select` now implies ``inline=True`` +on :func:`.insert`. This helps to fix a bug where an +INSERT...FROM SELECT construct would inadvertently be compiled +as "implicit returning" on supporting backends, which would +cause breakage in the case of an INSERT that inserts zero rows +(as implicit returning expects a row), as well as arbitrary +return data in the case of an INSERT that inserts multiple +rows (e.g. only the first row of many). +A similar change is also applied to an INSERT..VALUES +with multiple parameter sets; implicit RETURNING will no longer emit +for this statement either. As both of these constructs deal +with varible numbers of rows, the +:attr:`.ResultProxy.inserted_primary_key` accessor does not +apply. Previously, there was a documentation note that one +may prefer ``inline=True`` with INSERT..FROM SELECT as some databases +don't support returning and therefore can't do "implicit" returning, +but there's no reason an INSERT...FROM SELECT needs implicit returning +in any case. Regular explicit :meth:`.Insert.returning` should +be used to return variable numbers of result rows if inserted +data is needed. + +:ticket:`3169` + +.. _change_3027: + +``autoload_with`` now implies ``autoload=True`` +----------------------------------------------- + +A :class:`.Table` can be set up for reflection by passing +:paramref:`.Table.autoload_with` alone:: + + my_table = Table('my_table', metadata, autoload_with=some_engine) + +:ticket:`3027` + +.. _change_3266: + +DBAPI exception wrapping and handle_error() event improvements +-------------------------------------------------------------- + +SQLAlchemy's wrapping of DBAPI exceptions was not taking place in the +case where a :class:`.Connection` object was invalidated, and then tried +to reconnect and encountered an error; this has been resolved. + +Additionally, the recently added :meth:`.ConnectionEvents.handle_error` +event is now invoked for errors that occur upon initial connect, upon +reconnect, and when :func:`.create_engine` is used given a custom connection +function via :paramref:`.create_engine.creator`. + +The :class:`.ExceptionContext` object has a new datamember +:attr:`.ExceptionContext.engine` that will always refer to the :class:`.Engine` +in use, in those cases when the :class:`.Connection` object is not available +(e.g. on initial connect). + + +:ticket:`3266` + +.. _change_3243: + +ForeignKeyConstraint.columns is now a ColumnCollection +------------------------------------------------------ + +:attr:`.ForeignKeyConstraint.columns` was previously a plain list +containing either strings or :class:`.Column` objects, depending on +how the :class:`.ForeignKeyConstraint` was constructed and whether it was +associated with a table. The collection is now a :class:`.ColumnCollection`, +and is only initialized after the :class:`.ForeignKeyConstraint` is +associated with a :class:`.Table`. A new accessor +:attr:`.ForeignKeyConstraint.column_keys` +is added to unconditionally return string keys for the local set of +columns regardless of how the object was constructed or its current +state. + + +.. _bug_3170: + +null(), false() and true() constants are no longer singletons +------------------------------------------------------------- + +These three constants were changed to return a "singleton" value +in 0.9; unfortunately, that would lead to a query like the following +to not render as expected:: + + select([null(), null()]) + +rendering only ``SELECT NULL AS anon_1``, because the two :func:`.null` +constructs would come out as the same ``NULL`` object, and +SQLAlchemy's Core model is based on object identity in order to +determine lexical significance. The change in 0.9 had no +importance other than the desire to save on object overhead; in general, +an unnamed construct needs to stay lexically unique so that it gets +labeled uniquely. + +:ticket:`3170` + +.. _change_3204: + +SQLite/Oracle have distinct methods for temporary table/view name reporting +--------------------------------------------------------------------------- + +The :meth:`.Inspector.get_table_names` and :meth:`.Inspector.get_view_names` +methods in the case of SQLite/Oracle would also return the names of temporary +tables and views, which is not provided by any other dialect (in the case +of MySQL at least it is not even possible). This logic has been moved +out to two new methods :meth:`.Inspector.get_temp_table_names` and +:meth:`.Inspector.get_temp_view_names`. + +Note that reflection of a specific named temporary table or temporary view, +either by ``Table('name', autoload=True)`` or via methods like +:meth:`.Inspector.get_columns` continues to function for most if not all +dialects. For SQLite specifically, there is a bug fix for UNIQUE constraint +reflection from temp tables as well, which is :ticket:`3203`. + +:ticket:`3204` + +Dialect Improvements and Changes - Postgresql +============================================= + +New Postgresql Table options +----------------------------- + +Added support for PG table options TABLESPACE, ON COMMIT, +WITH(OUT) OIDS, and INHERITS, when rendering DDL via +the :class:`.Table` construct. + +.. seealso:: + + :ref:`postgresql_table_options` + +:ticket:`2051` + +.. _feature_get_enums: + +New get_enums() method with Postgresql Dialect +---------------------------------------------- + +The :func:`.inspect` method returns a :class:`.PGInspector` object in the +case of Postgresql, which includes a new :meth:`.PGInspector.get_enums` +method that returns information on all available ``ENUM`` types:: + + from sqlalchemy import inspect, create_engine + + engine = create_engine("postgresql+psycopg2://host/dbname") + insp = inspect(engine) + print(insp.get_enums()) + +.. seealso:: + + :meth:`.PGInspector.get_enums` + +.. _feature_2891: -In the above statement we expect to see "ORDER BY id_count", as opposed to a -re-statement of the function. The string argument given is actively -matched to an entry in the columns clause during compilation, so the above -statement would produce as we expect, without warnings (though note that -the ``"name"`` expression has been resolved to ``users.name``!):: +Postgresql Dialect reflects Materialized Views, Foreign Tables +-------------------------------------------------------------- - SELECT users.name, count(users.id) AS id_count - FROM users GROUP BY users.name ORDER BY id_count +Changes are as follows: -However, if we refer to a name that cannot be located, then we get -the warning again, as below:: +* the :class:`Table` construct with ``autoload=True`` will now match a name + that exists in the database as a materialized view or foriegn table. - stmt = select([ - user.c.name, - func.count(user.c.id).label("id_count") - ]).order_by("some_label") +* :meth:`.Inspector.get_view_names` will return plain and materialized view + names. -The output does what we say, but again it warns us:: +* :meth:`.Inspector.get_table_names` does **not** change for Postgresql, it + continues to return only the names of plain tables. - SAWarning: Can't resolve label reference 'some_label'; converting to - text() (this warning may be suppressed after 10 occurrences) +* A new method :meth:`.PGInspector.get_foreign_table_names` is added which + will return the names of tables that are specifically marked as "foreign" + in the Postgresql schema tables. - SELECT users.name, count(users.id) AS id_count - FROM users ORDER BY some_label +The change to reflection involves adding ``'m'`` and ``'f'`` to the list +of qualifiers we use when querying ``pg_class.relkind``, but this change +is new in 1.0.0 to avoid any backwards-incompatible surprises for those +running 0.9 in production. -The above behavior applies to all those places where we might want to refer -to a so-called "label reference"; ORDER BY and GROUP BY, but also within an -OVER clause as well as a DISTINCT ON clause that refers to columns (e.g. the -Postgresql syntax). +:ticket:`2891` -We can still specify any arbitrary expression for ORDER BY or others using -:func:`.text`:: +.. _change_3264: - stmt = select([users]).order_by(text("some special expression")) +Postgresql ``has_table()`` now works for temporary tables +--------------------------------------------------------- -The upshot of the whole change is that SQLAlchemy now would like us -to tell it when a string is sent that this string is explicitly -a :func:`.text` construct, or a column, table, etc., and if we use it as a -label name in an order by, group by, or other expression, SQLAlchemy expects -that the string resolves to something known, else it should again -be qualified with :func:`.text` or similar. +This is a simple fix such that "has table" for temporary tables now works, +so that code like the following may proceed:: -:ticket:`2992` + from sqlalchemy import * -.. _change_3163: + metadata = MetaData() + user_tmp = Table( + "user_tmp", metadata, + Column("id", INT, primary_key=True), + Column('name', VARCHAR(50)), + prefixes=['TEMPORARY'] + ) -Event listeners can not be added or removed from within that event's runner ---------------------------------------------------------------------------- + e = create_engine("postgresql://scott:tiger@localhost/test", echo='debug') + with e.begin() as conn: + user_tmp.create(conn, checkfirst=True) -Removal of an event listener from inside that same event itself would -modify the elements of a list during iteration, which would cause -still-attached event listeners to silently fail to fire. To prevent -this while still maintaining performance, the lists have been replaced -with ``collections.deque()``, which does not allow any additions or -removals during iteration, and instead raises ``RuntimeError``. + # checkfirst will succeed + user_tmp.create(conn, checkfirst=True) -:ticket:`3163` +The very unlikely case that this behavior will cause a non-failing application +to behave differently, is because Postgresql allows a non-temporary table +to silently overwrite a temporary table. So code like the following will +now act completely differently, no longer creating the real table following +the temporary table:: -.. _change_3169: + from sqlalchemy import * -The INSERT...FROM SELECT construct now implies ``inline=True`` --------------------------------------------------------------- + metadata = MetaData() + user_tmp = Table( + "user_tmp", metadata, + Column("id", INT, primary_key=True), + Column('name', VARCHAR(50)), + prefixes=['TEMPORARY'] + ) -Using :meth:`.Insert.from_select` now implies ``inline=True`` -on :func:`.insert`. This helps to fix a bug where an -INSERT...FROM SELECT construct would inadvertently be compiled -as "implicit returning" on supporting backends, which would -cause breakage in the case of an INSERT that inserts zero rows -(as implicit returning expects a row), as well as arbitrary -return data in the case of an INSERT that inserts multiple -rows (e.g. only the first row of many). -A similar change is also applied to an INSERT..VALUES -with multiple parameter sets; implicit RETURNING will no longer emit -for this statement either. As both of these constructs deal -with varible numbers of rows, the -:attr:`.ResultProxy.inserted_primary_key` accessor does not -apply. Previously, there was a documentation note that one -may prefer ``inline=True`` with INSERT..FROM SELECT as some databases -don't support returning and therefore can't do "implicit" returning, -but there's no reason an INSERT...FROM SELECT needs implicit returning -in any case. Regular explicit :meth:`.Insert.returning` should -be used to return variable numbers of result rows if inserted -data is needed. + e = create_engine("postgresql://scott:tiger@localhost/test", echo='debug') + with e.begin() as conn: + user_tmp.create(conn, checkfirst=True) -:ticket:`3169` + m2 = MetaData() + user = Table( + "user_tmp", m2, + Column("id", INT, primary_key=True), + Column('name', VARCHAR(50)), + ) -.. _change_3027: + # in 0.9, *will create* the new table, overwriting the old one. + # in 1.0, *will not create* the new table + user.create(conn, checkfirst=True) -``autoload_with`` now implies ``autoload=True`` ------------------------------------------------ +:ticket:`3264` -A :class:`.Table` can be set up for reflection by passing -:paramref:`.Table.autoload_with` alone:: +.. _feature_gh134: - my_table = Table('my_table', metadata, autoload_with=some_engine) +Postgresql FILTER keyword +------------------------- -:ticket:`3027` +The SQL standard FILTER keyword for aggregate functions is now supported +by Postgresql as of 9.4. SQLAlchemy allows this using +:meth:`.FunctionElement.filter`:: -.. _change_3243: + func.count(1).filter(True) -ForeignKeyConstraint.columns is now a ColumnCollection ------------------------------------------------------- +.. seealso:: -:attr:`.ForeignKeyConstraint.columns` was previously a plain list -containing either strings or :class:`.Column` objects, depending on -how the :class:`.ForeignKeyConstraint` was constructed and whether it was -associated with a table. The collection is now a :class:`.ColumnCollection`, -and is only initialized after the :class:`.ForeignKeyConstraint` is -associated with a :class:`.Table`. A new accessor -:attr:`.ForeignKeyConstraint.column_keys` -is added to unconditionally return string keys for the local set of -columns regardless of how the object was constructed or its current -state. + :meth:`.FunctionElement.filter` + + :class:`.FunctionFilter` -Dialect Changes -=============== +Dialect Improvements and Changes - MySQL +============================================= MySQL internal "no such table" exceptions not passed to event handlers ---------------------------------------------------------------------- @@ -1634,6 +1650,48 @@ on MySQL:: :ticket:`3263` +.. _change_2984: + +Drizzle Dialect is now an External Dialect +------------------------------------------ + +The dialect for `Drizzle `_ is now an external +dialect, available at https://bitbucket.org/zzzeek/sqlalchemy-drizzle. +This dialect was added to SQLAlchemy right before SQLAlchemy was able to +accommodate third party dialects well; going forward, all databases that aren't +within the "ubiquitous use" category are third party dialects. +The dialect's implementation hasn't changed and is still based on the +MySQL + MySQLdb dialects within SQLAlchemy. The dialect is as of yet +unreleased and in "attic" status; however it passes the majority of tests +and is generally in decent working order, if someone wants to pick up +on polishing it. + +Dialect Improvements and Changes - SQLite +============================================= + +.. _change_2984: + +SQLite named and unnamed UNIQUE and FOREIGN KEY constraints will inspect and reflect +------------------------------------------------------------------------------------- + +UNIQUE and FOREIGN KEY constraints are now fully reflected on +SQLite both with and without names. Previously, foreign key +names were ignored and unnamed unique constraints were skipped. In particular +this will help with Alembic's new SQLite migration features. + +To achieve this, for both foreign keys and unique constraints, the result +of PRAGMA foreign_keys, index_list, and index_info is combined with regular +expression parsing of the CREATE TABLE statement overall to form a complete +picture of the names of constraints, as well as differentiating UNIQUE +constraints that were created as UNIQUE vs. unnamed INDEXes. + +:ticket:`3244` + +:ticket:`3261` + +Dialect Improvements and Changes - SQL Server +============================================= + .. _change_3182: PyODBC driver name is required with hostname-based SQL Server connections @@ -1660,44 +1718,8 @@ types has been changed for SQL Server 2012 and greater, with options to control the behavior completely, based on deprecation guidelines from Microsoft. See :ref:`mssql_large_type_deprecation` for details. -.. _change_3204: - -SQLite/Oracle have distinct methods for temporary table/view name reporting ---------------------------------------------------------------------------- - -The :meth:`.Inspector.get_table_names` and :meth:`.Inspector.get_view_names` -methods in the case of SQLite/Oracle would also return the names of temporary -tables and views, which is not provided by any other dialect (in the case -of MySQL at least it is not even possible). This logic has been moved -out to two new methods :meth:`.Inspector.get_temp_table_names` and -:meth:`.Inspector.get_temp_view_names`. - -Note that reflection of a specific named temporary table or temporary view, -either by ``Table('name', autoload=True)`` or via methods like -:meth:`.Inspector.get_columns` continues to function for most if not all -dialects. For SQLite specifically, there is a bug fix for UNIQUE constraint -reflection from temp tables as well, which is :ticket:`3203`. - -:ticket:`3204` - -SQLite named and unnamed UNIQUE and FOREIGN KEY constraints will inspect and reflect -------------------------------------------------------------------------------------- - -UNIQUE and FOREIGN KEY constraints are now fully reflected on -SQLite both with and without names. Previously, foreign key -names were ignored and unnamed unique constraints were skipped. In particular -this will help with Alembic's new SQLite migration features. - -To achieve this, for both foreign keys and unique constraints, the result -of PRAGMA foreign_keys, index_list, and index_info is combined with regular -expression parsing of the CREATE TABLE statement overall to form a complete -picture of the names of constraints, as well as differentiating UNIQUE -constraints that were created as UNIQUE vs. unnamed INDEXes. - -:ticket:`3244` - -:ticket:`3261` - +Dialect Improvements and Changes - Oracle +============================================= .. _change_3220: @@ -1725,19 +1747,3 @@ Keywords such as COMPRESS, ON COMMIT, BITMAP: :ref:`oracle_table_options` :ref:`oracle_index_options` - -.. _change_2984: - -Drizzle Dialect is now an External Dialect ------------------------------------------- - -The dialect for `Drizzle `_ is now an external -dialect, available at https://bitbucket.org/zzzeek/sqlalchemy-drizzle. -This dialect was added to SQLAlchemy right before SQLAlchemy was able to -accommodate third party dialects well; going forward, all databases that aren't -within the "ubiquitous use" category are third party dialects. -The dialect's implementation hasn't changed and is still based on the -MySQL + MySQLdb dialects within SQLAlchemy. The dialect is as of yet -unreleased and in "attic" status; however it passes the majority of tests -and is generally in decent working order, if someone wants to pick up -on polishing it. diff --git a/doc/build/orm/session.rst b/doc/build/orm/session.rst index 08ef9303ef..97506e2105 100644 --- a/doc/build/orm/session.rst +++ b/doc/build/orm/session.rst @@ -1944,7 +1944,7 @@ transactions set the flag ``twophase=True`` on the session:: # before committing both transactions session.commit() -.. _session_sql_expressions: +.. _flush_embedded_sql_expressions: Embedding SQL Insert/Update Expressions into a Flush ===================================================== @@ -2567,7 +2567,7 @@ The bulk insert / update methods lose a significant amount of functionality versus traditional ORM use. The following is a listing of features that are **not available** when using these methods: -* persistence along :meth:`.relationship` linkages +* persistence along :func:`.relationship` linkages * sorting of rows within order of dependency; rows are inserted or updated directly in the order in which they are passed to the methods @@ -2577,12 +2577,12 @@ are **not available** when using these methods: * Functionality related to primary key mutation, ON UPDATE cascade -* SQL expression inserts / updates (e.g. :ref:`session_sql_expressions`) +* SQL expression inserts / updates (e.g. :ref:`flush_embedded_sql_expressions`) * ORM events such as :meth:`.MapperEvents.before_insert`, etc. The bulk session methods have no event support. -Features that **are available** include:: +Features that **are available** include: * INSERTs and UPDATEs of mapped objects