From: Mike Bayer Date: Wed, 19 Oct 2016 16:52:55 +0000 (-0400) Subject: Rewrite migration notes for [ticket:3514] X-Git-Tag: rel_1_1_3~9 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=76ec285ba452acf36d725799896904477a9c2dbd;p=thirdparty%2Fsqlalchemy%2Fsqlalchemy.git Rewrite migration notes for [ticket:3514] The change to "evaluates none" datatypes in the ORM was not fully described in the migration notes, missing the key behavioral change that a column which is missing a default entirely will not receive a value for a missing JSON column now. The issue here touched upon a revisit of the assumptions in [ticket:3514], but overall the old behavior "worked" mostly because the ORM wants to explicitly render NULL into an INSERT for column values that are missing, which itself is a legacy behavior which should be considered for possible removal in a future major release. Given that "missing ORM value + no column default set up == dont put it in the INSERT" would be the most intuitive behavior, the move in [ticket:3514] represents a step in this direction. Change-Id: I454d5bb0773bd73d9864925dcc47f1f0810e33ba Fixes: #3830 --- diff --git a/doc/build/changelog/migration_11.rst b/doc/build/changelog/migration_11.rst index eddb2030b8..ce17fd3f64 100644 --- a/doc/build/changelog/migration_11.rst +++ b/doc/build/changelog/migration_11.rst @@ -1676,30 +1676,64 @@ NULL values as well as expression handling. .. _change_3514: -JSON "null" is inserted as expected with ORM operations, regardless of column default present -^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ +JSON "null" is inserted as expected with ORM operations, omitted when not present +^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^ The :class:`.types.JSON` type and its descendant types :class:`.postgresql.JSON` and :class:`.mysql.JSON` have a flag :paramref:`.types.JSON.none_as_null` which when set to True indicates that the Python value ``None`` should translate into a SQL NULL rather than a JSON NULL value. This flag defaults to False, -which means that the column should *never* insert SQL NULL or fall back -to a default unless the :func:`.null` constant were used. However, this would -fail in the ORM under two circumstances; one is when the column also contained -a default or server_default value, a positive value of ``None`` on the mapped -attribute would still result in the column-level default being triggered, +which means that the Python value ``None`` should result in a JSON NULL value. + +This logic would fail, and is now corrected, in the following circumstances: + +1. When the column also contained a default or server_default value, +a positive value of ``None`` on the mapped attribute that expects to persist +JSON "null" would still result in the column-level default being triggered, replacing the ``None`` value:: + class MyObject(Base): + # ... + + json_value = Column(JSON(none_as_null=False), default="some default") + + # would insert "some default" instead of "'null'", + # now will insert "'null'" obj = MyObject(json_value=None) session.add(obj) - session.commit() # would fire off default / server_default, not encode "'none'" + session.commit() + +2. When the column *did not* contain a default or server_default value, a missing +value on a JSON column configured with none_as_null=False would still render +JSON NULL rather than falling back to not inserting any value, behaving +inconsistently vs. all other datatypes:: -The other is when the :meth:`.Session.bulk_insert_mappings` -method were used, ``None`` would be ignored in all cases:: + class MyObject(Base): + # ... + + some_other_value = Column(String(50)) + json_value = Column(JSON(none_as_null=False)) + + # would result in NULL for some_other_value, + # but json "'null'" for json_value. Now results in NULL for both + # (the json_value is omitted from the INSERT) + obj = MyObject() + session.add(obj) + session.commit() +This is a behavioral change that is backwards incompatible for an application +that was relying upon this to default a missing value as JSON null. This +essentially establishes that a **missing value is distinguished from a present +value of None**. See :ref:`behavior_change_3514` for further detail. + +3. When the :meth:`.Session.bulk_insert_mappings` method were used, ``None`` +would be ignored in all cases:: + + # would insert SQL NULL and/or trigger defaults, + # now inserts "'null'" session.bulk_insert_mappings( MyObject, - [{"json_value": None}]) # would insert SQL NULL and/or trigger defaults + [{"json_value": None}]) The :class:`.types.JSON` type now implements the :attr:`.TypeEngine.should_evaluate_none` flag, @@ -1708,18 +1742,6 @@ automatically based on the value of :paramref:`.types.JSON.none_as_null`. Thanks to :ticket:`3061`, we can differentiate when the value ``None`` is actively set by the user versus when it was never set at all. -If the attribute is not set at all, then column level defaults *will* -fire off and/or SQL NULL will be inserted as expected, as was the behavior -previously. Below, the two variants are illustrated:: - - obj = MyObject(json_value=None) - session.add(obj) - session.commit() # *will not* fire off column defaults, will insert JSON 'null' - - obj = MyObject() - session.add(obj) - session.commit() # *will* fire off column defaults, and/or insert SQL NULL - The feature applies as well to the new base :class:`.types.JSON` type and its descendant types. @@ -2063,6 +2085,53 @@ as intended by the :func:`.type_coerce` function. Key Behavioral Changes - ORM ============================ +.. _behavior_change_3514: + +JSON Columns will not insert JSON NULL if no value is supplied and no default is established +-------------------------------------------------------------------------------------------- + +As detailed in :ref:`change_3514`, :class:`.types.JSON` will not render +a JSON "null" value if the value is missing entirely. To prevent SQL NULL, +a default should be set up. Given the following mapping:: + + class MyObject(Base): + # ... + + json_value = Column(JSON(none_as_null=False), nullable=False) + +The following flush operation will fail with an integrity error:: + + obj = MyObject() # note no json_value + session.add(obj) + session.commit() # will fail with integrity error + +If the default for the column should be JSON NULL, set this on the +Column:: + + class MyObject(Base): + # ... + + json_value = Column( + JSON(none_as_null=False), nullable=False, default=JSON.NULL) + +Or, ensure the value is present on the object:: + + obj = MyObject(json_value=None) + session.add(obj) + session.commit() # will insert JSON NULL + +Note that setting ``None`` for the default is the same as omitting it entirely; +the :paramref:`.types.JSON.none_as_null` flag does not impact the value of ``None`` +passed to :paramref:`.Column.default` or :paramref:`.Column.server_default`:: + + # default=None is the same as omitting it entirely, does not apply JSON NULL + json_value = Column(JSON(none_as_null=False), nullable=False, default=None) + + +.. seealso:: + + :ref:`change_3514` + .. _change_3641: Columns no longer added redundantly with DISTINCT + ORDER BY diff --git a/lib/sqlalchemy/orm/persistence.py b/lib/sqlalchemy/orm/persistence.py index 24a33ee8dc..2bc189c1d3 100644 --- a/lib/sqlalchemy/orm/persistence.py +++ b/lib/sqlalchemy/orm/persistence.py @@ -396,6 +396,12 @@ def _collect_insert_commands( params[col.key] = value if not bulk: + # for all the columns that have no default and we don't have + # a value and where "None" is not a special value, add + # explicit None to the INSERT. This is a legacy behavior + # which might be worth removing, as it should not be necessary + # and also produces confusion, given that "missing" and None + # now have distinct meanings for colkey in mapper._insert_cols_as_none[table].\ difference(params).difference(value_params): params[colkey] = None diff --git a/lib/sqlalchemy/sql/sqltypes.py b/lib/sqlalchemy/sql/sqltypes.py index 118c26070a..cae23902b0 100644 --- a/lib/sqlalchemy/sql/sqltypes.py +++ b/lib/sqlalchemy/sql/sqltypes.py @@ -1796,6 +1796,13 @@ class JSON(Indexable, TypeEngine): from sqlalchemy import null conn.execute(table.insert(), data=null()) + .. note:: + + :paramref:`.JSON.none_as_null` does **not** apply to the + values passed to :paramref:`.Column.default` and + :paramref:`.Column.server_default`; a value of ``None`` passed for + these parameters means "no default present". + .. seealso:: :attr:`.types.JSON.NULL` diff --git a/lib/sqlalchemy/sql/type_api.py b/lib/sqlalchemy/sql/type_api.py index 217f7016b2..689b4c79bd 100644 --- a/lib/sqlalchemy/sql/type_api.py +++ b/lib/sqlalchemy/sql/type_api.py @@ -182,6 +182,13 @@ class TypeEngine(Visitable): the :obj:`~.expression.null` SQL construct in an INSERT statement or associated with an ORM-mapped attribute. + .. note:: + + The "evaulates none" flag does **not** apply to a value + of ``None`` passed to :paramref:`.Column.default` or + :paramref:`.Column.server_default`; in these cases, ``None`` + still means "no default". + .. versionadded:: 1.1 .. seealso::