Added new attribute event :meth:`.AttributeEvents.bulk_replace`.
This event is triggered when a collection is assigned to a
relationship, before the incoming collection is compared with the
existing one. This early event allows for conversion of incoming
non-ORM objects as well. The event is integrated with the
``@validates`` decorator.
The ``@validates`` decorator now allows the decorated method to receive
objects from a "bulk collection set" operation that have not yet
been compared to the existing collection. This allows incoming values
to be converted to compatible ORM objects as is already allowed
from an "append" event. Note that this means that the
``@validates`` method is called for **all** values during a collection
assignment, rather than just the ones that are new.
Change-Id: I27f59db008d9e521d31a3e30143d7cd997e4b7b3
Fixes: #3896
don't match the actual names of the columns. Previously this would
do a simple name-based match and fail with an AttributeError.
+ .. change:: 3896_a
+ :tags: feature, orm
+ :tickets: 3896
+
+ Added new attribute event :meth:`.AttributeEvents.bulk_replace`.
+ This event is triggered when a collection is assigned to a
+ relationship, before the incoming collection is compared with the
+ existing one. This early event allows for conversion of incoming
+ non-ORM objects as well. The event is integrated with the
+ ``@validates`` decorator.
+
+ .. seealso::
+
+ :ref:`change_3896_event`
+
+ .. change:: 3896_b
+ :tags: bug, orm
+ :tickets: 3896
+
+ The ``@validates`` decorator now allows the decorated method to receive
+ objects from a "bulk collection set" operation that have not yet
+ been compared to the existing collection. This allows incoming values
+ to be converted to compatible ORM objects as is already allowed
+ from an "append" event. Note that this means that the
+ ``@validates`` method is called for **all** values during a collection
+ assignment, rather than just the ones that are new.
+
+ .. seealso::
+
+ :ref:`change_3896_validates`
+
.. change:: 3938
:tags: bug, engine
:tickets: 3938
New Features and Improvements - ORM
===================================
+.. _change_3896_validates:
+
+A @validates method receives all values on bulk-collection set before comparison
+--------------------------------------------------------------------------------
+
+A method that uses ``@validates`` will now receive all members of a collection
+during a "bulk set" operation, before comparison is applied against the
+existing collection.
+
+Given a mapping as::
+
+ class A(Base):
+ __tablename__ = 'a'
+ id = Column(Integer, primary_key=True)
+ bs = relationship("B")
+
+ @validates('bs')
+ def convert_dict_to_b(self, key, value):
+ return B(data=value['data'])
+
+ class B(Base):
+ __tablename__ = 'b'
+ id = Column(Integer, primary_key=True)
+ a_id = Column(ForeignKey('a.id'))
+ data = Column(String)
+
+Above, we could use the validator as follows, to convert from an incoming
+dictionary to an instance of ``B`` upon collection append::
+
+ a1 = A()
+ a1.bs.append({"data": "b1"})
+
+However, a collection assignment would fail, since the ORM would assume
+incoming objects are already instances of ``B`` as it attempts to compare them
+to the existing members of the collection, before doing collection appends
+which actually invoke the validator. This would make it impossible for bulk
+set operations to accomodate non-ORM objects like dictionaries that needed
+up-front modification::
+
+ a1 = A()
+ a1.bs = [{"data": "b1"}]
+
+The new logic uses the new :meth:`.AttributeEvents.bulk_replace` event to ensure
+that all values are sent to the ``@validates`` function up front.
+
+As part of this change, this means that validators will now receive
+**all** members of a collection upon bulk set, not just the members that
+are new. Supposing a simple validator such as::
+
+ class A(Base):
+ # ...
+
+ @validates('bs')
+ def validate_b(self, key, value):
+ assert value.data is not None
+ return value
+
+Above, if we began with a collection as::
+
+ a1 = A()
+
+ b1, b2 = B(data="one"), B(data="two")
+ a1.bs = [b1, b2]
+
+And then, replaced the collection with one that overlaps the first::
+
+ b3 = B(data="three")
+ a1.bs = [b2, b3]
+
+Previously, the second assignment would trigger the ``A.validate_b``
+method only once, for the ``b3`` object. The ``b2`` object would be seen
+as being already present in the collection and not validated. With the new
+behavior, both ``b2`` and ``b3`` are passed to ``A.validate_b`` before passing
+onto the collection. It is thus important that valiation methods employ
+idempotent behavior to suit such a case.
+
+.. seealso::
+
+ :ref:`change_3896_event`
+
+:ticket:`3896`
+
+.. _change_3896_event:
+
+New bulk_replace event
+----------------------
+
+To suit the validation use case described in :ref:`change_3896_validates`,
+a new :meth:`.AttributeEvents.bulk_replace` method is added, which is
+called in conjunction with the :meth:`.AttributeEvents.append` and
+:meth:`.AttributeEvents.remove` events. "bulk_replace" is called before
+"append" and "remove" so that the collection can be modified ahead of comparison
+to the existing collection. After that, individual items
+are appended to a new target collection, firing off the "append"
+event for items new to the collection, as was the previous behavior.
+Below illustrates both "bulk_replace" and
+"append" at the same time, including that "append" will receive an object
+already handled by "bulk_replace" if collection assignment is used.
+A new symbol :attr:`~.attributes.OP_BULK_REPLACE` may be used to determine
+if this "append" event is the second part of a bulk replace::
+
+ from sqlalchemy.orm.attributes import OP_BULK_REPLACE
+
+ @event.listens_for(SomeObject.collection, "bulk_replace")
+ def process_collection(target, values, initiator):
+ values[:] = [_make_value(value) for value in values]
+
+ @event.listens_for(SomeObject.collection, "append", retval=True)
+ def process_collection(target, value, initiator):
+ # make sure bulk_replace didn't already do it
+ if initiator is None or initiator.op is not OP_BULK_REPLACE:
+ return _make_value(value)
+ else:
+ return value
+
+
+:ticket:`3896`
+
New Features and Improvements - Core
====================================
OP_REMOVE = util.symbol("REMOVE")
OP_APPEND = util.symbol("APPEND")
OP_REPLACE = util.symbol("REPLACE")
+OP_BULK_REPLACE = util.symbol("BULK_REPLACE")
class Event(object):
:var impl: The :class:`.AttributeImpl` which is the current event
initiator.
- :var op: The symbol :attr:`.OP_APPEND`, :attr:`.OP_REMOVE` or
- :attr:`.OP_REPLACE`, indicating the source operation.
+ :var op: The symbol :attr:`.OP_APPEND`, :attr:`.OP_REMOVE`,
+ :attr:`.OP_REPLACE`, or :attr:`.OP_BULK_REPLACE`, indicating the
+ source operation.
"""
iterable = iter(iterable)
new_values = list(iterable)
+ evt = Event(self, OP_BULK_REPLACE)
+
+ self.dispatch.bulk_replace(state, new_values, evt)
+
old = self.get(state, dict_, passive=PASSIVE_ONLY_PERSISTENT)
if old is PASSIVE_NO_RESULT:
old = self.initialize(state, dict_)
dict_[self.key] = user_data
collections.bulk_replace(
- new_values, old_collection, new_collection)
+ new_values, old_collection, new_collection,
+ initiator=evt)
del old._sa_adapter
self.dispatch.dispose_collection(state, old, old_collection)
impl = old_state.manager[key].impl
if initiator.impl is not impl or \
- initiator.op not in (OP_REPLACE, OP_REMOVE):
+ initiator.op is OP_APPEND:
impl.pop(old_state,
old_dict,
state.obj(),
initiator.parent_token is not child_impl.parent_token:
_acceptable_key_err(state, initiator, child_impl)
elif initiator.impl is not child_impl or \
- initiator.op not in (OP_APPEND, OP_REPLACE):
+ initiator.op is OP_REMOVE:
child_impl.append(
child_state,
child_dict,
initiator.parent_token is not child_impl.parent_token:
_acceptable_key_err(state, initiator, child_impl)
elif initiator.impl is not child_impl or \
- initiator.op not in (OP_APPEND, OP_REPLACE):
+ initiator.op is OP_REMOVE:
child_impl.append(
child_state,
child_dict,
instance_dict(child)
child_impl = child_state.manager[key].impl
if initiator.impl is not child_impl or \
- initiator.op not in (OP_REMOVE, OP_REPLACE):
+ initiator.op is OP_APPEND:
child_impl.pop(
child_state,
child_dict,
self.attr = getattr(d['owner_cls'], self._key).impl
-def bulk_replace(values, existing_adapter, new_adapter):
+def bulk_replace(values, existing_adapter, new_adapter, initiator=None):
"""Load a new collection, firing events based on prior like membership.
Appends instances in ``values`` onto the ``new_adapter``. Events will be
for member in values or ():
if member in additions:
- appender(member)
+ appender(member, _sa_initiator=initiator)
elif member in constants:
appender(member, _sa_initiator=False)
if existing_adapter:
for member in removals:
- existing_adapter.fire_remove_event(member)
+ existing_adapter.fire_remove_event(member, initiator=initiator)
def prepare_instrumentation(factory):
def append(self, target, value, initiator):
"""Receive a collection append event.
+ The append event is invoked for each element as it is appended
+ to the collection. This occurs for single-item appends as well
+ as for a "bulk replace" operation.
+
:param target: the object instance receiving the event.
If the listener is registered with ``raw=True``, this will
be the :class:`.InstanceState` object.
:param initiator: An instance of :class:`.attributes.Event`
representing the initiation of the event. May be modified
from its original value by backref handlers in order to control
- chained event propagation.
-
- .. versionchanged:: 0.9.0 the ``initiator`` argument is now
- passed as a :class:`.attributes.Event` object, and may be
- modified by backref handlers within a chain of backref-linked
- events.
-
+ chained event propagation, as well as be inspected for information
+ about the source of the event.
:return: if the event was registered with ``retval=True``,
the given value, or a new effective value, should be returned.
+ .. seealso::
+
+ :meth:`.AttributeEvents.bulk_replace`
+
+ """
+
+ def bulk_replace(self, target, values, initiator):
+ """Receive a collection 'bulk replace' event.
+
+ This event is invoked for a sequence of values as they are incoming
+ to a bulk collection set operation, which can be
+ modified in place before the values are treated as ORM objects.
+ This is an "early hook" that runs before the bulk replace routine
+ attempts to reconcile which objects are already present in the
+ collection and which are being removed by the net replace operation.
+
+ It is typical that this method be combined with use of the
+ :meth:`.AttributeEvents.append` event. When using both of these
+ events, note that a bulk replace operation will invoke
+ the :meth:`.AttributeEvents.append` event for all new items,
+ even after :meth:`.AttributeEvents.bulk_replace` has been invoked
+ for the collection as a whole. In order to determine if an
+ :meth:`.AttributeEvents.append` event is part of a bulk replace,
+ use the symbol :attr:`~.attributes.OP_BULK_REPLACE` to test the
+ incoming initiator::
+
+ from sqlalchemy.orm.attributes import OP_BULK_REPLACE
+
+ @event.listens_for(SomeObject.collection, "bulk_replace")
+ def process_collection(target, values, initiator):
+ values[:] = [_make_value(value) for value in values]
+
+ @event.listens_for(SomeObject.collection, "append", retval=True)
+ def process_collection(target, value, initiator):
+ # make sure bulk_replace didn't already do it
+ if initiator is None or initiator.op is not OP_BULK_REPLACE:
+ return _make_value(value)
+ else:
+ return value
+
+
+
+ .. versionadded:: 1.2
+
+ :param target: the object instance receiving the event.
+ If the listener is registered with ``raw=True``, this will
+ be the :class:`.InstanceState` object.
+ :param value: a sequence (e.g. a list) of the values being set. The
+ handler can modify this list in place.
+ :param initiator: An instance of :class:`.attributes.Event`
+ representing the initiation of the event.
+
"""
def remove(self, target, value, initiator):
if include_removes:
def append(state, value, initiator):
- if include_backrefs or not detect_is_backref(state, initiator):
+ if (
+ initiator.op is not attributes.OP_BULK_REPLACE and
+ (include_backrefs or not detect_is_backref(state, initiator))
+ ):
return validator(state.obj(), key, value, False)
else:
return value
+ def bulk_set(state, values, initiator):
+ if include_backrefs or not detect_is_backref(state, initiator):
+ obj = state.obj()
+ values[:] = [
+ validator(obj, key, value, False) for value in values]
+
def set_(state, value, oldvalue, initiator):
if include_backrefs or not detect_is_backref(state, initiator):
return validator(state.obj(), key, value, False)
else:
def append(state, value, initiator):
- if include_backrefs or not detect_is_backref(state, initiator):
+ if (
+ initiator.op is not attributes.OP_BULK_REPLACE and
+ (include_backrefs or not detect_is_backref(state, initiator))
+ ):
return validator(state.obj(), key, value)
else:
return value
+ def bulk_set(state, values, initiator):
+ if include_backrefs or not detect_is_backref(state, initiator):
+ obj = state.obj()
+ values[:] = [
+ validator(obj, key, value) for value in values]
+
def set_(state, value, oldvalue, initiator):
if include_backrefs or not detect_is_backref(state, initiator):
return validator(state.obj(), key, value)
return value
event.listen(desc, 'append', append, raw=True, retval=True)
+ event.listen(desc, 'bulk_replace', bulk_set, raw=True)
event.listen(desc, 'set', set_, raw=True, retval=True)
if include_removes:
event.listen(desc, "remove", remove, raw=True, retval=True)
call('addresses', a1, False), call('addresses', a2, False),
# set to [a2, a3] - this is a remove of a1,
# append of a3. the appends are first.
+ # in 1.2 due to #3896, we also get 'a2' in the
+ # validates as it is part of the set
+ call('addresses', a2, False),
call('addresses', a3, False),
call('addresses', a1, True),
])
+ def test_validator_bulk_collection_set(self):
+ users, addresses, Address = (self.tables.users,
+ self.tables.addresses,
+ self.classes.Address)
+
+ class User(fixtures.ComparableEntity):
+
+ @validates('addresses', include_removes=True)
+ def validate_address(self, key, item, remove):
+ if not remove:
+ assert isinstance(item, str)
+ else:
+ assert isinstance(item, Address)
+ item = Address(email_address=item)
+ return item
+
+ mapper(User, users, properties={
+ 'addresses': relationship(Address)
+ })
+ mapper(Address, addresses)
+
+ u1 = User()
+ u1.addresses.append("e1")
+ u1.addresses.append("e2")
+ eq_(
+ u1.addresses,
+ [Address(email_address="e1"), Address(email_address="e2")]
+ )
+ u1.addresses = ["e3", "e4"]
+ eq_(
+ u1.addresses,
+ [Address(email_address="e3"), Address(email_address="e4")]
+ )
+
def test_validator_multi_warning(self):
users = self.tables.users
projects / thirdparty / sqlalchemy / sqlalchemy.git / commitdiff
