Check collection less than two items remaining before firing scalar backref remove

Check collection less than two items remaining before firing scalar backref remove

Fixed long-standing issue where duplicate collection members would cause a
backref to delete the association between the member and its parent object
when one of the duplicates were removed, as occurs as a side effect of
swapping two objects in one statement.

Fixes: #1103
Change-Id: Ic12877f7bd5a4eb688091725a78410748e7fdf16