Don't apply sets or similar to objects in IdentitySet

Don't apply sets or similar to objects in IdentitySet

Modified the internal "identity set" implementation, which is a set that
hashes objects on their id() rather than their hash values, to not actually
call the ``__hash__()`` method of the objects, which are typically
user-mapped objects.  Some methods were calling this method as a side
effect of the implementation.

Fixes: #5304
Change-Id: I0ed8762f47622215a54dcad9f210377b1becf8e8