Do not assert that the close handler being removed must be in
the list because comm_close removes all close handlers before any FD handlers
are fired.
There also seems to be an unrelated(?) problem: comm_remove_close_handler
does not really remove the callback. It only cancels the call. It should
probably remove the callback as well to prevent an unlikely situation where
the close handler list grows "too much".