network/address: forget address even if we could not remove it

network/address: forget address even if we could not remove it

If we could not remove an address, then previously the corresponding
Address object was never removed, as it was freed only when we receive
remove notification from the kernel. So, we might confused that the
address still exists and being removed, and might block reconfiguring
the address.

With this change, even if we fail to remove an address, the
corresponding Address object will be freed.