Also cleanup the space for the rbt nodes

Also cleanup the space for the rbt nodes

As we are in overmem state we want to free more memory than we are
adding so we need to add in an allowance for the rbtnodes that may
have been added and the names stored with them.  There is the node
for the owner name and a possible ENT node if there was a node split.

(cherry picked from commit 7e462c2b26299f77944fb225923b7b01e0017856)