[MINOR] move the listener reference from fd to session

[MINOR] move the listener reference from fd to session

The listener referenced in the fd was only used to check the
listener state upon session termination. There was no guarantee
that the FD had not been reassigned by the moment it was processed,
so this was a bit racy. Having it in the session is more robust.