When delay pools are active on a CONNECT tunnel and the pool is drained
the I/O loop cycles very often transferring 1 byte until the pool is
topped-up at the end of the second.
Instead of looping constantly trying to read 1 byte at a time, add an
asynchronous event to wait for a few I/O cycles or until more bytes can
be read.
To protect against infinite loops of waiting when many tunnels are
competing for the pool allowance we only delay for a limited number of
loops before allowing at least 1 byte through. Also, the amount of time
waited is an odd fraction of 1 second so re-tries naturally spread
across any given second fairly, with connections rotating closer or
further from the time when pool topup happens. That behaviour still
allows some variance in service times, but overall the CPU consumption
and (as a result) total proxy speed appears to be much improved.
NP: Initial production testing shows a 36% RPS speed increase,
with a 50% reduction in total CPU usage.