When the flow engine enters emergency mode, 3 things happen:
1. a different set of (lower) timeout values are applied
2. the flow manager runs more often
3. worker threads go get a flow directly from the hash table
Testing showed that performance went down significantly due to concurrency
issues:
1. worker threads would fight each other over the hash access
2. flow manager would get in the way of workers
This patch changes the behavior in 2 ways:
1. it makes the flow manager slightly less aggressive. It will still
try to run ~3 times per second, but no longer 10 times.
This should be reducing the contention. At the same time flows
won't time out faster if they are checked many times per second.
2. The 'get a used flow' logic optimizes the use of atomics by only
doing an atomic operation once, and while doing so reserving
a slice of the hash per worker.
The worker will also give up much quicker, to avoid the overhead
of hash walking and taking and releasing locks.
These combined changes show much better 'under stress' behavior, esp
on multi-NUMA systems.