migration/postcopy: Spatial locality page hint for preempt mode

migration/postcopy: Spatial locality page hint for preempt mode

The preempt mode postcopy has been introduced for a while.  From latency
POV, it should always win the vanilla postcopy.

However there's one thing missing when preempt mode is enabled right now,
which is the spatial locality hint when there're page requests from the
destination side.

In vanilla postcopy, as long as a page request was unqueued, it will update
the PSS of the precopy background stream, so that after a page request the
background thread will move the pages after whatever was requested.  It's
pretty much a natural behavior when there's only one channel anyway, and
one scanner to send the pages.

Preempt mode didn't follow that, because preempt mode has its own channel
and its own PSS (which doesn't linearly scan the guest memory, but
dedicated to resolve page requested from destination).  So the page request
process and the background migration process are completely separate.

This patch adds the hint explicitly for preempt mode.  With that, whenever
the preempt mode receives a page request on the source, it will service the
remote page fault in the return path, then it'll provide a hint to the
background thread so that we'll start sending the pages right after the
requested ones in the background, assuming the follow up pages have a
higher chance to be accessed later.

NOTE: since the background migration thread and return path thread run
completely concurrently, it doesn't always mean the hint will be applied
every single time.  For example, it's possible that the return path thread
receives multiple page requests in a row without the background thread
getting the chance to consume one.  In such case, the preempt thread only
provide the hint if the previous hint has been consumed.  After all,
there's no point queuing hints when we only have one linear scanner.

This could measureably improve the simple sequential memory access pattern
during postcopy (when preempt is on).  For random accesses, I can measure a
slight increase of remote page fault latency from ~500us -> ~600us, that
could be a trade-off to have such hint mechanism, and after all that's
still greatly improved comparing to vanilla postcopy on random (~10ms).

The patch is verified by our QE team in a video streaming test case, to
reduce the pause of the video from ~1min to a few seconds when switching
over to postcopy with preempt mode.

Reported-by: Xiaohui Li <xiaohli@redhat.com>
Tested-by: Xiaohui Li <xiaohli@redhat.com>
Reviewed-by: Juraj Marcin <jmarcin@redhat.com>
Link: https://lore.kernel.org/r/20250424220705.195544-1-peterx@redhat.com
Signed-off-by: Peter Xu <peterx@redhat.com>