multi: process pending, one by one

Before curl 8.14.0, when pending was a list, `process_pending_handles()`
move a single transfer to processing. In 8.14.0 we changed that to move
all pending transfers to processing. This lead to unwanted performance
drops as reported in #18017.

Restore the old behaviour.

While the old behviour is better, the overall handling of "pending"
transfers is not optimal, since we do not keep track of the "condition"
a pending transfer is waiting on. This means, when moving a single,
pending transfer, we might move one that still cannot be processed while
another that could is kept pending.

Since we trigger `process_pending_handles()` from various changes, the
stalled pending will eventually make it to the processing queue, but
this is not optimal.

Fixes #18017
Reported-by: rm-rmonaghan on github
Closes #18056

diff --git a/lib/multi.c b/lib/multi.c
index 2042719d0675c9290faaed0e5cc2445896c5aeb6..79d44796ec871d6782b942230030ef617357d4b5 100644 (file)
--- a/lib/multi.c
+++ b/lib/multi.c
@@ -3672,9 +3672,13 @@ static void process_pending_handles(struct Curl_multi *multi)
   if(Curl_uint_bset_first(&multi->pending, &mid)) {
     do {
       struct Curl_easy *data = Curl_multi_get_easy(multi, mid);
-      DEBUGASSERT(data);
-      if(data)
+      if(data) {
         move_pending_to_connect(multi, data);
+        break;
+      }
+      /* transfer no longer known, should not happen */
+      Curl_uint_bset_remove(&multi->pending, mid);
+      DEBUGASSERT(0);
     }
     while(Curl_uint_bset_next(&multi->pending, mid, &mid));
   }