Fix ENTRY_ABORTED assertion in sendClientOldEntry() (#1903)

    FATAL: assertion failed: client_side_reply.cc:392:
    "!EBIT_TEST(http->storeEntry()->flags, ENTRY_ABORTED)"

The replaced assertion was wrong because a stale entry may be aborted
while we are revalidating it. The exact real-world conditions that
triggered this assertion are unknown, but many conditions lead to
aborted entries. The assertion can be triggered in lab tests using a hit
transaction that collapses on a miss transaction that runs into storage
problems (e.g., a memory cache that ran out of usable space).

Also adjusted cacheHit() to avoid similar problems. We have not
reproduced them, but no code maintains the asserted invariant on the
cacheHit() path either. Moreover, async-called cacheHit() initiates
processExpired() that leads to problematic sendClientOldEntry() call, so
seeing an aborted StoreEntry at cacheHit() time is probably a matter of
sufficient concurrency levels and asynchronous callback delays.

diff --git a/src/client_side_reply.cc b/src/client_side_reply.cc
index 3deab499dc8c187293f2bb63b4186e1833c012f7..db7d80d78c546821744fada7a487c445c00d3624 100644 (file)
--- a/src/client_side_reply.cc
+++ b/src/client_side_reply.cc
@@ -388,8 +388,15 @@ clientReplyContext::sendClientOldEntry()
 {
     /* Get the old request back */
     restoreState();
+
+    if (EBIT_TEST(http->storeEntry()->flags, ENTRY_ABORTED)) {
+        debugs(88, 3, "stale entry aborted while we revalidated: " << *http->storeEntry());
+        http->updateLoggingTags(LOG_TCP_MISS);
+        processMiss();
+        return;
+    }
+
     /* here the data to send is in the next nodes buffers already */
-    assert(!EBIT_TEST(http->storeEntry()->flags, ENTRY_ABORTED));
     Assure(matchesStreamBodyBuffer(lastStreamBufferedBytes));
     Assure(!lastStreamBufferedBytes.offset);
     sendMoreData(lastStreamBufferedBytes);
@@ -551,7 +558,12 @@ clientReplyContext::cacheHit(const StoreIOBuffer result)
         return;
     }
 
-    assert(!EBIT_TEST(e->flags, ENTRY_ABORTED));
+    if (EBIT_TEST(e->flags, ENTRY_ABORTED)) {
+        debugs(88, 3, "refusing aborted " << *e);
+        http->updateLoggingTags(LOG_TCP_MISS);
+        processMiss();
+        return;
+    }
 
     /*
      * Got the headers, now grok them