*index: ignore misordered References if In-Reply-To exists

Fix our indexers to favor In-Reply-To during the Message-ID
unique-fication phase.  Some MUAs will generate an incorrect
References header ordering and put its direct In-Reply-To at the
head of the References list instead of at the tail.

Having the same Message-ID in both References and In-Reply-To
inadvertently caused the In-Reply-To value to get dropped by the
`uniq_mids' subroutine.  To fix this, we reverse the values
before `uniq_mids' and reverse the result again since
PublicInbox::SearchThread (and Mail::Thread before) depends on
the In-Reply-To being last.

An expensive(*) --rethread --reindex will be required to fix
this on the existing data set.  I've only tested this change
with a three message inbox consisting of the following
Message-IDs:

	<153126248868.14533.9751473662727327569.stgit@warthog.procyon.org.uk>
	<CA+55aFzEjPUGZFk7PnM0T6YEn5uRrscgyCHyhc_cYz0m8ejdLA@mail.gmail.com>
	<29128.1531356361@warthog.procyon.org.uk>

Where <29128.1531356361@warthog.procyon.org.uk> is the message
with the out-of-order References header.

(*) - reindexing lore/all would take days or even a week :<

Reported-by: Askar Safin <safinaskar@zohomail.com>
Link: https://public-inbox.org/meta/198d28845f7.bd9650eb4485.7082558303836689927@zohomail.com/
Link: https://lore.kernel.org/linux-fsdevel/29128.1531356361@warthog.procyon.org.uk/raw

diff --git a/lib/PublicInbox/MID.pm b/lib/PublicInbox/MID.pm
index 36c058550cc3b83b02f05368e87d9cf3453f7758..a08c19e514f947c509c6a84457434776c98332ea 100644 (file)
--- a/lib/PublicInbox/MID.pm
+++ b/lib/PublicInbox/MID.pm
@@ -107,7 +107,12 @@ sub references ($) {
                        $addr{$_} = 1 for (PublicInbox::Address::emails($v));
                }
        }
-       uniq_mids(\@mids, \%addr);
+       @mids = reverse @mids; # keep IRT last (reversed again below)
+       my $ret = uniq_mids(\@mids, \%addr);
+
+       # preserve original ordering, PublicInbox::SearchThread favors last
+       @$ret = reverse @$ret;
+       $ret;
 }
 
 sub uniq_mids ($;$) {