disallow NUL characters in Message-ID and List-Id

While MTAs seem to stop '\0' from appearing in headers, users
fetching archives via git remain susceptible to having '\0' land
in archives.  So we'll filter them out of Xapian and SQLite DBs
to avoid interopability problems with CLI tools since there's no
known messages in lore or any of my archives which feature them.

Avoiding '\0' will ensure all indexed Message-IDs and List-Ids
can be specified from the command-line (although some characters
will still require $(printf) contortions).

As with Message-ID, List-Id fields with /\n\t\r/ characters will
also be stripped for indexing.  I will assume whatever went wrong
with the References: header in
<https://public-inbox.org/git/656C30A1EFC89F6B2082D9B6@localhost/raw>
could also happen to the List-Id header.

This is inspired by commit aca47e05a6026c12c768753c87e6ff769ef6bee4
(Import: Don't copy nulls from emails into git, 2018-07-07)

diff --git a/lib/PublicInbox/MID.pm b/lib/PublicInbox/MID.pm
index 97cf3a54225b6e6841e8c5d6c034d3e3f239a280..36c058550cc3b83b02f05368e87d9cf3453f7758 100644 (file)
--- a/lib/PublicInbox/MID.pm
+++ b/lib/PublicInbox/MID.pm
@@ -115,7 +115,7 @@ sub uniq_mids ($;$) {
        my @ret;
        $seen ||= {};
        foreach my $mid (@$mids) {
-               $mid =~ tr/\n\t\r//d;
+               $mid =~ tr/\n\t\r\0//d;
                if (length($mid) > MAX_MID_SIZE) {
                        warn "Message-ID: <$mid> too long, truncating\n";
                        $mid = substr($mid, 0, MAX_MID_SIZE);

diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 32598b7c02a1bf956690ef7799718b35d7199b5c..f569428cd28d565fef42dae20a8f98e08d59e069 100644 (file)
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -414,6 +414,7 @@ sub index_list_id ($$$) {
        for my $l ($hdr->header_raw('List-Id')) {
                $l =~ /<([^>]+)>/ or next;
                my $lid = lc $1;
+               $lid =~ tr/\n\t\r\0//d; # same rules as Message-ID
                $doc->add_boolean_term('G' . $lid);
                index_phrase($self, $lid, 1, 'XL'); # probabilistic
        }