From: Eric Wong <e@80x24.org>
Date: Thu, 19 Jul 2018 03:21:38 +0000 (+0000)
Subject: searchidx: respect XAPIAN_FLUSH_THRESHOLD env if set
X-Git-Tag: v1.2.0~480
X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=7faaa2ab3464880aabbba909acd48a4b11b67452;p=thirdparty%2Fpublic-inbox.git

searchidx: respect XAPIAN_FLUSH_THRESHOLD env if set

Xapian documents and respect XAPIAN_FLUSH_THRESHOLD to define
the interval in documents to flush, so don't override it with
our own BATCH_BYTES.  This is helpful for initial indexing for
those on slower storage but enough RAM.

It is unnecessary for -watch and frequent incremental indexing;
and it increases transaction times if -watch is playing "catch-up"
if it was stopped for a while.

The original BATCH_BYTES was tuned for a machine with little
memory as the default XAPIAN_FLUSH_THRESHOLD of 10000 documents
was causing swap storms.  Using document counts also proved an
innaccurate estimator of RAM usage compared to the actual bytes
processed.
---

diff --git a/lib/PublicInbox/SearchIdx.pm b/lib/PublicInbox/SearchIdx.pm
index 04e853069..bb60506ce 100644
--- a/lib/PublicInbox/SearchIdx.pm
+++ b/lib/PublicInbox/SearchIdx.pm
@@ -22,7 +22,8 @@ require PublicInbox::Git;
 use Compress::Zlib qw(compress);
 
 use constant {
-	BATCH_BYTES => 1_000_000,
+	BATCH_BYTES => defined($ENV{XAPIAN_FLUSH_THRESHOLD}) ?
+			0x7fffffff : 1_000_000,
 	DEBUG => !!$ENV{DEBUG},
 };