From: Eric Wong <e@80x24.org>
Date: Fri, 10 Feb 2017 03:30:40 +0000 (+0000)
Subject: repo: search index flushes for excessive active refs
X-Git-Url: http://git.ipfire.org/?a=commitdiff_plain;h=7c3d8e174c15193851f811f22694ddb47cb11c65;p=thirdparty%2Fpublic-inbox.git

repo: search index flushes for excessive active refs

For certain repos, having too many active refs will cause
memory usage problems.  Mitigate the Xapian problems, for
now, and consider a switch to GDBM_File or similar for
repos with more refs.
---

diff --git a/lib/PublicInbox/RepoGitSearchIdx.pm b/lib/PublicInbox/RepoGitSearchIdx.pm
index 67d7ec3fd..d2b4597ed 100644
--- a/lib/PublicInbox/RepoGitSearchIdx.pm
+++ b/lib/PublicInbox/RepoGitSearchIdx.pm
@@ -345,10 +345,16 @@ sub index_top_ref ($$$) {
 		print $progress "$refname => $tip\n" if $progress;
 		replace_or_add($db, $doc_id, $ref_doc);
 	}
+	$db->flush;
 
 	# update all decorated refs which got snowballed into this one
 	delete $active->{$refname};
+	my $n = 100;
 	foreach my $ref (keys %$active) {
+		if (--$n <= 0) {
+			$db->flush;
+			$n = 100;
+		}
 		$ref_doc = get_doc($self, \$doc_id, 'ref', $ref);
 		$ref_doc->set_data($active->{$ref});
 		if ($progress) {