doc updates

diff --git a/Documentation/RelNotes/v2.0.0.wip b/Documentation/RelNotes/v2.0.0.wip
index 41e8e29255cb19a35301767480f2b0192f849576..534cddf1056818373de7427d110605ff52f08cdc 100644 (file)
--- a/Documentation/RelNotes/v2.0.0.wip
+++ b/Documentation/RelNotes/v2.0.0.wip
@@ -9,11 +9,16 @@ around improved integration between inboxes and coderepos for
 solver.  Portability and reliability is also improved, especially
 in the internal process management of lei.
 
+Search now supports `thread:{SUBQUERY}' for installations
+with Xapian development headers (and necessary C++ toolchain).
+More Xapian functionality will be available in the future
+for users with development headers installed.
+
 public-inbox-cindex is a new command to index coderepos for
 WWW search and perform automatic associations between
-coderepos and inboxes.  This makes solver vastly more useful
-for the WWW UI as admins will no longer have to manually
-associate coderepos with inboxes.
+coderepos and inboxes.  This makes blob reconstruction via
+solver vastly more useful for the WWW UI as admins no longer
+have to manually associate coderepos with inboxes.
 
 public-inbox-clone gains the ability to mirror entire (or partial)
 grokmirror-compatible manifests.
@@ -26,7 +31,7 @@ extindex due to the need to interoperate with SHA-1 epochs.
 
 Upgrading:
 
-  lei users need to "lei daemon-kill" after installation to load
+  lei users must "lei daemon-kill" after installation to load
   new code.  Normal daemons (read-only, and public-inbox-watch)
   will also need restarts, of course, but there's no
   backwards-incompatible data format changes so rolling back to
@@ -63,6 +68,9 @@ treewide
 
   * embrace ->DESTROY in more places for control flow and reliability
 
+  * enforced error handling for ENOSPC, FS corruption, etc. for
+    parallel indexing
+
 PublicInbox::WWW
 
   * support `+' in inbox names
@@ -138,8 +146,8 @@ portability
 
   * various fixes for CentOS 7.x
 
-  * fix excessive pipelining to `git cat-file' on systems with small
-    getdelim(3) buffers (mainly affects musl)
+  * fix `git cat-file --batch' on systems with small getdelim(3)
+    buffers (mainly affects musl)
 
   * support Alpine Linux, Dragonfly, NetBSD and OpenBSD.  This resulted
     not only in bugfixes to our code, but also to Dragonfly and OpenBSD.
@@ -198,7 +206,11 @@ Search improvements (lei and PublicInbox::WWW)
 
   * --reindex and long-running (ext)index operations will auto
     checkpoint every 5s to avoid SQLite timeouts in public-facing
-    frontends where SQLite WAL isn't possible.
+    frontends where SQLite WAL isn't possible, use
+    `--commit=SECONDS'
+
+  * --wal switch added to public-inbox-init(1), public-inbox-index(1),
+    public-inbox-extindex(1), public-inbox-convert(1)
 
   * publicinbox.$NAME.altid is now supported with public-inbox-extindex
 

diff --git a/Documentation/public-inbox-extindex.pod b/Documentation/public-inbox-extindex.pod
index 42e2d3c560645b46b7c6185f70698e4480ea35df..136edf6f537563c2ac5f89c087ddab7bb86eb80f 100644 (file)
--- a/Documentation/public-inbox-extindex.pod
+++ b/Documentation/public-inbox-extindex.pod
@@ -36,6 +36,8 @@ along with L<DBD::SQLite> and L<DBI> Perl modules.
 
 =item --wal
 
+=item --commit SECONDS
+
 These switches behave as they do for L<public-inbox-index(1)>
 
 =item --all

diff --git a/Documentation/public-inbox-index.pod b/Documentation/public-inbox-index.pod
index c839f74ecafff423c554a80dfcaea23f6fd9ff94..53ed9e8812ba00a2d7bcc3901509b379a7139bc6 100644 (file)
--- a/Documentation/public-inbox-index.pod
+++ b/Documentation/public-inbox-index.pod
@@ -211,12 +211,28 @@ Passed directly to L<git-log(1)> to limit changes for C<--reindex>
 
 =item --wal
 
-Enable WAL (Write-Ahead-Log) on SQLite files.  This may reduce
+Enable WAL (Write-Ahead-Log) on SQLite files.  WAL may reduce
 fragmentation and improve performance; but requires read-only
-processes (e.g. public-inbox-netd) to have write access
-to all directories containing *.sqlite3 files.
+processes (e.g. L<public-inbox-netd(1)>) to have write access
+to all directories containing *.sqlite3 files.  This only
+needs to be used once as the setting is persistent.
 See L<https://sqlite.org/wal.html> for more info.
 
+Available in public-inbox 2.0.0+.
+
+=item --commit SECONDS
+
+Force commits after a given interval if the L</--batch-size> or
+L</publicinbox.indexBatchSize> threshold is not reached.  For
+non-WAL users, it should be under 30s to avoid the default
+L<DBD::SQLite(3pm)> busy timeout in read-only processes.
+Use a higher value if your index has no concurrent readers
+while indexing.
+
+Default: 15 seconds
+
+Available in public-inbox 2.0.0+.
+
 =back
 
 =head1 FILES
@@ -249,16 +265,18 @@ Default: none
 
 =item publicinbox.indexBatchSize
 
-Flushes changes to the filesystem and releases locks after
+Commits changes to the filesystem and releases locks after
 indexing the given number of bytes.  The default value of C<1m>
 (one megabyte) is low to minimize memory use and reduce
 contention with parallel invocations of L<public-inbox-mda(1)>,
 L<public-inbox-learn(1)>, and L<public-inbox-watch(1)>.
 
+L<--commit> may also be specified to force commits after a
+certain amount of time if using a large value, here.
+
 Increase this value on powerful systems to improve throughput at
 the expense of memory use.  The reduction of lock granularity
-may not be noticeable on fast systems.  With SSDs, values above
-C<4m> have little benefit.
+may not be noticeable on fast systems.
 
 For L<public-inbox-v2-format(5)> inboxes, this value is
 multiplied by the number of Xapian shards.  Thus a typical v2
@@ -273,7 +291,7 @@ in excess of 10x this value.
 This option is available in public-inbox 1.6 or later.
 public-inbox 1.5 and earlier used the current default, C<1m>.
 
-Default: 1m (one megabyte)
+Default: 1m (one megabyte) on 32-bit, 8m on 64-bit
 
 =item publicinbox.indexSequentialShard
 

diff --git a/TODO b/TODO
index 1932b25ffc946bf3761956f080265aaa8e7529cf..f85fc55425e56a9c45e9cd97d2f754b9bd6da1e8 100644 (file)
--- a/TODO
+++ b/TODO
@@ -21,13 +21,11 @@ all need to be considered for everything we introduce.)
   {over,msgmap}.sqlite3 (and Xapian) files can remain on a
   different FS than git repos.
 
-* support keeping CoW on btrfs for SQLite files,
-
 * investigate SQLite WAL feasibility with CoW enabled on btrfs
 
 * investigate fragmentation effects on btrfs even without CoW
 
-* adjustable commit interval for -index, -extindex and lei
+* adjustable commit interval for lei (done for -extindex + -index)
 
 * SQLite VACUUM support on -compact