From b4b5186b5c64abfe2ffc78e1495ed4cb478365f8 Mon Sep 17 00:00:00 2001 From: Eric Wong Date: Sun, 7 Sep 2025 10:00:19 +0000 Subject: [PATCH] doc updates --- Documentation/RelNotes/v2.0.0.wip | 26 ++++++++++++++------ Documentation/public-inbox-extindex.pod | 2 ++ Documentation/public-inbox-index.pod | 32 +++++++++++++++++++------ TODO | 4 +--- 4 files changed, 47 insertions(+), 17 deletions(-) diff --git a/Documentation/RelNotes/v2.0.0.wip b/Documentation/RelNotes/v2.0.0.wip index 41e8e2925..534cddf10 100644 --- a/Documentation/RelNotes/v2.0.0.wip +++ b/Documentation/RelNotes/v2.0.0.wip @@ -9,11 +9,16 @@ around improved integration between inboxes and coderepos for solver. Portability and reliability is also improved, especially in the internal process management of lei. +Search now supports `thread:{SUBQUERY}' for installations +with Xapian development headers (and necessary C++ toolchain). +More Xapian functionality will be available in the future +for users with development headers installed. + public-inbox-cindex is a new command to index coderepos for WWW search and perform automatic associations between -coderepos and inboxes. This makes solver vastly more useful -for the WWW UI as admins will no longer have to manually -associate coderepos with inboxes. +coderepos and inboxes. This makes blob reconstruction via +solver vastly more useful for the WWW UI as admins no longer +have to manually associate coderepos with inboxes. public-inbox-clone gains the ability to mirror entire (or partial) grokmirror-compatible manifests. @@ -26,7 +31,7 @@ extindex due to the need to interoperate with SHA-1 epochs. Upgrading: - lei users need to "lei daemon-kill" after installation to load + lei users must "lei daemon-kill" after installation to load new code. Normal daemons (read-only, and public-inbox-watch) will also need restarts, of course, but there's no backwards-incompatible data format changes so rolling back to @@ -63,6 +68,9 @@ treewide * embrace ->DESTROY in more places for control flow and reliability + * enforced error handling for ENOSPC, FS corruption, etc. for + parallel indexing + PublicInbox::WWW * support `+' in inbox names @@ -138,8 +146,8 @@ portability * various fixes for CentOS 7.x - * fix excessive pipelining to `git cat-file' on systems with small - getdelim(3) buffers (mainly affects musl) + * fix `git cat-file --batch' on systems with small getdelim(3) + buffers (mainly affects musl) * support Alpine Linux, Dragonfly, NetBSD and OpenBSD. This resulted not only in bugfixes to our code, but also to Dragonfly and OpenBSD. @@ -198,7 +206,11 @@ Search improvements (lei and PublicInbox::WWW) * --reindex and long-running (ext)index operations will auto checkpoint every 5s to avoid SQLite timeouts in public-facing - frontends where SQLite WAL isn't possible. + frontends where SQLite WAL isn't possible, use + `--commit=SECONDS' + + * --wal switch added to public-inbox-init(1), public-inbox-index(1), + public-inbox-extindex(1), public-inbox-convert(1) * publicinbox.$NAME.altid is now supported with public-inbox-extindex diff --git a/Documentation/public-inbox-extindex.pod b/Documentation/public-inbox-extindex.pod index 42e2d3c56..136edf6f5 100644 --- a/Documentation/public-inbox-extindex.pod +++ b/Documentation/public-inbox-extindex.pod @@ -36,6 +36,8 @@ along with L and L Perl modules. =item --wal +=item --commit SECONDS + These switches behave as they do for L =item --all diff --git a/Documentation/public-inbox-index.pod b/Documentation/public-inbox-index.pod index c839f74ec..53ed9e881 100644 --- a/Documentation/public-inbox-index.pod +++ b/Documentation/public-inbox-index.pod @@ -211,12 +211,28 @@ Passed directly to L to limit changes for C<--reindex> =item --wal -Enable WAL (Write-Ahead-Log) on SQLite files. This may reduce +Enable WAL (Write-Ahead-Log) on SQLite files. WAL may reduce fragmentation and improve performance; but requires read-only -processes (e.g. public-inbox-netd) to have write access -to all directories containing *.sqlite3 files. +processes (e.g. L) to have write access +to all directories containing *.sqlite3 files. This only +needs to be used once as the setting is persistent. See L for more info. +Available in public-inbox 2.0.0+. + +=item --commit SECONDS + +Force commits after a given interval if the L or +L threshold is not reached. For +non-WAL users, it should be under 30s to avoid the default +L busy timeout in read-only processes. +Use a higher value if your index has no concurrent readers +while indexing. + +Default: 15 seconds + +Available in public-inbox 2.0.0+. + =back =head1 FILES @@ -249,16 +265,18 @@ Default: none =item publicinbox.indexBatchSize -Flushes changes to the filesystem and releases locks after +Commits changes to the filesystem and releases locks after indexing the given number of bytes. The default value of C<1m> (one megabyte) is low to minimize memory use and reduce contention with parallel invocations of L, L, and L. +L<--commit> may also be specified to force commits after a +certain amount of time if using a large value, here. + Increase this value on powerful systems to improve throughput at the expense of memory use. The reduction of lock granularity -may not be noticeable on fast systems. With SSDs, values above -C<4m> have little benefit. +may not be noticeable on fast systems. For L inboxes, this value is multiplied by the number of Xapian shards. Thus a typical v2 @@ -273,7 +291,7 @@ in excess of 10x this value. This option is available in public-inbox 1.6 or later. public-inbox 1.5 and earlier used the current default, C<1m>. -Default: 1m (one megabyte) +Default: 1m (one megabyte) on 32-bit, 8m on 64-bit =item publicinbox.indexSequentialShard diff --git a/TODO b/TODO index 1932b25ff..f85fc5542 100644 --- a/TODO +++ b/TODO @@ -21,13 +21,11 @@ all need to be considered for everything we introduce.) {over,msgmap}.sqlite3 (and Xapian) files can remain on a different FS than git repos. -* support keeping CoW on btrfs for SQLite files, - * investigate SQLite WAL feasibility with CoW enabled on btrfs * investigate fragmentation effects on btrfs even without CoW -* adjustable commit interval for -index, -extindex and lei +* adjustable commit interval for lei (done for -extindex + -index) * SQLite VACUUM support on -compact -- 2.47.3