When indexing millions of messages, Xapian has a tendency to
slow down as each shard gets bigger. --split-shards allows
Xapian to work on temporary shards (more akin to "epochs")
and uses xapian-compact(1) to commit the finalized changes
for readers. The result is roughly 2x faster for millions
of messages.
The downside of this switch is temporary space use increases by
2-3x and incremental changes are not visible to readers until
all indexing is complete. It has no useful effect on
--reindex, but --reindex is typically faster than initial
indexing anyways since space is already allocated.
It still takes days to create a new extindex of lore, but
fewer days than before.
Another beneficial side effect of this switch is it also
tends to reduce the effect of fragmentation for --cow users
on btrfs.