mkfs: increase the minimum log size to 64MB when possible

Recently, the upstream maintainers have been taking a lot of heat on
account of writer threads encountering high latency when asking for log
grant space when the log is small.  The reported use case is a heavily
threaded indexing product logging trace information to a filesystem
ranging in size between 20 and 250GB.  The meetings that result from the
complaints about latency and stall warnings in dmesg both from this use
case and also a large well known cloud product are now consuming 25% of
the maintainer's weekly time and have been for months.

For small filesystems, the log is small by default because we have
defaulted to a ratio of 1:2048 (or even less).  For grown filesystems,
this is even worse, because big filesystems generate big metadata.
However, the log size is still insufficient even if it is formatted at
the larger size.

On a 220GB filesystem, the 99.95% latencies observed with a 200-writer
file synchronous append workload running on a 44-AG filesystem (with 44
CPUs) spread across 4 hard disks showed:

	99.5%
Log(MB)	Latency(ms)	BW (MB/s)	xlog_grant_head_wait
10	520		243		1875
20	220		308		540
40	140		360		6
80	92		363		0
160	86		364		0

For 4 NVME, the results were:

10	201		409		898
20	177		488		144
40	122		550		0
80	120		549		0
160	121		545		0

This shows pretty clearly that we could reduce the amount of time that
threads spend waiting on the XFS log by increasing the log size to at
least 40MB regardless of size.  We then repeated the benchmark with a
cloud system and an old machine to see if there were any ill effects on
less stable hardware.

For cloudy iscsi block storage, the results were:

10	390		176		2584
20	173		186		357
40	37		187		0
80	40		183		0
160	37		183		0

A decade-old machine w/ 24 CPUs and a giant spinning disk RAID6 array
produced this:

10	55		5.4		0
20	40		5.9		0
40	62		5.7		0
80	66		5.7		0
160	25		5.4		0

From the first three scenarios, it is clear that there are gains to be
had by sizing the log somewhere between 40 and 80MB -- the long tail
latency drops quite a bit, and programs are no longer blocking on the
log's transaction space grant heads.  Split the difference and set the
log size floor to 64MB.

Signed-off-by: Darrick J. Wong <djwong@kernel.org>

diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c
index ad776492ba87741fc3fba408820a37e7dddbd390..84dbb799f3d2627893d2b98c6e8d8c05dbe3d730 100644 (file)
--- a/mkfs/xfs_mkfs.c
+++ b/mkfs/xfs_mkfs.c
@@ -18,6 +18,14 @@
 #define GIGABYTES(count, blog) ((uint64_t)(count) << (30 - (blog)))
 #define MEGABYTES(count, blog) ((uint64_t)(count) << (20 - (blog)))
 
+/*
+ * Realistically, the log should never be smaller than 64MB.  Studies by the
+ * kernel maintainer in early 2022 have shown a dramatic reduction in long tail
+ * latency of the xlog grant head waitqueue when running a heavy metadata
+ * update workload when the log size is at least 64MB.
+ */
+#define XFS_MIN_REALISTIC_LOG_BLOCKS(blog)     (MEGABYTES(64, (blog)))
+
 /*
  * Use this macro before we have superblock and mount structure to
  * convert from basic blocks to filesystem blocks.
@@ -3259,6 +3267,28 @@ validate_log_size(uint64_t logblocks, int blocklog, int min_logblocks)
        }
 }
 
+/*
+ * Ensure that the log is large enough to provide reasonable performance on a
+ * modern system.
+ */
+static void
+calc_realistic_log_size(
+       struct mkfs_params      *cfg)
+{
+       unsigned int            realistic_log_blocks;
+
+       realistic_log_blocks = XFS_MIN_REALISTIC_LOG_BLOCKS(cfg->blocklog);
+
+       /*
+        * If the "realistic" size is more than 7/8 of the AG, this is a tiny
+        * filesystem and we don't care.
+        */
+       if (realistic_log_blocks > (cfg->agsize * 7 / 8))
+               return;
+
+       cfg->logblocks = max(cfg->logblocks, realistic_log_blocks);
+}
+
 static void
 clamp_internal_log_size(
        struct mkfs_params      *cfg,
@@ -3362,6 +3392,8 @@ _("external log device size %lld blocks too small, must be at least %lld blocks\
                        cfg->logblocks = cfg->logblocks >> cfg->blocklog;
                }
 
+               calc_realistic_log_size(cfg);
+
                clamp_internal_log_size(cfg, mp, min_logblocks);
 
                validate_log_size(cfg->logblocks, cfg->blocklog, min_logblocks);