From: Darrick J. Wong Date: Fri, 11 Mar 2022 01:32:02 +0000 (-0800) Subject: mkfs: increase the minimum log size to 64MB when possible X-Git-Tag: libxfs-5.16-sync_2022-03-17~52 X-Git-Url: http://git.ipfire.org/gitweb.cgi?a=commitdiff_plain;h=6fb7848dfcc3d7afa8cc03335fe343977f895299;p=thirdparty%2Fxfsprogs-dev.git mkfs: increase the minimum log size to 64MB when possible Recently, the upstream maintainers have been taking a lot of heat on account of writer threads encountering high latency when asking for log grant space when the log is small. The reported use case is a heavily threaded indexing product logging trace information to a filesystem ranging in size between 20 and 250GB. The meetings that result from the complaints about latency and stall warnings in dmesg both from this use case and also a large well known cloud product are now consuming 25% of the maintainer's weekly time and have been for months. For small filesystems, the log is small by default because we have defaulted to a ratio of 1:2048 (or even less). For grown filesystems, this is even worse, because big filesystems generate big metadata. However, the log size is still insufficient even if it is formatted at the larger size. On a 220GB filesystem, the 99.95% latencies observed with a 200-writer file synchronous append workload running on a 44-AG filesystem (with 44 CPUs) spread across 4 hard disks showed: 99.5% Log(MB) Latency(ms) BW (MB/s) xlog_grant_head_wait 10 520 243 1875 20 220 308 540 40 140 360 6 80 92 363 0 160 86 364 0 For 4 NVME, the results were: 10 201 409 898 20 177 488 144 40 122 550 0 80 120 549 0 160 121 545 0 This shows pretty clearly that we could reduce the amount of time that threads spend waiting on the XFS log by increasing the log size to at least 40MB regardless of size. We then repeated the benchmark with a cloud system and an old machine to see if there were any ill effects on less stable hardware. For cloudy iscsi block storage, the results were: 10 390 176 2584 20 173 186 357 40 37 187 0 80 40 183 0 160 37 183 0 A decade-old machine w/ 24 CPUs and a giant spinning disk RAID6 array produced this: 10 55 5.4 0 20 40 5.9 0 40 62 5.7 0 80 66 5.7 0 160 25 5.4 0 From the first three scenarios, it is clear that there are gains to be had by sizing the log somewhere between 40 and 80MB -- the long tail latency drops quite a bit, and programs are no longer blocking on the log's transaction space grant heads. Split the difference and set the log size floor to 64MB. Signed-off-by: Darrick J. Wong --- diff --git a/mkfs/xfs_mkfs.c b/mkfs/xfs_mkfs.c index ad776492b..84dbb799f 100644 --- a/mkfs/xfs_mkfs.c +++ b/mkfs/xfs_mkfs.c @@ -18,6 +18,14 @@ #define GIGABYTES(count, blog) ((uint64_t)(count) << (30 - (blog))) #define MEGABYTES(count, blog) ((uint64_t)(count) << (20 - (blog))) +/* + * Realistically, the log should never be smaller than 64MB. Studies by the + * kernel maintainer in early 2022 have shown a dramatic reduction in long tail + * latency of the xlog grant head waitqueue when running a heavy metadata + * update workload when the log size is at least 64MB. + */ +#define XFS_MIN_REALISTIC_LOG_BLOCKS(blog) (MEGABYTES(64, (blog))) + /* * Use this macro before we have superblock and mount structure to * convert from basic blocks to filesystem blocks. @@ -3259,6 +3267,28 @@ validate_log_size(uint64_t logblocks, int blocklog, int min_logblocks) } } +/* + * Ensure that the log is large enough to provide reasonable performance on a + * modern system. + */ +static void +calc_realistic_log_size( + struct mkfs_params *cfg) +{ + unsigned int realistic_log_blocks; + + realistic_log_blocks = XFS_MIN_REALISTIC_LOG_BLOCKS(cfg->blocklog); + + /* + * If the "realistic" size is more than 7/8 of the AG, this is a tiny + * filesystem and we don't care. + */ + if (realistic_log_blocks > (cfg->agsize * 7 / 8)) + return; + + cfg->logblocks = max(cfg->logblocks, realistic_log_blocks); +} + static void clamp_internal_log_size( struct mkfs_params *cfg, @@ -3362,6 +3392,8 @@ _("external log device size %lld blocks too small, must be at least %lld blocks\ cfg->logblocks = cfg->logblocks >> cfg->blocklog; } + calc_realistic_log_size(cfg); + clamp_internal_log_size(cfg, mp, min_logblocks); validate_log_size(cfg->logblocks, cfg->blocklog, min_logblocks);