Theodore Ts'o [Tue, 4 Jan 2022 03:45:37 +0000 (22:45 -0500)]
tests: support older versions of timeout in r_corrupt_fs
Older versions of the timeout program in coreutils don't support the
-v option. (This is apparently still in use in the GNU/FreeBSD Debain
port since coreutils hasn't built successfully since Coreutils version
8.28.)
Theodore Ts'o [Tue, 28 Dec 2021 17:33:15 +0000 (12:33 -0500)]
reisze2fs: sanity check free block group counts when calculating minimum size
If one or more block group descriptor's free blocks count is insane,
it's possible this can lead to a infinite loop in the function
calculate_minimum_resize_size(), which is called by resize2fs -P or
resize2fs -M.
Add some sanity checks to avoid this. In the case where the file
system is corrupt, this will result in resize2fs -P reporting an
incorrect value, but that's OK, since when we try to do an actual
resize operation, resize2fs requires that the file system be freshly
checked using e2fsck.
https://github.com/tytso/e2fsprogs/issues/94
Fixes: ac94445fc01f ("resize2fs: make minimum size estimates more reliable for mounted fs") Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Tue, 21 Dec 2021 19:55:32 +0000 (14:55 -0500)]
setup-schroot: add some additional packages needed to build debian packages
On older Debian systems, "apt-get build-dep e2fsprogs" might not bring
in all of the packages needed to build in the most recent versions of
e2fsprogs. So explicitly try to install some additional packages
including dh-exec, udev, systemd, and cron.
Theodore Ts'o [Tue, 21 Dec 2021 19:28:51 +0000 (14:28 -0500)]
libuuid: try to use getrandom() or getentropy() if available
If getrandom() or getentropy() is available, use these interfaces in
favor of opening /dev/[u]random. This avoids a potential TSAN problem
that could potentially cause a fd leak when trying to open
/dev/urandom. (Which is not a disaster, but these interfaces are more
foolproof and avoids needing to open a file descriptor in a library,
which is a good thing.)
Theodore Ts'o [Sat, 11 Dec 2021 03:40:40 +0000 (22:40 -0500)]
e2fsck: update the bg_checksum after fixing problems in the bg descriptor
Otherwise, we break the block group descriptor's checksum, and while
this gets fixed by e2fsck, it results unnecessary messages printed or
questions asked of the system administrator.
Theodore Ts'o [Thu, 9 Dec 2021 15:55:54 +0000 (10:55 -0500)]
libext2fs: don't old the CACHE_MTX while doing I/O
A report a deadlock problem caused by I/O errors (caused by e2fsck's
error handler trying to write to a bad block to perform a forced
rewrite) uncovered that we were holding the CACHE_MTX while doing read
operations. This serialized read operations which destroyed the
performance benefits from doing parallel bitmap loading (or the
parallel e2fsck processing under development).
So restructure the code in unix_read_blk64() so that the read is
always done into the user-provided buffer, and then copied into the
cache afterwards.
e2fsck: skip sorting extents if there are no valid extents
At the end of a fast commit replay, e2fsck tries merging extents in a
inode. This patch fixes a bug in this logic where we were continuing
this action even if there were no extents to merge resulting in
accessing illegal memory.
Speed up an off-line resize of a 10GB file system to 64TB located on
tmpfs from 90 seconds to 16 seconds by extracting block group bitmaps
using a population count function to count the blocks in use instead
checking each bit in the block bitmap.
resize2fs: adjust new size of the file system to allow a successful resize
The previous commit in this series (commit 50088b1996cc: "resize2fs:
attempt to keep the # of inodes valid by removing the last bg") allows
a successful off-line resize of a file system with the default 16k
inode ratio to be grown to support a 64TB storage device by dropping
the last block group so the number of inodes is just below the maximum
2**32-1 number of inodes.
However, this is not a complete solution, for two reasons. First,
this adjustment happens after resize2fs has started potentially making
changes to the file system in the off-line (unmounted) case, which
means resize2fs will do a lot of unnecessary work. Secondly, in the
on-line resize case, passing the original requested size to the kernel
causes the kernel fail the online resize request.
So teach resize2fs to adjust the new size of the file system much
earlier, which avoids both problems.
resize2fs: attempt to keep the # of inodes valid by removing the last bg
If a the 10GB file system (with the default inode ratio size of 16k)
is resized to 64TB, the number of inodes will become 2**32 --- one
above the maximum allowed number of inodes of 2**32-1. In
adjust_fs_info(), we already try drop the last block group if there
isn't sufficient space in the last block group to support the metadata
for that block group. So if dropping the last block group allows the
number of inodes to valid, we should try that as well. In some cases
this will mean resizing a file system to 64TB will result in it be
resized to a size of 64TB - 128MB, which is close enough for
government work.
Jan Kara [Mon, 23 Aug 2021 15:41:27 +0000 (17:41 +0200)]
debugfs: Fix headers for quota commands
list_quota and get_quota commands have 'blocks' header while what they
actually show is a used space in bytes. Fix the header to state 'space'
instead.
Signed-off-by: Jan Kara <jack@suse.cz> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Jan Kara [Mon, 23 Aug 2021 15:41:25 +0000 (17:41 +0200)]
e2fsck: Do not trash user limits when processing orphan list
When e2fsck was loading quotas to process orphan list, it was loading
only quota usage. However subsequent quota writeout has effectively
overwritten quota limits, loosing them forever. Make sure quota limits
are preserved over orphan replay.
Signed-off-by: Jan Kara <jack@suse.cz> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Jan Kara [Mon, 23 Aug 2021 15:41:24 +0000 (17:41 +0200)]
tune2fs: Fix conversion of quota files
When tune2fs is enabling quota feature, it looks for old-style quota
files and tries to transfer limits stored in these files into newly
created hidded quota files. However the code doing the transfer setups
the quota scan wrongly and instead of transferring limits we transfer
usage. So not only quota limits are lost (at least they can still be
recovered from the old quota files) but also usage information may be
wrong if the accounting in e2fsprogs does not exactly match the
accounting in quota-tools (which is actually the case). Fix the setup of
the quota scan.
Signed-off-by: Jan Kara <jack@suse.cz> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Jan Kara [Mon, 23 Aug 2021 15:41:23 +0000 (17:41 +0200)]
quota: Rename quota_update_limits() to quota_read_all_dquots()
quota_update_limits() is a misnomer because what it actually does is
that it updates 'usage' counters and leaves 'limit' counters intact.
Rename quota_update_limits() to quota_read_all_dquots() and while
changing prototype also add a flags argument so that callers can control
which quota information is actually updated from the disk.
Signed-off-by: Jan Kara <jack@suse.cz> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Jan Kara [Mon, 23 Aug 2021 15:41:21 +0000 (17:41 +0200)]
quota: Add support to version 0 quota format
Version 0 quota format differs from version 1 by having only 32-bit
counters for inodes and block limits. For many installations this is not
limiting and thus the format is widely used. Also quota tools still
create quota files with this format by default. Add support for this
quota format to e2fsprogs so that we can seamlessly convert quota files
in this format into our internal quota files.
Signed-off-by: Jan Kara <jack@suse.cz> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Sun, 22 Aug 2021 14:07:15 +0000 (10:07 -0400)]
tests: update expect file for u_direct_io
The u_direct_io test is normally not run (since it requires root
privileges); as a result, when the mke2fs.conf defaults were changed,
I didn't notice that the expected output for u_direct_io test needed
to be updated.
Fixes: d730be5ceeba ("tests: update mke2fs.conf to create 256 byte inodes by default" Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Mon, 16 Aug 2021 12:18:59 +0000 (08:18 -0400)]
debian: switch to using build dependency on debhelper-compat
This is preferred in favor of using the debhelper/compat file, and we
no longer worry about supporting Debian Jessie or Debian Stretch
(at least without Stretch Backports).
Theodore Ts'o [Sat, 14 Aug 2021 21:07:53 +0000 (17:07 -0400)]
tests: update mke2fs.conf to create 256 byte inodes by default
The regression tests have their own private copy of mke2fs which is
used when tests create file systems. Since we are now using 256 byte
inodes by default, the tests should reflect this.
While we're at it, modify the r_move_itable test so it actually tests
moving the inode table.
Theodore Ts'o [Sat, 14 Aug 2021 14:39:13 +0000 (10:39 -0400)]
mke2fs: warn that bigalloc is experimental only for large cluster sizes
Since we have done a lot of testing with a cluster size equal to 64k
(or 16 times the default 4k block size), mke2fs will only warn for
bigalloc file systems where the cluster size is greater than 16 times
the block size.
Darrick J. Wong [Thu, 12 Aug 2021 23:22:22 +0000 (16:22 -0700)]
mke2fs: warn about missing y2038 support when formatting fresh ext4 fs
Filesystems with 128-byte inodes do not support timestamps beyond the
year 2038. Since we're now less than 16.5 years away from that point,
it's time to start warning users about this lack of support when they
format an ext4 filesystem with small inodes.
(Note that even for ext2 and ext3, we changed the default for
non-small file systems in 2008 in commit commit b1631cce648e ("Create
new filesystems with 256-byte inodes by default").)
So change the mke2fs.conf file to specify 256-byte inodes even for
small filesystems, and then add a warning to mke2fs itself if someone
is trying to make us format a file system with 128-byte inodes. This
can be suppressed by setting the boolean option warn_y2038_dates in
the mke2fs.conf file to false, which we do in the case of GNU Hurd,
since it only supports 128 byte inodes as of this writing.
[ Patch reworked by tytso to only warn in the case of GNU Hurd, since
the default for ext2/ext3 was changed for all but small file systems
in 2008. ]
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Lukas Czerner [Fri, 6 Aug 2021 09:58:17 +0000 (11:58 +0200)]
libext2fs: remove augmented rbtree functionality
Rbtree code was originally taken from linux kernel. This includes the
augmented rbtree functionality, however this was never intended to be
used and is not used still. Just remove it.
Lukas Czerner [Fri, 6 Aug 2021 09:58:16 +0000 (11:58 +0200)]
libext2fs: fix unexpected NULL variable
The ext2fs_check_mount_point() function can be called with mtpt being
NULL as for example from ext2fs_check_if_mounted(). However in the
is_swap_device condition we use the mtpt in strncpy without checking
whether it is non-null first.
This should not be a problem on linux since the previous attempt to open
the device exclusively would have prevented us from ever reaching the
problematic strncpy. However it's still a bug and can cause problems on
other systems, fix it by conditioning strncpy on mtpt not being null.
Darrick J. Wong [Thu, 5 Aug 2021 15:43:28 +0000 (08:43 -0700)]
tests: skip u_direct_io if losetup fails
This new test requires a loop device to run testing. While it checks
for some "obvious" parameters that might cause the test to fail such as
not being root and no losetup executable, it doesn't actually check that
the losetup -a call succeeds. This causes a test regression in my
package building container (where there is only a minimal /dev with no
loop devices available) so I can't build debian packages.
Fix the test to skip out if we can't create a loop device.
Signed-off-by: Darrick J. Wong <djwong@kernel.org> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Zhiqiang Liu [Wed, 28 Jul 2021 01:56:47 +0000 (09:56 +0800)]
hashmap: change return value type of ext2fs_hashmap_add()
In ext2fs_hashmap_add(), new entry is allocated by calling
malloc(). If malloc() return NULL, it will cause a
segmentation fault problem.
Here, we change return value type of ext2fs_hashmap_add()
from void to int. If allocating new entry fails, we will
return -1, and the callers should also verify the return
value of ext2fs_hashmap_add().
Theodore Ts'o [Tue, 3 Aug 2021 01:08:01 +0000 (21:08 -0400)]
mke2fs: fix creating a file system image w/o a pre-existing file
The mke2fs program should allow creating a file system image when an
explicit file system size is specified, even if the file doesn't yet
exist. By deferring the call to check_plausible() in commit 942b00cb9d2f ("mke2fs: do not warn about a pre-existing partition
table when using a non-zero offset") this behaviour was broken.
Fix this regression by explicitly creating the file if the file system
size is specified.
Samuel Holland [Sat, 29 May 2021 03:14:04 +0000 (22:14 -0500)]
libext2fs: improve error handling in POSIX ACL conversions
When encoding a POSIX ACL to the EXT4 ACL format, if an unknown tag
is encountered, that entry is silently ignored. It would be better
to return an error to inform the user that the ACL is incompatible.
Also fix the mismatched indentation in the opposite function.
Signed-off-by: Samuel Holland <samuel@sholland.org>
setup-schroot: install the acl and libreadline-dev packages
The acl package is needed to run the m_rootdir_acl test. The
libreadline-dev package will drag in the shared library package for
libreadline (libreadlineN) so that the developer running debugfs will
be able to use line editing.
libext2fs: fix translation of Posix ACL's on big-endian systems
The ACL returned by the kernel in lgetxattr(2) is returned in Little
Endian, even on Big Endian systems. Fix the functions
convert_posix_acl_to_disk_buffer() and convert_disk_buffer_to_posix_acl()
to work correctly on Big Endian systems. This fixes a failure of
the test m_rootdir_acl.