Since we have changed the eofblocks handling in e2fsck (by removing
the need for the EOFBLOCKS_FL flag), we should have a test to check
out how we handle uninitialized and initialized blocks which are exist
beyond i_size, with files that have (and don't have) the EOFBLOCKS_FL.
Daniel Norberg [Tue, 10 Apr 2012 13:24:53 +0000 (15:24 +0200)]
com_err.pc: correct include path
compile_et puts #include <et/com_err.h> in
generated header files so pkg-config --cflags
com_err should provide the path to the directory
containing et/.
e2fsck.conf.5: add buggy_init_scripts to the man page
We have renamed buggy_init_scripts to accept_time_fudge. Explain this
so that people who find buggy_init_scripts in older e2fsck.conf files
understand what is going on.
libext2fs: improve testing coverage of tst_bitmaps
Improve the test coverage of tst_bitmaps by:
(a) adding the ability to test the legacy (32-bit) bitmap code
(b) adding tests for ext2fs_find_first_zero_inode_bitmap2() and
ext2fs_find_first_zero_block_bitmap2()
The recent regressions caused by the addition (and use) of
ext2fs_find_first_zero_inode_bitmap2() would have been caught if we
had added these tests first. (Another object lesson in why unit tests
are critically important!)
libext2fs: use correct types in ext2fs_find_first_zero_block_bitmap2()
Fortunately nothing was using this inline function, so we'll just fix
the types in its function signature, which were nonsensical (this was
caused by a cut-and-paste error).
libext2fs: add 32-bit compat code for ext2fs_find_first_zero_generic_bmap()
The lack of 32-bit support was causing febootstrap to crash since it
wasn't passing EXT2_FLAG_64BITS when opening the file system, so we
were still using the legacy bitmaps.
Also add support for bigalloc bitmap into the ffz functions.
Don't assume that the presence of mntent.h means that setmntent() exists
Change autoconf to test for setmntent() and use that to decide whether
to use getmntent() and setmntent(), since some systems don't have
setmntent() but they do have the mntent.h header file.
Also, remove the includes of mntent.h from e2fsck and mke2fs and other
places where it is not needed.
mke2fs: don't fail creating the journal if /etc/mtab is missing
The ext2fs_add_journal_inode() function calls
ext2fs_check_mount_point(), which can fail if /etc/mtab is missing.
This causes mke2fs to fail in the middle of the file system format
process; mke2fs calls ext2fs_check_mount_point() already (and has
appropriate fallbacks that calls fails), so add a flag so that mke2fs
can request ext2fs_add_journal_inode() to skip trying to call
e2fsck_check_mount_point().
Theodore Ts'o [Tue, 27 Mar 2012 18:48:18 +0000 (11:48 -0700)]
debian: add pointer for e2fsprogs-udeb to libcomerr2.shlibs
The udeb for btrfs-tools need libcom_err.so.2, which is packaged as a
part of e2fsprogs-udeb since we don't have a separate libcomerr2 udeb.
So we need to make sure the shlibs file has an explicit pointer to
handle this case.
Theodore Ts'o [Mon, 26 Mar 2012 23:31:06 +0000 (16:31 -0700)]
debian: compile without using dietlibc if SKIP_DIETLIBC=yes
Some distributions such as Ubuntu don't have dietlibc. If
SKIP_DIETLIBC=yes appears in rules.custom, this will change the
control file and cause the build rules to skip using dietlibc when
building the BUILD-BF version of e2fsprogs.
Theodore Ts'o [Tue, 27 Mar 2012 00:37:10 +0000 (17:37 -0700)]
debian: try to include rules.custom if it exists
The rules.custom file will never exist upstream, but by trying to
include it if it exists, it will make it easier to people create
custom builds of e2fsprogs by overriding various configuration
parameters in the debian rules file, while minimizing the chance of
future merge conflicts.
Theodore Ts'o [Tue, 27 Mar 2012 00:57:58 +0000 (17:57 -0700)]
mke2fs.conf: allow use of a custom mke2fs.conf file
Rename mke2fs.conf to mke2fs.conf.in, so that the makefile can choose
to use either mke2fs.conf.in or mke2fs.conf.custom.in (if it is
present). If there is custom configuration file, it's likely that it
is very different from the upstream mke2fs.conf.in, so by having the
separate mke2fs.conf.custom.in file, it minimizes merge conflicts if
the upstream mke2fs.conf file changes.
This function searches a bitmap for the first zero bit within a range.
It checks if there is a bitmap backend specific implementation
available (if the relevant field in bitmap_ops is non-NULL). If not,
it uses a generic and slow method by repeatedly calling test_bmap() in
a loop. Also change ext2fs_new_inode() to use this new function.
This change in itself does not result in a large speedup, rather it
refactors the code in preparation for the introduction of a faster
find_first_zero() for bitarray based bitmaps.
Signed-off-by: Sami Liedes <sami.liedes@iki.fi> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Sami Liedes [Thu, 22 Mar 2012 23:46:56 +0000 (19:46 -0400)]
resize2fs: use EXT2_FLAG_64BITS
By passing EXT2_FLAG_64BITS to ext2fs_open2() we can avoid some
unnecessary redirection in critical paths. While resize2fs does not
currently otherwise support so big filesystems that this would matter,
passing this flag is entirely harmless and only tells libext2fs that
the caller has been recompiled against current headers.
With this change the CPU time needed to shrink a 100G filesystem drops
by 20%.
Signed-off-by: Sami Liedes <sami.liedes@iki.fi> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Sami Liedes [Thu, 22 Mar 2012 23:42:38 +0000 (19:42 -0400)]
libext2fs: move a modulo operation out of a hot loop.
Filesystem shrinking in particular is a heavy user of this loop in
ext2fs_new_inode(). This change makes resize2fs use 24% less CPU time
for shrinking a 100G filesystem.
Signed-off-by: Sami Liedes <sami.liedes@iki.fi> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Lukas Czerner [Thu, 22 Mar 2012 23:42:11 +0000 (18:42 -0500)]
e2fsck: remove EXT4_EOFBLOCKS_FL flag handling
We've decided to remove EOFBLOCKS_FL from the ext4 file system entirely,
because it is not actually very useful and it is causing more problems
than it solves. We're going to remove it from e2fsprogs first and then
after the new e2fsprogs version is common enough we can remove the
kernel part as well.
This commit changes e2fsck to not check for EOFBLOCKS_FL. Instead we
simply search for initialized extents past the i_size as this should not
happen. Uninitialized extents can be past the i_size as we can do
fallocate with KEEP_SIZE flag.
Also remove the EXT4_EOFBLOCKS_FL from lib/ext2fs/ext2_fs.h since it is
no longer needed.
Theodore Ts'o [Thu, 22 Mar 2012 20:00:49 +0000 (16:00 -0400)]
mke2fs: add option to use direct I/O (mke2fs -D)
This can be useful when using mke2fs on loaded servers, since
otherwise mke2fs can dirty a huge amount of memory very quickly,
leading to other applications not being happy at all.
Theodore Ts'o [Thu, 22 Mar 2012 03:40:24 +0000 (23:40 -0400)]
e2fsck: report ext2fs_open2() and ext2fs_check_desc() errors
Print the actual errors returned by ext2fs_open2() and
ext2fs_check_desc() before we fall back to the backup block group
descriptors so that it's easier to see if there is some obscure
failure that is causing e2fsck to think that it should use the backup
block group descriptors.
Theodore Ts'o [Thu, 22 Mar 2012 03:18:49 +0000 (23:18 -0400)]
e2fsck: avoid unnecessary close/reopen for non-MMP filesystems
For file systems that do not use MMP, there's no reason to close the
file system and then re-open the file system a second time, since
EXT2_FLAG_SKIP_MMP has no meaning for non-MMP file systems anyway.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu> Cc: Andreas Dilger <adilger@whamcloud.com>
Andreas Dilger [Tue, 20 Mar 2012 03:54:03 +0000 (23:54 -0400)]
tests: add field sizes to inode/super struct tests
In addition to validating the ordering of fields within the inode
and superblock structures, also validate the field sizes. Otherwise
it is possible to incorrectly change the size of one of these fields
without getting any kind of error from these tests. Failures would
only show up later in the test image checks if the field that is
changed is before another in-use field.
Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Andreas Dilger [Mon, 19 Mar 2012 03:48:45 +0000 (23:48 -0400)]
tests: have "make testnew" target use local mke2fs
The "make testnew" target in the tests/ subdirectory (used to
help create new test cases) should use the locally-built mke2fs
binary instead of the installed system binary, to avoid cases
where the system binary does not support some new functionality
being tested.
Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Sun, 18 Mar 2012 03:21:00 +0000 (23:21 -0400)]
e2fsck: add logging capability
Add the ability to log messages about a file system to a specified
directory, using a file name templace that can be specified in
/etc/e2fsck.conf. This allows us to suppress the output of overly
verbose e2fsck outputs while still allowing the full logging output to
go to an appropriate file.
Theodore Ts'o [Wed, 14 Mar 2012 21:44:54 +0000 (17:44 -0400)]
e2fsck: add ability to limit the # of problems of a particular type
This throttles the output of a particular problem type, to avoid a
bottleneck caused by (for example) printing a large number of
characters over a rate-limited a serial console.
Theodore Ts'o [Sun, 11 Mar 2012 20:19:10 +0000 (16:19 -0400)]
e2fsck: check for zero length extent
If an extent has e_len set to zero, the kernel will oops with a
BUG_ON. Unfortunately, e2fsck wasn't catching this case. The kernel
needs to be fixed to notice this case and call ext4_error() instead of
failing an assertion check, but e2fsck should catch this case and
repair it (by deleting the errant extent).
Lukas Czerner [Sun, 11 Mar 2012 19:36:45 +0000 (15:36 -0400)]
e2fsck: do not discard itable if discard doen't zero data
We do not want to discard inode table if the underlying device does not
return zeros when reading non-provisioned blocks. The reason is that if
the inode table is not zeroed yet, then discard would not help us since
we would have to zero it anyway. In the case that inode table was
already zeroed, then the discard would cause subsequent reads to contain
non-deterministic data so we would not be able to assume that the inode
table was zeroed and we would need to zero it again, which does not
really make sense.
This commit adds check to prevent inode table from being discarded if
the discard does not zero data.
Lukas Czerner [Sun, 11 Mar 2012 19:35:06 +0000 (15:35 -0400)]
e2fsck: do not discard when in read only mode
When argument '-n' was specified and should run in read-only mode, we
should not attempt to discard anything. In order to do that we have to
check for E2F_OPT_NO flag and clear E2F_OPT_DISCARD flag if E2F_OPT_NO
is set.
This commit fixes the problem when we would mark inode tables as zeroed
(EXT2_BG_INODE_ZEROED) even when e2fsck is running in read-only mode. We
also move the check for E2F_OPT_NO so we can clear E2F_OPT_DISCARD as
early as possible.
Theodore Ts'o [Sun, 11 Mar 2012 18:15:19 +0000 (14:15 -0400)]
e2fsck: optimize CPU usage in check_{block,inode}_bitmaps()
The boolean expression (!skip_group || csum_flag) is always true,
since if csum_flag is FALSE, skip_group must also be FALSE. Hence, we
can just remove the expression from the conditional altogether, thus
simplifying the code and making it easier to read/understand.
Also, in the case where the bit is set in the bitmap, there's no point
repeatedly setting first_free to be ext2fs_block_count(fs->super).
Lukas Czerner [Mon, 5 Mar 2012 07:49:34 +0000 (08:49 +0100)]
e2fsck: do not forget to discard last block group
Previously when running e2fsck with '-E discard' argument the end of
the last group has not been discarded. This patch fixes it so we
always discard the end of the last group if needed.
Lukas Czerner [Sun, 11 Mar 2012 16:55:45 +0000 (12:55 -0400)]
e2fsck: discard only unused parts of inode table
When calling e2fsck with '-E discard' option it might happen that
valid inodes are discarded accidentally. This is because we just
discard the part of inode table which lies past the free inode count.
This is terribly wrong (sorry!).
This patch fixes it so only the free parts of an inode table
is discarded, leaving used inodes intact. This was tested with highly
fragmented inode tables with block size 4k and 1k.
Mike Frysinger [Wed, 7 Mar 2012 01:21:36 +0000 (20:21 -0500)]
libext2fs: check for fallocate symbol before using it
If we have newer kernel headers which define FALLOC_FL_PUNCH_HOLE, but we
are on an older glibc which lacks fallocate, we end up trying to use the
func anyways. Check the ifdef that autoconf already set up for us.
Andreas Dilger [Fri, 24 Feb 2012 07:43:52 +0000 (00:43 -0700)]
libquota: don't include system quota.h header
Don't use the system <sys/quota.h> header in mkquota.c, since there
is a local e2fsprogs version of quota.h that is already included and
has the desired quota constants, and avoids symbol conflicts with the
system <sys/quota.h> on other platforms (in particular OSX).
Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Mike Frysinger [Mon, 27 Feb 2012 06:12:53 +0000 (01:12 -0500)]
libext2fs: include sys/time.h for gettimeofday()
Building on my glibc-2.15 system hits a warning:
gen_bitmap64.c: In function 'ext2fs_alloc_generic_bmap':
gen_bitmap64.c:127:2: warning: implicit declaration of function
'gettimeofday' [-Wimplicit-function-declaration]
Include sys/time.h if it's available for the prototype.
Signed-off-by: Mike Frysinger <vapier@gentoo.org> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Phillip Susi [Mon, 27 Feb 2012 05:56:49 +0000 (00:56 -0500)]
e2image: skip uninitialized bitmaps and inode tables
When the block bitmap is uninitialized, skip copying it.
When the inode bitmap is uninitialized, skip copying it,
as well as the inode table. When there are unused inodes
towards the end of the table, skip those blocks too.
Theodore Ts'o [Mon, 27 Feb 2012 05:51:39 +0000 (00:51 -0500)]
mke2fs: make sure bg 0's unused inode count field is zero'ed for mke2fs -S
There was a bug/typo in commit ba9e0afc5 which caused the first block
group (bg #0) to not have its unused inode count field to get set to
zero in the case of mke2fs -S. This caused inodes in the first block
group to not be recoverable via mke2fs -S. Oops.
Theodore Ts'o [Tue, 21 Feb 2012 01:27:58 +0000 (20:27 -0500)]
libext2fs: don't mark the superblock as dirty if the fs was opened r/o
If the file system is read/only opened with a backup superblock, and
the file system has uninit_bg enabled, the super block must not be
marked as dirty; otherwise, ext2fs_close() will call ext2fs_flush(),
which will fail, since the file descriptor for the block device was
opened read/only, and then the file descriptor won't actually be
closed.
This is normally not a problem since most of the time the program will
exit shortly after calling ext2fs_close(), and many programs don't
bother checking the error return from ext2fs_close(), especially if
the file system was opened read/only.
A big exception to this is e2fsck, since it opens and close the file
systems during its startup, and to make matters worse, registers an
error handler which will noisly complain about the failed writes
caused by ext2fs_flush().
Fix this by not marking the superblock as dirty if the file system was
opened read/only. The changes to the block group descriptors to clear
the uninit bits will still happen, so that e2fsck -n will properly
scan the whole file system. However, those changes will get dropped
when the file system handle is closed.
Theodore Ts'o [Mon, 20 Feb 2012 23:18:13 +0000 (18:18 -0500)]
debian: fix dpkg-buildflags so it works when the shell is dash
The buildd's use dash as the shell, so we can't use >& in the rules
file. This was causing the buildd-built e2fsprogs binary packages to
be built without the hardening flags.
The replica is a feature which stores multiple copies of the key
metadata blocks so a single block failure in failure-prone media
(read: certain types of flash storage) doesn't take out the entire
file system.
Discussion on the upstream list proved not to be very positive on this
feature; the arguments were that it added complexity that wasn't
warrented, since common practice in industry is to insist on reliable
media, and if media is unreliable, you're kind of toast anyway (unless
the file system is being used as the back-end store of a cluster file
system where checksuming and data replication is happening above the
local disk file system level). So, this feature is being developed
out of tree.
We reserve the code points so that other people won't accidentally
step on them. Since it's not upstream, it's a soft reservation, but
it's not like we have any shortage of RO_COMPAT features. We are a
bit more tight on reserved inodes, but EXT2_BOOT_LOADER_INO and
EXT2_UNDEL_DIR_INO are not currently used anywhere, and
EXT2_EXCLUDE_INO is a reservation for another out-of-tree feature.
There are no features currently being discussed which require a
reserved inode, but if a need were to arise, we can claw back code
point reservations that were never used or not in tree, as those will
always be considered lower priority than in-tree features.
Cc: Aditya Kali <adityakali@google.com> Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Theodore Ts'o [Fri, 17 Feb 2012 14:23:12 +0000 (09:23 -0500)]
e2image: attempt to use ftruncate64 to set i_size for raw images
If ftruncate64() exists, try to use it to set i_size. This isn't
guaranteed to work, per SuSv3, but if it doesn't work, it's guaranteed
to return an error. So for file systems and/or operating systems that
don't support extending i_size via ftruncate64(), fall back to writing
the trailing null.
Theodore Ts'o [Fri, 17 Feb 2012 14:19:33 +0000 (09:19 -0500)]
e2image: fix logic bug which could cause a raw image not to be extended
If the size of the last "hole" in the raw file was an exact multiple
of a megabyte, then we wouldn't write a null at the end of the file in
order to extend the size of the raw image to correspond with the file
system size. Thanks to Lukas Czerner for suggesting the fix, and
Phillip Susi for pointing out the problem.
Theodore Ts'o [Fri, 17 Feb 2012 04:16:34 +0000 (23:16 -0500)]
mke2fs: allow file systems w/ uninit_bg to be recovered with mke2fs -S
The command mke2fs -S is used as a last ditch recovery command to
write new superblock and block group descriptors, but _not_ to destroy
the inode table in hopes of recovering from a badly corrupted file
system. If the uninit_bg feature is enabled, we need to make sure to
clear the unused inodes count field in the block group descriptors or
else e2fsck -fy will leave the file system completely empty.
Thanks to Akira Fujita for reporting this problem.
Jan Kara [Wed, 15 Feb 2012 23:41:33 +0000 (18:41 -0500)]
resize2fs: take s_first_data_block into account for resize2fs -M
calculate_minimum_resize_size() forgot to account s_first_data_block
into minimum filesystem size. Thus in case the size of filesystem was
such that the last group had the minimal size (50 blocks + metadata
overhead), the code in adjust_fs_info() decided the group is unneeded,
removed it, and in some cases the resizing then failed with ENOSPC.
Fix the issue by properly accounting for s_first_data_block in
calculate_minimum_resize_size().
Signed-off-by: Jan Kara <jack@suse.cz> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Wed, 15 Feb 2012 23:29:37 +0000 (18:29 -0500)]
debian: enable use of dpkg-buildflags if present
On newer Debian-based distributions, dpkg-buildflags should be called
by the debian rules file to get the appropriate values for CFLAGS,
LDFLAGS, etc. This will also allow Debian to build with security
hardened build flags, which is a release goal for Debian Wheezy.
If dpkg-buildflags is not present, then use some safe defaults (what
is currently being set by dpkg-buildpackage in Ubuntu 10.04 LTS, which
is a common enough ancient distribution that many people might still
care about).
Theodore Ts'o [Wed, 15 Feb 2012 22:25:32 +0000 (17:25 -0500)]
libext2fs: fix BLOCK_ABORT handling in the block iterator for extents
When processing files that contain extents, the block iterator
functions were not properly handling the BLOCK_ABORT bit. This could
cause problems such as ext2fs_link() adding a directory entry multiple
times.
Thanks to Darrick Wong <djwong@us.ibm.com> for reporting this.
Darrick J. Wong [Wed, 15 Feb 2012 20:11:46 +0000 (15:11 -0500)]
libext2fs: when truncating file, punch out blocks at end
Currently, ext2fs_file_set_size2 punches out data blocks between the
end of the file and infinity when truncate_block <= old_truncate
(i.e. when you've made the file longer). This is not a useful
behavior, particularly since it *fails* to punch out the data blocks
when the file is shortened (i.e. truncate_block < old_truncate). This
seems to be the result of the test being backwards, so fix the code to
punch only when the file is getting shorter.
Signed-off-by: Darrick J. Wong <djwong@us.ibm.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Eric Sandeen [Wed, 15 Feb 2012 20:11:35 +0000 (15:11 -0500)]
mke2fs: recognize mke4fs program name
We shipped "mke4fs" alongside mke2fs in RHEL5, so that ext4-capable
utilities could be installed without disturbing the venerable e2fsprogs-1.39
shipped in RHEL5 from the beginning. But it surprised some users that
"mke4fs" created ext2 filesystems by default rather than ext4.
While it was my intent to have the renamed binaries behave exactly
like the stock ones, it seems that there is some precedence for
handling "mkeNfs" in the code, so seems reasonable to add
mke4fs -> ext4 as well.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Tue, 14 Feb 2012 22:01:48 +0000 (17:01 -0500)]
debugfs: use ss_safe_getenv() instead of getenv()
In the case where debugfs (or rdebugfs) is installed setgid disk, or
some such, we need to disable the use of environment variables for the
obvious reasons.
Andreas Dilger [Wed, 8 Feb 2012 02:11:21 +0000 (21:11 -0500)]
libext2fs: quiet spurious group checksum errors
If we have to read the backup group descriptor checksums, the UNINIT
flags are cleared to ensure that all of the inodes in the filesystem
are scanned. However, the code that reset the UNINIT flags did not
reset the group checksum, and this produced many spurious error
messages in e2fsck.
Group descriptor 0 checksum is invalid. FIXED.
Group descriptor 1 checksum is invalid. FIXED.
:
:
Recompute checksums after modifying group descriptors to avoid these
error messages. Remove expected error messages in f_illitable_flexbg.
Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Mon, 6 Feb 2012 18:47:48 +0000 (13:47 -0500)]
e2fsck: allow the block bitmap to be uninit when the inode bitmap is in use
Removing this check will allow us to eventually eliminate code from
the kernel which forcibly initialized the block bitmap when the inode
bitmap is first used. This would eliminate a required journal credit
and extra disk write.