Theodore Ts'o [Wed, 2 May 2012 18:21:28 +0000 (14:21 -0400)]
Support systems without posix_memalign() and memalign()
MacOS 10.5 doesn't have posix_memalign() nor memalign(), but it does
have valloc(). The Android SDK would like to be built on MacOS 10.5,
so I've added support for a good-enough emulation of memalign()'s
functionality using valloc(), with an explicit test to make sure
valloc() is returning a pointer which is sufficiently aligned given
the requested alignment. This won't work if you try to operate on a
file system with a 16k blocksize using an e2fsprogs built on MacOS
10.5 system, but it is good enough for the common case of 4k
blocksize file systems, and we will let the memory allocation fail in
the alignment is not good enough.
I've also added a unit test for ext2fs_get_memalign() so we can be
sure it's working as expected. I've tested the code paths with
HAVE_POSIX_MEMALIGN defined, HAVE_POSIX_MEMALIGN undefined, and
HAVE_POSIX_MEMALIGN and HAVE_MEMALIGN undefined on an x86 Linux
system, and so I know the valloc() code path works OK. The simplistic
(and less safe) patch at:
Theodore Ts'o [Mon, 7 May 2012 18:41:49 +0000 (14:41 -0400)]
libext2fs: factor out I/O buffer allocation
Create a new function, io_channel_alloc_buf() which allocates I/O
buffers with appropriate alignment if we are using direct I/O. The
original code was sometimes using a larger alignment factor than
necessary, and would always request an aligned memory buffer even when
it was not necessary since the block device was not opened with
O_DIRECT.
Theodore Ts'o [Mon, 7 May 2012 17:25:44 +0000 (13:25 -0400)]
libext2fs: refactor Direct I/O alignment requirement calculations
Create a new function, ext2fs_get_dio_alignment(), which returns the
alignment requirements for direct I/O. This way we can factor out the
code from MMP and the Unix I/O manager. The two modules weren't
consistently calculating the alignment factors, and in particular MMP
would sometimes use a larger alignment factor than was strictly
necessary.
Theodore Ts'o [Mon, 7 May 2012 16:56:07 +0000 (12:56 -0400)]
libext2fs: move the alignment field from unix_io to the io_manager
The align field which indicated the required data alignment of data
buffers was stored in a field specific to the unix_io manager. Move
it to the top-level io_channel structure so it can be better
generalized.
Theodore Ts'o [Wed, 2 May 2012 20:28:48 +0000 (16:28 -0400)]
tests: add new tests f_jnl_32bit and f_jnl_64bit
Add regression tests which make sure e2fsprogs understands the current
32-bit and 64-bit journal format. If a patch breaks the on-disk
format, these tests should warn us about that.
Andreas Dilger [Tue, 24 Apr 2012 20:22:48 +0000 (16:22 -0400)]
e2fsck: quiet harmless inode/blocks errors
Don't consider only an error in the superblock summary as incorrect.
The kernel does not update this field except at unmount time, so
don't print errors during a "-n" run if there is nothing else wrong.
Any other unfixed errors will themselves mark the filesystem invalid.
Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Aditya Kali [Tue, 24 Apr 2012 18:46:08 +0000 (14:46 -0400)]
e2fsck,libquota: Update quota only if its inconsistent
Currently fsck recomputes quotas and overwrites quota files
whenever its run. This causes unnecessary modification of
filesystem even when quotas were never inconsistent. We also
lose the limits information because of this. With this patch,
e2fsck compares the computed quotas to the on-disk quotas
(while updating the in-memory limits) and writes out the
quota inode only if it is inconsistent.
Signed-off-by: Aditya Kali <adityakali@google.com> Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Aditya Kali [Tue, 24 Apr 2012 18:36:42 +0000 (14:36 -0400)]
e2fsck: Fix check for hidden quota files
Currently e2fsck always incorrectly detects that quota inodes
need to be hidden (even if they are already hidden) and
modifies the superblock unnecessarily. This patch fixes the
check for hidden quota files and avoids modifying the
filesystem if quota inodes are already hidden.
Also, zero-out the old quota inode so that next fsck scan
doesn't complain.
Signed-off-by: Aditya Kali <adityakali@google.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Since we have changed the eofblocks handling in e2fsck (by removing
the need for the EOFBLOCKS_FL flag), we should have a test to check
out how we handle uninitialized and initialized blocks which are exist
beyond i_size, with files that have (and don't have) the EOFBLOCKS_FL.
Daniel Norberg [Tue, 10 Apr 2012 13:24:53 +0000 (15:24 +0200)]
com_err.pc: correct include path
compile_et puts #include <et/com_err.h> in
generated header files so pkg-config --cflags
com_err should provide the path to the directory
containing et/.
e2fsck.conf.5: add buggy_init_scripts to the man page
We have renamed buggy_init_scripts to accept_time_fudge. Explain this
so that people who find buggy_init_scripts in older e2fsck.conf files
understand what is going on.
libext2fs: improve testing coverage of tst_bitmaps
Improve the test coverage of tst_bitmaps by:
(a) adding the ability to test the legacy (32-bit) bitmap code
(b) adding tests for ext2fs_find_first_zero_inode_bitmap2() and
ext2fs_find_first_zero_block_bitmap2()
The recent regressions caused by the addition (and use) of
ext2fs_find_first_zero_inode_bitmap2() would have been caught if we
had added these tests first. (Another object lesson in why unit tests
are critically important!)
libext2fs: use correct types in ext2fs_find_first_zero_block_bitmap2()
Fortunately nothing was using this inline function, so we'll just fix
the types in its function signature, which were nonsensical (this was
caused by a cut-and-paste error).
libext2fs: add 32-bit compat code for ext2fs_find_first_zero_generic_bmap()
The lack of 32-bit support was causing febootstrap to crash since it
wasn't passing EXT2_FLAG_64BITS when opening the file system, so we
were still using the legacy bitmaps.
Also add support for bigalloc bitmap into the ffz functions.
Don't assume that the presence of mntent.h means that setmntent() exists
Change autoconf to test for setmntent() and use that to decide whether
to use getmntent() and setmntent(), since some systems don't have
setmntent() but they do have the mntent.h header file.
Also, remove the includes of mntent.h from e2fsck and mke2fs and other
places where it is not needed.
mke2fs: don't fail creating the journal if /etc/mtab is missing
The ext2fs_add_journal_inode() function calls
ext2fs_check_mount_point(), which can fail if /etc/mtab is missing.
This causes mke2fs to fail in the middle of the file system format
process; mke2fs calls ext2fs_check_mount_point() already (and has
appropriate fallbacks that calls fails), so add a flag so that mke2fs
can request ext2fs_add_journal_inode() to skip trying to call
e2fsck_check_mount_point().
Theodore Ts'o [Tue, 27 Mar 2012 18:48:18 +0000 (11:48 -0700)]
debian: add pointer for e2fsprogs-udeb to libcomerr2.shlibs
The udeb for btrfs-tools need libcom_err.so.2, which is packaged as a
part of e2fsprogs-udeb since we don't have a separate libcomerr2 udeb.
So we need to make sure the shlibs file has an explicit pointer to
handle this case.
Theodore Ts'o [Mon, 26 Mar 2012 23:31:06 +0000 (16:31 -0700)]
debian: compile without using dietlibc if SKIP_DIETLIBC=yes
Some distributions such as Ubuntu don't have dietlibc. If
SKIP_DIETLIBC=yes appears in rules.custom, this will change the
control file and cause the build rules to skip using dietlibc when
building the BUILD-BF version of e2fsprogs.
Theodore Ts'o [Tue, 27 Mar 2012 00:37:10 +0000 (17:37 -0700)]
debian: try to include rules.custom if it exists
The rules.custom file will never exist upstream, but by trying to
include it if it exists, it will make it easier to people create
custom builds of e2fsprogs by overriding various configuration
parameters in the debian rules file, while minimizing the chance of
future merge conflicts.
Theodore Ts'o [Tue, 27 Mar 2012 00:57:58 +0000 (17:57 -0700)]
mke2fs.conf: allow use of a custom mke2fs.conf file
Rename mke2fs.conf to mke2fs.conf.in, so that the makefile can choose
to use either mke2fs.conf.in or mke2fs.conf.custom.in (if it is
present). If there is custom configuration file, it's likely that it
is very different from the upstream mke2fs.conf.in, so by having the
separate mke2fs.conf.custom.in file, it minimizes merge conflicts if
the upstream mke2fs.conf file changes.
This function searches a bitmap for the first zero bit within a range.
It checks if there is a bitmap backend specific implementation
available (if the relevant field in bitmap_ops is non-NULL). If not,
it uses a generic and slow method by repeatedly calling test_bmap() in
a loop. Also change ext2fs_new_inode() to use this new function.
This change in itself does not result in a large speedup, rather it
refactors the code in preparation for the introduction of a faster
find_first_zero() for bitarray based bitmaps.
Signed-off-by: Sami Liedes <sami.liedes@iki.fi> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Sami Liedes [Thu, 22 Mar 2012 23:46:56 +0000 (19:46 -0400)]
resize2fs: use EXT2_FLAG_64BITS
By passing EXT2_FLAG_64BITS to ext2fs_open2() we can avoid some
unnecessary redirection in critical paths. While resize2fs does not
currently otherwise support so big filesystems that this would matter,
passing this flag is entirely harmless and only tells libext2fs that
the caller has been recompiled against current headers.
With this change the CPU time needed to shrink a 100G filesystem drops
by 20%.
Signed-off-by: Sami Liedes <sami.liedes@iki.fi> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Sami Liedes [Thu, 22 Mar 2012 23:42:38 +0000 (19:42 -0400)]
libext2fs: move a modulo operation out of a hot loop.
Filesystem shrinking in particular is a heavy user of this loop in
ext2fs_new_inode(). This change makes resize2fs use 24% less CPU time
for shrinking a 100G filesystem.
Signed-off-by: Sami Liedes <sami.liedes@iki.fi> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Lukas Czerner [Thu, 22 Mar 2012 23:42:11 +0000 (18:42 -0500)]
e2fsck: remove EXT4_EOFBLOCKS_FL flag handling
We've decided to remove EOFBLOCKS_FL from the ext4 file system entirely,
because it is not actually very useful and it is causing more problems
than it solves. We're going to remove it from e2fsprogs first and then
after the new e2fsprogs version is common enough we can remove the
kernel part as well.
This commit changes e2fsck to not check for EOFBLOCKS_FL. Instead we
simply search for initialized extents past the i_size as this should not
happen. Uninitialized extents can be past the i_size as we can do
fallocate with KEEP_SIZE flag.
Also remove the EXT4_EOFBLOCKS_FL from lib/ext2fs/ext2_fs.h since it is
no longer needed.
Theodore Ts'o [Thu, 22 Mar 2012 20:00:49 +0000 (16:00 -0400)]
mke2fs: add option to use direct I/O (mke2fs -D)
This can be useful when using mke2fs on loaded servers, since
otherwise mke2fs can dirty a huge amount of memory very quickly,
leading to other applications not being happy at all.
Theodore Ts'o [Thu, 22 Mar 2012 03:40:24 +0000 (23:40 -0400)]
e2fsck: report ext2fs_open2() and ext2fs_check_desc() errors
Print the actual errors returned by ext2fs_open2() and
ext2fs_check_desc() before we fall back to the backup block group
descriptors so that it's easier to see if there is some obscure
failure that is causing e2fsck to think that it should use the backup
block group descriptors.
Theodore Ts'o [Thu, 22 Mar 2012 03:18:49 +0000 (23:18 -0400)]
e2fsck: avoid unnecessary close/reopen for non-MMP filesystems
For file systems that do not use MMP, there's no reason to close the
file system and then re-open the file system a second time, since
EXT2_FLAG_SKIP_MMP has no meaning for non-MMP file systems anyway.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu> Cc: Andreas Dilger <adilger@whamcloud.com>
Andreas Dilger [Tue, 20 Mar 2012 03:54:03 +0000 (23:54 -0400)]
tests: add field sizes to inode/super struct tests
In addition to validating the ordering of fields within the inode
and superblock structures, also validate the field sizes. Otherwise
it is possible to incorrectly change the size of one of these fields
without getting any kind of error from these tests. Failures would
only show up later in the test image checks if the field that is
changed is before another in-use field.
Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Andreas Dilger [Mon, 19 Mar 2012 03:48:45 +0000 (23:48 -0400)]
tests: have "make testnew" target use local mke2fs
The "make testnew" target in the tests/ subdirectory (used to
help create new test cases) should use the locally-built mke2fs
binary instead of the installed system binary, to avoid cases
where the system binary does not support some new functionality
being tested.
Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Sun, 18 Mar 2012 03:21:00 +0000 (23:21 -0400)]
e2fsck: add logging capability
Add the ability to log messages about a file system to a specified
directory, using a file name templace that can be specified in
/etc/e2fsck.conf. This allows us to suppress the output of overly
verbose e2fsck outputs while still allowing the full logging output to
go to an appropriate file.
Theodore Ts'o [Wed, 14 Mar 2012 21:44:54 +0000 (17:44 -0400)]
e2fsck: add ability to limit the # of problems of a particular type
This throttles the output of a particular problem type, to avoid a
bottleneck caused by (for example) printing a large number of
characters over a rate-limited a serial console.
Theodore Ts'o [Sun, 11 Mar 2012 20:19:10 +0000 (16:19 -0400)]
e2fsck: check for zero length extent
If an extent has e_len set to zero, the kernel will oops with a
BUG_ON. Unfortunately, e2fsck wasn't catching this case. The kernel
needs to be fixed to notice this case and call ext4_error() instead of
failing an assertion check, but e2fsck should catch this case and
repair it (by deleting the errant extent).
Lukas Czerner [Sun, 11 Mar 2012 19:36:45 +0000 (15:36 -0400)]
e2fsck: do not discard itable if discard doen't zero data
We do not want to discard inode table if the underlying device does not
return zeros when reading non-provisioned blocks. The reason is that if
the inode table is not zeroed yet, then discard would not help us since
we would have to zero it anyway. In the case that inode table was
already zeroed, then the discard would cause subsequent reads to contain
non-deterministic data so we would not be able to assume that the inode
table was zeroed and we would need to zero it again, which does not
really make sense.
This commit adds check to prevent inode table from being discarded if
the discard does not zero data.
Lukas Czerner [Sun, 11 Mar 2012 19:35:06 +0000 (15:35 -0400)]
e2fsck: do not discard when in read only mode
When argument '-n' was specified and should run in read-only mode, we
should not attempt to discard anything. In order to do that we have to
check for E2F_OPT_NO flag and clear E2F_OPT_DISCARD flag if E2F_OPT_NO
is set.
This commit fixes the problem when we would mark inode tables as zeroed
(EXT2_BG_INODE_ZEROED) even when e2fsck is running in read-only mode. We
also move the check for E2F_OPT_NO so we can clear E2F_OPT_DISCARD as
early as possible.
Theodore Ts'o [Sun, 11 Mar 2012 18:15:19 +0000 (14:15 -0400)]
e2fsck: optimize CPU usage in check_{block,inode}_bitmaps()
The boolean expression (!skip_group || csum_flag) is always true,
since if csum_flag is FALSE, skip_group must also be FALSE. Hence, we
can just remove the expression from the conditional altogether, thus
simplifying the code and making it easier to read/understand.
Also, in the case where the bit is set in the bitmap, there's no point
repeatedly setting first_free to be ext2fs_block_count(fs->super).
Lukas Czerner [Mon, 5 Mar 2012 07:49:34 +0000 (08:49 +0100)]
e2fsck: do not forget to discard last block group
Previously when running e2fsck with '-E discard' argument the end of
the last group has not been discarded. This patch fixes it so we
always discard the end of the last group if needed.
Lukas Czerner [Sun, 11 Mar 2012 16:55:45 +0000 (12:55 -0400)]
e2fsck: discard only unused parts of inode table
When calling e2fsck with '-E discard' option it might happen that
valid inodes are discarded accidentally. This is because we just
discard the part of inode table which lies past the free inode count.
This is terribly wrong (sorry!).
This patch fixes it so only the free parts of an inode table
is discarded, leaving used inodes intact. This was tested with highly
fragmented inode tables with block size 4k and 1k.
Mike Frysinger [Wed, 7 Mar 2012 01:21:36 +0000 (20:21 -0500)]
libext2fs: check for fallocate symbol before using it
If we have newer kernel headers which define FALLOC_FL_PUNCH_HOLE, but we
are on an older glibc which lacks fallocate, we end up trying to use the
func anyways. Check the ifdef that autoconf already set up for us.
Andreas Dilger [Fri, 24 Feb 2012 07:43:52 +0000 (00:43 -0700)]
libquota: don't include system quota.h header
Don't use the system <sys/quota.h> header in mkquota.c, since there
is a local e2fsprogs version of quota.h that is already included and
has the desired quota constants, and avoids symbol conflicts with the
system <sys/quota.h> on other platforms (in particular OSX).
Signed-off-by: Andreas Dilger <adilger@whamcloud.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Mike Frysinger [Mon, 27 Feb 2012 06:12:53 +0000 (01:12 -0500)]
libext2fs: include sys/time.h for gettimeofday()
Building on my glibc-2.15 system hits a warning:
gen_bitmap64.c: In function 'ext2fs_alloc_generic_bmap':
gen_bitmap64.c:127:2: warning: implicit declaration of function
'gettimeofday' [-Wimplicit-function-declaration]
Include sys/time.h if it's available for the prototype.
Signed-off-by: Mike Frysinger <vapier@gentoo.org> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Phillip Susi [Mon, 27 Feb 2012 05:56:49 +0000 (00:56 -0500)]
e2image: skip uninitialized bitmaps and inode tables
When the block bitmap is uninitialized, skip copying it.
When the inode bitmap is uninitialized, skip copying it,
as well as the inode table. When there are unused inodes
towards the end of the table, skip those blocks too.
Theodore Ts'o [Mon, 27 Feb 2012 05:51:39 +0000 (00:51 -0500)]
mke2fs: make sure bg 0's unused inode count field is zero'ed for mke2fs -S
There was a bug/typo in commit ba9e0afc5 which caused the first block
group (bg #0) to not have its unused inode count field to get set to
zero in the case of mke2fs -S. This caused inodes in the first block
group to not be recoverable via mke2fs -S. Oops.
Theodore Ts'o [Tue, 21 Feb 2012 01:27:58 +0000 (20:27 -0500)]
libext2fs: don't mark the superblock as dirty if the fs was opened r/o
If the file system is read/only opened with a backup superblock, and
the file system has uninit_bg enabled, the super block must not be
marked as dirty; otherwise, ext2fs_close() will call ext2fs_flush(),
which will fail, since the file descriptor for the block device was
opened read/only, and then the file descriptor won't actually be
closed.
This is normally not a problem since most of the time the program will
exit shortly after calling ext2fs_close(), and many programs don't
bother checking the error return from ext2fs_close(), especially if
the file system was opened read/only.
A big exception to this is e2fsck, since it opens and close the file
systems during its startup, and to make matters worse, registers an
error handler which will noisly complain about the failed writes
caused by ext2fs_flush().
Fix this by not marking the superblock as dirty if the file system was
opened read/only. The changes to the block group descriptors to clear
the uninit bits will still happen, so that e2fsck -n will properly
scan the whole file system. However, those changes will get dropped
when the file system handle is closed.
Theodore Ts'o [Mon, 20 Feb 2012 23:18:13 +0000 (18:18 -0500)]
debian: fix dpkg-buildflags so it works when the shell is dash
The buildd's use dash as the shell, so we can't use >& in the rules
file. This was causing the buildd-built e2fsprogs binary packages to
be built without the hardening flags.