Theodore Ts'o [Wed, 5 Feb 2014 02:45:51 +0000 (21:45 -0500)]
debian: fix dpkg-buildpackage for Debian Squeeze
Commit becb01ce84d breaks building e2fsprogs with dpkg 1.15.8 which is
used in Debian 6.0 (Squeeze), since it doesn't support package
specifications qualified with an architecture (i.e., "dpkg-query -W
libblkid1:amd64").
Debian only needs to use its own version of libblkid and libuuid for
versions of Debian 5.0 (Lenny) or before. So default to using
util-linux-ng, instead of trying to test the version number of
libblkid1.
Lenny was released in February, 2009, and the current stable Debian
release is 7.x, so it is two stable releases back as of February 2014.
In the unlikely case someone needs to build a modern version of
e2fsprogs on a version of Debian which is five years old or older, can
create the file Debian/rules.custom with the line:
Theodore Ts'o [Thu, 30 Jan 2014 23:48:23 +0000 (18:48 -0500)]
Fix up the Makefiles dependencies in lib/ext2fs and lib/quota
Also use angle brackets for the #include of dirpaths.h to avoid the
need to manually massage the Makefile.in for the util directory. This
is needed because we have to create a fake dirpaths.h file in the util
directory. The fake dirpaths.h file is rquired to break the circular
dependency caused by util/subst creating dirpaths.h, while
util/subst.c is including config.h, which includes dirpaths.h.
Theodore Ts'o [Thu, 30 Jan 2014 23:02:37 +0000 (18:02 -0500)]
blkid: suppress Coverity warning
The getopt() function will never let optarg be NULL (at least without
using the GNU double-colon extension, which we don't use because it's
not portable), so don't bother checking for that case. It's harmless,
but it triggers a Coverity warning elsewhere, since it thinks optarg
could in fact be NULL.
Theodore Ts'o [Thu, 30 Jan 2014 22:10:46 +0000 (17:10 -0500)]
quota: fix uninitiaized memory reference in mke2fs with quota enabled
Initialize the on-disk structure before we fill it in, to avoid the
following valgrind warning:
Conditional jump or move depends on uninitialised value(s)
at 0x4323A8: qtree_entry_unused (quotaio_tree.c:40)
by 0x431218: v2r1_mem2diskdqblk (quotaio_v2.c:85)
by 0x432409: qtree_write_dquot (quotaio_tree.c:336)
by 0x431136: v2_commit_dquot (quotaio_v2.c:264)
by 0x42FB63: quota_write_inode (mkquota.c:126)
by 0x408BE6: create_quota_inodes (mke2fs.c:2466)
by 0x409A2D: main (mke2fs.c:2850)
Theodore Ts'o [Tue, 28 Jan 2014 19:44:23 +0000 (14:44 -0500)]
mke2fs: allow metadata blocks to be at the beginning of the file system
Add the extended options packed_meta_blocks and journal_location_front
which causes mke2fs to place the metadata blocks at the beginning of
the file system.
Theodore Ts'o [Sun, 12 Jan 2014 03:11:42 +0000 (22:11 -0500)]
Add support for new compat feature "sparse_super2"
In practice, it is **extremely** rare for users to try to use more
than the first backup superblock located at the beginning of block
group #1. (i.e., at block number 32768 for file systems with a 4k
block size). This new compat feature restricts the backup superblock
to block group #1 and the last block group in the file system.
Aside from reducing the overhead of the file system by a small number
of blocks, by eliminating the rest of the backup superblocks, it
allows us to have a much more flexible metadata layout. For example,
we can force all of the allocation bitmaps and inode table blocks to
the beginning of the disk, which allows most of the disk to be
exclusively used for contiguous data blocks.
This simplifies taking advantage of certain HDD specific features,
such as Shingled Magnetic Recording (aka Shingled Drives), and the
TCG's OPAL Storage Specification where having a simple mapping between
LBA block ranges and the data blocks used by the file system can make
life much simpler.
Theodore Ts'o [Mon, 20 Jan 2014 00:35:33 +0000 (19:35 -0500)]
libext2fs: optimize ext2fs_new_block2()
If there are hundreds of thousands of blocks which are in use before
the first free block, it is much, MUCH faster to use
ext2fs_find_first_zero_block_bitmap2() instead of searching the
allocation bitmap bit by bit.
Theodore Ts'o [Sun, 19 Jan 2014 21:47:21 +0000 (16:47 -0500)]
libext2fs: optimize ext2fs_allocate_group_table()
By using ext2fs_mark_block_bitmap_range2 and/or
ext2fs_block_alloc_stats_range(), we can significantly speed up the
time needed by mke2fs to allocate the inode table.
For example, the CPU time needed to run the command "mke2fs -t ext4
/tmp/foo.img 32T" (where tmpfs was mounted on /tmp) was decreased from
21.7 CPU seconds down to under 1.7 seconds.
Theodore Ts'o [Sun, 19 Jan 2014 21:35:50 +0000 (16:35 -0500)]
libext2fs: add ext2fs_block_alloc_stats_range()
This function is more efficient than using ext2fs_block_alloc_stats2()
for each block in a range. The efficiencies come from being able to
set a block range in the block bitmap at once, and from being update
the block group descriptors once per block group. Especially now that
we are checksuming the block group descriptors, and we are using red
black trees for the allocation bitmaps, these changes can make a huge
difference in the CPU time used by mke2fs when creating very large
file systems.
Theodore Ts'o [Sun, 19 Jan 2014 06:24:30 +0000 (01:24 -0500)]
libext2fs: further clean up and rename check_block_uninit
Commit 8e44eb64bb (libext2fs: mark group data blocks when loading
block bitmap) simplified check_block_uninit since we are now
initializing the bitmap when it is loaded from disk. It left some
variables which were being set but never used, however. In addition,
since we only need check_block_uninit() to clear the block bitmap's
uninit flag, rename it to clear_block_uninit(), and only call it once
we have found a free block in ext2fs_new_blocks2().
This cleans up the code some and optimizes things if we need to search
multiple block groups trying to find a free block.
Signed-off-by: "Theodore Ts'o" <tytso@mit.edu> Cc: Darrick J. Wong <darrick.wong@oracle.com>
Theodore Ts'o [Mon, 13 Jan 2014 01:18:55 +0000 (20:18 -0500)]
libext2fs: build tst_bitmaps with rep invariants checking enabled
When building tst_bitmaps, enable #define DEBUG_RB, so we are
always testing the sanity of the in-memory representation of the
bitmap when using red-black trees as part of a "make check" run.
Theodore Ts'o [Thu, 16 Jan 2014 04:29:21 +0000 (23:29 -0500)]
libext2fs: fix off-by-one bug in ext2fs_extent_insert()
When inserting the first extent into an empty inode, the
ext2fs_extent_insert() leaves path->left set to 1 instead of 0. Since
path->curr is pointing at the last (only) extent in the file,
path->left should be 0.
This is mostly harmless, and gets corrected fairly quickly if the
calling applicaton jumps to a different part of the extent tree ---
for example, by calling ext2fs_extent_goto(), or calling
ext2fs_extent_get with the flags argument set to EXT2_EXTENT_ROOT.
Which is why we hadn't noticed this problem until now.
However, if you insert four extents using ext2fs_extent_insert, the
fourth insert will end up copying too many bytes in the i_block[]
array, since path->left is one larger than it should be. This results
in the inode fields i_generation, i_file_acl, and i_size_high getting
zeroed out.
Add functions which try to find the first set block or inode in a
bitmap. This is useful when trying to allocate a range of blocks
efficiently.
Like the find_first_zero family of functions, provide a generic O(N)
search function which will be used if there is no optimized version
provided by the red-black tree or bitarray functions.
Also, expand the test cases for ext2fs_find_first_zero_*() functions.
Darrick J. Wong [Sat, 11 Jan 2014 19:15:52 +0000 (14:15 -0500)]
tests: adjust test output to reflect block_uninit calculated block bitmaps
Now that libext2fs marks group metadata in the fs block bitmap, adjust
the expected test output to reflect expanded use of block_uninit and
the fact debugfs no longer prints block bitmap data that fails to
account for group data blocks.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Darrick J. Wong [Sat, 11 Jan 2014 19:15:51 +0000 (14:15 -0500)]
libext2fs: no need to clear BLOCK_UNINIT during ext2fs_reserve_super_and_bgd
Since the beginning of the uninit_bg feature, the kernel[1] and
e2fsck[2] have always been careful to detect the presence of the
BLOCK_UNINIT flag, and compute a block bitmap with any group metadata
blocks marked in that bitmap. With that in mind, I think it's safe to
say that this is a design feature of uninit_bg.
Now that we've trained libext2fs to have this same behavior whenever
it's loading a block bitmap, we no longer need to unset BLOCK_UNINIT
for a group that contains only its own group metadata -- kernel,
e2fsck, and e2fsprogs will handle this correctly.
Darrick J. Wong [Sat, 11 Jan 2014 19:05:02 +0000 (14:05 -0500)]
e2fsck: remove uninit block bitmap calculation
Since libext2fs now detects a BLOCK_UNINIT group and calculates the
group's block bitmap, we no longer need to emulate this behavior in
e2fsck. We can simply compare the found block map against the
filesystem's, and proceed from there.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Darrick J. Wong [Sat, 11 Jan 2014 19:04:48 +0000 (14:04 -0500)]
libext2fs: mark group data blocks when loading block bitmap
The kernel[1] and e2fsck[2] both react to a BLOCK_UNINIT group by
calculating the block bitmap that's needed to show all the group
blocks for that group (if any) and using that. However, when reading
bitmaps from disk, libext2fs simply imports a block of zeroes into the
bitmap, without bothering to check for group blocks. This erroneous
behavior results in the filesystem having a block bitmap that does not
accurately reflect disk contents, and worse yet makes it seem as
though superblocks, group descriptors, bitmaps, and inode tables are
"free" space on disk.
So, fix the block bitmap loading routines to calculate the correct
block bitmap for all groups and load it into the main fs block bitmap.
This also fixes bogus debugfs output such as:
Group 1: (Blocks 8193-16384) [INODE_UNINIT, BLOCK_UNINIT]
Checksum 0x1310, unused inodes 512
Backup superblock at 8193, Group descriptors at 8194-8217
Reserved GDT blocks at 8218-8473
Block bitmap at 283 (bg #0 + 282), Inode bitmap at 299 (bg #0 + 298)
Inode table at 442-569 (bg #0 + 441)
7911 free blocks, 512 free inodes, 0 directories, 512 unused inodes
Free blocks: 8193-16384
Free inodes: 513-1024
Notice how the "free blocks" range includes the backup sb & GDT area
and doesn't match the free block count.
Worse yet, debugfs' testb command will report those group descriptor
blocks as not being in use unless the user also instructs debugfs to
find a free block first. That is a rather surprising result:
debugfs: testb 8194
Block 8194 not in use
debugfs: ffb 1 16380
Free blocks found: 16380
debugfs: testb 8194
Block 8194 marked in use
Also, remove the part of check_block_uninit() that "fixes" the bitmap
since we're doing that at bitmap load time now.
Darrick J. Wong [Sat, 11 Jan 2014 18:58:15 +0000 (13:58 -0500)]
libext2fs: don't always read backup group descriptors on a 1k-block meta_bg fs
On a filesystem with 1K blocks and meta_bg enabled, opening a
filesystem with automatic superblock detection tries to compensate for
the fact that the superblock lives in block 1. However, the method by
which this is done is later misinterpreted to mean "read the backup
group descriptors", which is not what we want in this case.
Therefore, in ext2fs_open3() separate the 'group zero' adjustment into
its own variable so that we don't get fed backup group descriptors
when we try to load meta_bg group descriptors.
Furthermore, enhance ext2fs_descriptor_block_loc2() to perform its own
group zero correction. The other caller of this function neglects to
do any group-zero correction of their own, so this fixes them too.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Darrick J. Wong [Sat, 11 Jan 2014 18:54:57 +0000 (13:54 -0500)]
libext2fs: detect correct superblock adjustments when loading backup groups
If ext2fs_descriptor_block_loc2() is called with a meta_bg filesystem
and group_block is not the normal value, the function will return the
location of the backup group descriptor block in the next block group.
Unfortunately, it fails to account for the possibility that the backup
group contains a backup superblock but the regular superblock does
not. This is the case with block groups 48-49 on a meta_bg fs with 1k
blocks; in this case, libext2fs will fail to open the filesystem.
Therefore, teach the function to adjust for superblocks in the backup
group, if necessary.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Thu, 9 Jan 2014 20:59:29 +0000 (15:59 -0500)]
util: fix make dependencies for subst
Now that subst is using config.h, we need to fix its dependencies so
that if config.h is missing (i.e., after a "make clean" in the tree)
that it is rebuilt by the Makefile rule.
Theodore Ts'o [Thu, 9 Jan 2014 03:25:04 +0000 (22:25 -0500)]
Fix up workarounds for dietlibc breakage
The dietlibc doesn't support the TZ environment variable, which is
required by the standard. Work around this so that we can run the
regression test suite when building with dietlibc. (This is useful
for finding problems.)
With this change, the only thing which doesn't work as far as dietlibc
is concerned is the posix_memalign test, and the MMP support tests
(because posix_memalign isn't provided by dietlibc, sigh.)
Theodore Ts'o [Sun, 5 Jan 2014 06:09:07 +0000 (01:09 -0500)]
debian: renable use of internal uuid and blkid libraries for e2fsck.static
Now that the diet libc support is working again, we need to use the
internal uuid and blkid library in order for e2fsck.static to be
compiled properly.
Theodore Ts'o [Sun, 5 Jan 2014 05:15:54 +0000 (00:15 -0500)]
Use consistent e2fsprogs-standard types
Some different types such as u_int16_t and __uint32_t have snuck into
e2fsprogs. These types are not guaranteed by any standard, and they
are not provided by dietlibc. Convert them to __u16, __u32,
etc. since these are guaranteed to be provided by e2fsprogs' build.
Theodore Ts'o [Sun, 5 Jan 2014 01:44:29 +0000 (20:44 -0500)]
configure: fix --with-diet-libc
Newer versions of autoconf pull in AC_PROG_GCC as part of
AC_CANONICAL_HOST. So we need check for WITH_DIET_LIBC earlier in
configure.in.
Also, e2fsprogs now needs functions which are found in diet libc's
compat library. So add support for autoconf's LIBS function, and
automatically set libs to include -lcompat.
Finally, disable compiling e4defrag by deault if --with-diet-libc is
specified because the program has too many glibc dependencies.
Baruch Siach [Thu, 2 Jan 2014 18:05:37 +0000 (13:05 -0500)]
e4defrag: choose the best available posix_fadvise variant
Use posix_fadvise64() when available. This allows 64bit offsets on
32bit systems.
[ Modified by tytso to try to use fadvise64() as well, and to remove
the attempt to call the syscall directly, since because and
complexities caused by required dummy arguments on some
architectures, it's not worth the hair. ]
Theodore Ts'o [Fri, 3 Jan 2014 14:39:33 +0000 (09:39 -0500)]
debugfs: clean up read_journal_block() to avoid Coverity complaint
The read_journal_block() function was needlessly complicated, which
made it harder to read/maintain, and it also tripped up Coverity.
Cleaning it up also avoided some signed/unsigned casts, and allows us
to avoid passing got back to the caller, since it wasn't needed.
Theodore Ts'o [Fri, 3 Jan 2014 05:26:43 +0000 (00:26 -0500)]
subst: clean up various coverity nits
Add appropriate error checking for all error returns, and only open
each file that we need to manipulate once, to avoid potential
time-of-check/time-of-use races. (Not that this is likely for this
program, but the result is much more clean.)
We also preserve the atime in the case where the file has not changed.
Theodore Ts'o [Wed, 1 Jan 2014 04:02:44 +0000 (23:02 -0500)]
e2fsck: improve the "superblock corrupt" message
Previously, this message used 8193 as the example alternate
superblock. But for most file systems, the backup superblock is
located at 32768 (since most file systems have a block size of 4k, and
not 1k).
Theodore Ts'o [Tue, 31 Dec 2013 03:41:48 +0000 (22:41 -0500)]
update config.h.in file to enable loopback mount detection
Commit fbabd5c44c23 added loopback mount detection. However, we
failed to update the config.h file, so the code wasn't actually
enabled. Fix this oversight.
Theodore Ts'o [Mon, 30 Dec 2013 22:35:58 +0000 (17:35 -0500)]
debian: rationalize configure options management
As part of the debian build, the e2fsprogs sources are built multiple
times --- for the udeb packages, for e2fsck.static, etc. Make sure
that when we disable the use of the built-in blkid and uuid libraries
(in favor of the ones shipped with util-linux), that we do so for all
of the builds.
Previously we were using the internal blkid library for e2fsck.static.
While this isn't completely broken, the internal blkid library hasn't
been maintained in quite some time, so it's better to use the
util-linux blkid library instead.
Eric Whitney [Mon, 30 Dec 2013 21:43:46 +0000 (16:43 -0500)]
debugfs: restore and tweak original error messaging
In response to reviewer comments, commit fe56188b07 included changes
that modified some of the code used to output error messages when
checking user-supplied block numbers. These changes converted calls
to parse_ulonglong() to calls to strtoblk(). Because strtoblk() calls
parse_ulonglong(), and both output error messages, two redundant and
relatively generic messages were output on each error.
Fix this by removing the error message output from strtoblk(), and
extending it to accept an optional error message argument that it
supplies in lieu of a default to parse_ulonglong(). Also, revert to
the more descriptive original error messages with mods per reviewer
comments, and fix an error message in do_replace_node().
Signed-off-by: Eric Whitney <enwlinux@gmail.com> Signed-off-by: "Theodore Ts'o" <tytso@mit.edu>
Theodore Ts'o [Mon, 30 Dec 2013 20:48:06 +0000 (15:48 -0500)]
debian: use dh_strip --dbg-package to populate the dbg packages
Use the --dbg-package option instead of manually moving the files into
the debug packages. This is simpler to maintain, and avoids a problem
where creating directories using mkdir -p can create world-writeable
/usr/lib directory dependind on the umask used by the Debian buildd.
projects / thirdparty / e2fsprogs.git / log
