Palmer Dabbelt [Fri, 29 Dec 2017 18:19:51 +0000 (10:19 -0800)]
misc: rename copy_file_range to copy_file_chunk
As of 2.27, glibc will have a copy_file_range library call to wrap the
new copy_file_range system call. This conflicts with the function in
misc/create_inode.c, which this patch renames _copy_file_range.
Theodore Ts'o [Mon, 1 Jan 2018 23:59:16 +0000 (18:59 -0500)]
libext2fs: when writing bitmaps mark the fs as dirty if necessary
If any checksum fields are updated in the block group descriptors, we
need to set the EXT2_FLAG_DIRTY flag so that the block group
descriptors are written to disk.
Tyson Nottingham [Thu, 14 Dec 2017 07:13:12 +0000 (23:13 -0800)]
Add missing information to various man pages and usage texts
badblocks.c: Add -B and -X to usage text.
chattr.1.in: Add P to list of settable attributes.
e2image.8.in, e2image.c: Remove superfluous mention of -r option.
e2undo.8.in, e2undo.c: Add description for -z to man page and add -o and
-z to synopsis and usage text.
e4crypt.8.in: Add -p pad option to relevant command descriptions.
lsattr.1.in: Fix typo.
mke2fs.8.in: Add -C to synopsis. Add MKE2FS_DEVICE_PHYS_SECTSIZE
description and update MKE2FS_DEVICE_SECTSIZE description.
mke2fs.conf.5.in: Add devices stanza summary. Remove base_features and
default_features descriptions from defaults section, as they are
already described in fs_types section. Move enable_periodic_fsck and
force_undo to fs_types section, as they can be set per filesystem
type. Add missing lazy_journal_init and hugefiles_dir descriptions.
tune2fs.8.in: Remove unsupported -p option and deprecated -s option.
Add 64bit, ea_inode, large_dir, and metadata_csum_seed descriptions.
tune2fs.c: Remove unsupported -p option from usage text.
Add mmp_update_interval to extended option usage text.
Tyson Nottingham [Thu, 14 Dec 2017 07:13:11 +0000 (23:13 -0800)]
Add missing information to debugfs man pages and usage texts
Add descriptions for dump_unused, get_quota, list_quota,
set_current_time, show_debugfs_params, and supported_features
commands to debugfs man page. Add various missing command
arguments and correct a few typos in man page and usage texts.
Tyson Nottingham [Thu, 14 Dec 2017 07:13:09 +0000 (23:13 -0800)]
Add missing features and mount options to ext4 man page
Add descriptions for the metadata_csum and metadata_csum_seed
features, as well as nombcache and prjquota mount options. Also,
update a couple of descriptions.
Add swabbing for these missing fields. Reorder the swabbing
on other fields so that they are in struct order, and add
comments for fields that do not need swabbing, so it is easier
to see that all fields are being handled.
Since this seems to be a fairly common occurrence, add compile
time checks to ext2fs_swap_super(), ext2fs_swap_inode_large(),
and ext2fs_swap_group_desc2() via a new EXT2FS_BUILD_BUG_ON()
macro to cause a compile warning if fields are used from these
structs without a corresponding change to the swab functions.
Qian Yingjin [Wed, 13 Dec 2017 15:59:39 +0000 (10:59 -0500)]
debugfs: fix ncheck program output
Fix the bug that print the inode number not the program name
when reporting bad inode name during checking in ncheck.
Minor bug fix: it should reduce iw->inodes_left when found
matched inode number.
Dmitry V. Levin [Wed, 13 Dec 2017 04:05:45 +0000 (23:05 -0500)]
Fix -Wunused-variable compilation warnings
Fix the following compilation warnings:
sha256.c: In function 'main':
sha256.c:238:20: warning: unused variable 'md' [-Wunused-variable]
sha512.c: In function 'main':
sha512.c:286:20: warning: unused variable 'md' [-Wunused-variable]
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Dmitry V. Levin [Thu, 18 May 2017 17:46:49 +0000 (17:46 +0000)]
Fix compilation warnings about functions declared in sys/xattr.h
attr commit v2.4.47-34-g7921157 removes <attr/xattr.h>
and syscall wrappers with the following wording:
"The xattr syscalls are provided by glibc since ages [...]
This removes the need for the <attr/xattr.h> header;
use <sys/xattr.h> instead."
Check for <sys/xattr.h> and include it to fix the following compilation
warnings:
create_inode.c: In function 'set_inode_xattr':
create_inode.c:136:9: warning: implicit declaration of function 'llistxattr' [-Wimplicit-function-declaration]
create_inode.c:172:16: warning: implicit declaration of function 'lgetxattr' [-Wimplicit-function-declaration]
Signed-off-by: Dmitry V. Levin <ldv@altlinux.org> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Wed, 13 Dec 2017 02:46:36 +0000 (21:46 -0500)]
e2fsck: fix potential Floating Point Exception in show_stats()
If the free inodes count in the superblock is equal to the inodes
count in the superblock (which is not possible with a valid file
system and will be fixed by e2fsck unless it is prevented by, for
example, e2fsck -n), it is possible for e2fsck to crash due to a
divide by zero error.
Kazuya Mio [Tue, 5 Dec 2017 00:15:38 +0000 (00:15 +0000)]
e2freefrag: print 64bit free blocks count
e2freefrag prints incorrect free blocks count when the number of free blocks
is larger than 2^32.
To fix this problem, e2freefrag uses ext2fs_free_blocks_count(),
not fs->super->s_free_blocks_count.
Theodore Ts'o [Mon, 16 Oct 2017 04:38:00 +0000 (00:38 -0400)]
debian: fix FTBFS regression caused by debhelper/10.9
The problem is caused by obsolete packages referenced in the rules
file. The latest version of debhelper is no longer forgiving about
non-existing packages being referenced in debhelper commands.
Theodore Ts'o [Mon, 16 Oct 2017 04:28:45 +0000 (00:28 -0400)]
libext2fs: fix potential memory access overrun in ext2fs_inode_csum()
If the superblock has a revision level of 0, then s_inode_size is
undefined, and the actual inode size is 128 bytes. This is handled by
the EXT2_INODE_SIZE() helper macro. If s_inode_size is maliciously
set to a large value, and the s_rev_level is 0, then this could result
in an illegal memory pointer dereference.
Addresses-Debian-Bug: #878104 Reported-by: Jakub Wilk <jwilk@jwilk.net> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Andreas Dilger [Sun, 15 Oct 2017 20:23:24 +0000 (16:23 -0400)]
tests: shorten test descriptions to fit in 80 columns
While not critical to functionality, it is easier to see the pass/fail
status of each test if they are not wrapped onto the next line.
Shorten test descriptions so the name+description fit in 80 columns.
Signed-off-by: Andreas Dilger <adilger@dilger.ca> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Sun, 15 Oct 2017 20:19:04 +0000 (16:19 -0400)]
tests: fix d_inline_dump and f_mke2fs_baddisk from leaving behind temp files
These two tests could leave an temp file behind if the test is
skipped. In general test script files should avoid using 'exit',
since this bypasses the test cleanup scripting.
Andreas Dilger [Tue, 10 Oct 2017 22:39:52 +0000 (16:39 -0600)]
tests: don't leave temporary files at end of test
Don't leave temp files around after test has completed successfully.
Some test scripts were creating their own temporary files, and in
some cases test_one was skipping a test completely (e.g. slow) after
the temp file had been created.
Signed-off-by: Andreas Dilger <adilger@dilger.ca> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Sun, 15 Oct 2017 04:22:20 +0000 (00:22 -0400)]
misc: clean up error handling for ext2fs_run_ext3_journal()
The ext2fs_run_ext3_journal() function is in debugfs/journal.c, and in
some error conditions cases may close the passed-in file system handle.
Clean up the both the function so that it reliably clears the file
system handle if it has been freed, and its callers so that they do
not crash by dereferencing a null pointer if it has been freed.
Reported-by: Erkki Ruohtula <eru@netti.fi> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Lukas Czerner [Sat, 14 Oct 2017 14:42:30 +0000 (10:42 -0400)]
libext2fs: skip start_blk adjustment when stride and flex_bg is set
Currently some stride optimization is done in
ext2fs_allocate_group_table() by adjusting start_blk block where we
start allocating block, or inode bitmaps.
However in flex_bg case this is currently useless since the values are
going to be overridden anyway. Moreover in flex_bg case the group might
already be full and the stride optimization will fail. As a result file
system resize might fail needlessly in some situations.
It can be shown by this example:
mke2fs -b 1024 -i 1024 -E stride=8192 -t ext4 /dev/loop0 1024000
resize2fs /dev/loop0 102400000
resize2fs 1.43.5 (04-Aug-2017)
Resizing the filesystem on /dev/loop0 to 102400000 (1k) blocks.
./resize/resize2fs: Could not allocate block in ext2 filesystem while trying to resize /dev/loop0
Please run 'e2fsck -fy /dev/loop0' to fix the filesystem
after the aborted resize operation.
Fix this by not doing the stride adjustment in case of flex_bg.
Lukas Czerner [Sat, 14 Oct 2017 14:33:10 +0000 (10:33 -0400)]
libext2fs: prevent allocating inode table from already used blocks
Currently it's possible for ext2fs_allocate_group_table() to place inode
tables to blocks that are already occupied by different inode table.
This can be reproduced by resize2fs on the file system where we need to
move more than one inode table to a different location due to increase
in group descriptor blocks, inode and block bitmaps.
Best way I can reproduce this is to create big enough file system with
huge amount of inodes and without resize_inode
e2fsck -fn /dev/loop0 | less
e2fsck 1.43.5 (04-Aug-2017)
ext2fs_check_desc: Corrupt group descriptor: bad block for inode table
e2fsck: Group descriptors look bad... trying backup blocks...
e2fsck: The journal superblock is corrupt while checking journal for /dev/loop0
e2fsck: Cannot proceed with file system check
Superblock has an invalid journal (inode 8).
Clear? no
/dev/loop0: ********** WARNING: Filesystem still has errors **********
None of the settings are strictly necessary and it can be reproducer in
various ways. This is just an example of one easy way to reproduce this.
This bug was introduced with commit fccdbac39454 ("libext2fs: optimize
ext2fs_allocate_group_table()") and is caused by the fact that wrong
bitmap is used to mark the blocks as used.
Fix this by using ext2fs_mark_block_bitmap_range2() in both (flex_bg and
non flex_bg) cases and handle flex_bg case manually instead of relying
on ext2fs_block_alloc_stats_range() because there is no way in that
function to use different bitmap than fs->block_map.
libext2fs: remove useless test and assignment in strtohashbuf()
On transformation of str to hash, computed value is initialised before
first byte modulo 4. But it is already initialised before entering loop
and after processing last byte modulo 4. So the corresponding test and
initialisation could be removed.
debian: clean up conditional fuse2fs pacakge build rules
There's no point conditionalizing fuse2fs in the control file, since
we control whether or not fuse2fs is built in the rules file, and the
control file is going to be the same when built on all of the debian
build servers.
Also key off of DEB_HOST_ARCH_OS to determine whether or not we are
building for the Hurd, and define it so the right thing happens if
./debian/rules is run by hand on a Hurd system.
Andreas Dilger [Wed, 30 Aug 2017 06:05:25 +0000 (02:05 -0400)]
tests: report if a test is taking a long time
Print out a message if a test takes longer than 60s, with a
reminder to potentially add is_slow_test to that test, so
that it is easier to find which tests are taking a long time,
especially if running with "make -j check" or similar.
Add an exclusion for r_expand_full on MacOS since HFS+ does
not support sparse devices and doesn't like large test images.
Signed-off-by: Andreas Dilger <adilger@dilger.ca> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Andreas Dilger [Wed, 30 Aug 2017 05:56:08 +0000 (01:56 -0400)]
tune2fs: quiet llvm build warning
Quiet a relatively harmless build warning:
tune2fs.c:928:18: warning: '&&' within '||' [-Wlogical-op-parentheses]
if (pass == 1 && (inode->i_flags & EXT4_EA_INODE_FL) ||
~~~~~~~~~~^~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~ ~~
tune2fs.c:928:18: place parentheses around '&&' to silence warning
Signed-off-by: Andreas Dilger <adilger@dilger.ca> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Matthias Andree [Tue, 29 Aug 2017 05:02:36 +0000 (01:02 -0400)]
Search for GNU-compatible dd for self-tests.
This checks for a dd that supports iflag=fullblock oflag=append,
and looks at gdd and dd for now, and warns of failing self-tests
if neither supports these two flags.
Theodore Ts'o [Sat, 26 Aug 2017 17:42:30 +0000 (13:42 -0400)]
Silence valgrind warnings
Valgrind doesn't understand that the kernel will be initializing the
struct termios and struct loop_info64 structures. Since they occur in
functions which are not in the hot path, preinitialize to zero to
prevent valgrind from producing a huge number of false positives.
Jaco Kroon [Wed, 23 Aug 2017 18:21:43 +0000 (14:21 -0400)]
e2fsck: add optimization for heavily hard-linked file systems
In the case of file system with large number of hard links, e2fsck can
take a large amount of time in pass 2 due to binary search lookup of
inode numbers. This implements a memory trade-off (storing 2 bytes
in-memory for each inode to store inode counts).
For a 40TB filesystem with 2.8bn inodes this map alone requires 5.7GB
of RAM. For this reason, we don't enable this optimization by
default. It can be enabled using either an extended option to e2fsck
or via a seting in e2fsck.conf.
Even when the fullmap optimization is enabled, we don't use this for
the icount structure in pass 1. This is because the gain CPU gain is
nearly nil for that pass and the sacrificed memory does not justify
the increase in RAM.
(It could be that during pass 1, if more than 17% if possible inodes
has link_count>1 (466m inodes in the 40TB with 2.8bn possible inodes
case) then it becomes more memory efficient to use the full map
implementation in terms of memory. However, this is extremely
unlikely given that most file systems are heavily over-provisioned in
terms of the number of inodes in the system.)
Jaco Kroon [Wed, 23 Aug 2017 17:54:25 +0000 (13:54 -0400)]
e2fsck: optimize out the use region_t in scan_extent_node()
Since extents have a guarantee of being monotonically increasing we
merely need to check that block n+1 starts after block n. This is a
simple enough check and we can perform this by calculating the next
expected logical block instead of using the region usage tracking data
abstraction.
Theodore Ts'o [Wed, 23 Aug 2017 14:57:25 +0000 (10:57 -0400)]
tune2fs: explain why an fsck is needed
Currently tune2fs just says without any explanation, "run fsck -f".
Add a short explanation that a freshly checked file system is required
to reduce user confusion. (We could add even more details, but
hopefully this is enough.)
Theodore Ts'o [Wed, 23 Aug 2017 14:30:09 +0000 (10:30 -0400)]
mke2fs: automatically use 256 byte inodes if project feature enabled
If the inode size is not implicitly requested on the command line, and
it is too small to support the project feature, automatically promote
the inode size to be 256 bytes so that the project feature will work.
Note the previous test to check for a too-small inode size didn't work
because it checked before inode size was set in fs_param. Hence, it
was possible to create file systems with a 128 byte inode and the
project feature enabled.
Theodore Ts'o [Tue, 22 Aug 2017 16:15:26 +0000 (12:15 -0400)]
debian: remove support for pre-multiarch versions of Debian
All versions of Debian after Wheezy support Multiarch, so we can
simply the Debian control.in and rules file by removing support for
older versions of Debian without Multiarch support.
Theodore Ts'o [Tue, 22 Aug 2017 15:23:21 +0000 (11:23 -0400)]
libext2fs: avoid potential out-of-bounds write if pread/pread64 fails
In unix_io.c's raw_read_block(), if the initial attempt to call
pread/pread64 fails because the offset is insane, the variable
"actual" is left at -1, and then when lseek fails, the cleanup
function will try to clear (as an out-of-bounds write) a single byte
before the buffer. Fix this.
Addresses-Debian-Bug: #871539
Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reported-by: Jakub Wilk <jwilk@jwilk.net>
Theodore Ts'o [Tue, 22 Aug 2017 14:37:10 +0000 (10:37 -0400)]
debian: remove support libuuid/libblkid packages
Remove support for util-linux prior to 2.16, when e2fsprogs provided
its own copy of libuuid and libblkid. This is only needed for Debian
distributions prior to Wheezy, which is no longer supported.
Theodore Ts'o [Tue, 22 Aug 2017 04:54:15 +0000 (00:54 -0400)]
libsupport: don't try accessing the project quota for 128 byte inodes
If the file system has 128 byte inode, it's not possible to access its
project quota id, since the inode is too small. So prevent a
potential out-of-bounds read in get_qid().
The problem was found by valgrind and American Fuzzy Lop.
Theodore Ts'o [Tue, 22 Aug 2017 01:20:38 +0000 (21:20 -0400)]
e2fsck: in ask_yn() fall back to English yes/no characters
In the case of missing translations, if the translation for y/n is
missing due to fuzzy transactions, such that user is told to use
<y/n>, those characters will work correctly.
Theodore Ts'o [Mon, 14 Aug 2017 23:52:39 +0000 (19:52 -0400)]
e2fsck: add optimization for large, fragmented sparse files
The code which checks for overlapping logical blocks in an extent tree
is O(h*e) in time, where h is the number of holes in the file, and e
is the number of extents in the file. So a file with a large number
of holes can take e2fsck a long time process. Optimize this taking
advantage of the fact the vast majority of the time, region_allocate()
is called with increasing logical block numbers, so we are almost
always append onto the end of the region list.