Matthias Andree [Tue, 29 Aug 2017 05:02:36 +0000 (01:02 -0400)]
Search for GNU-compatible dd for self-tests.
This checks for a dd that supports iflag=fullblock oflag=append,
and looks at gdd and dd for now, and warns of failing self-tests
if neither supports these two flags.
Theodore Ts'o [Sat, 26 Aug 2017 17:42:30 +0000 (13:42 -0400)]
Silence valgrind warnings
Valgrind doesn't understand that the kernel will be initializing the
struct termios and struct loop_info64 structures. Since they occur in
functions which are not in the hot path, preinitialize to zero to
prevent valgrind from producing a huge number of false positives.
Jaco Kroon [Wed, 23 Aug 2017 18:21:43 +0000 (14:21 -0400)]
e2fsck: add optimization for heavily hard-linked file systems
In the case of file system with large number of hard links, e2fsck can
take a large amount of time in pass 2 due to binary search lookup of
inode numbers. This implements a memory trade-off (storing 2 bytes
in-memory for each inode to store inode counts).
For a 40TB filesystem with 2.8bn inodes this map alone requires 5.7GB
of RAM. For this reason, we don't enable this optimization by
default. It can be enabled using either an extended option to e2fsck
or via a seting in e2fsck.conf.
Even when the fullmap optimization is enabled, we don't use this for
the icount structure in pass 1. This is because the gain CPU gain is
nearly nil for that pass and the sacrificed memory does not justify
the increase in RAM.
(It could be that during pass 1, if more than 17% if possible inodes
has link_count>1 (466m inodes in the 40TB with 2.8bn possible inodes
case) then it becomes more memory efficient to use the full map
implementation in terms of memory. However, this is extremely
unlikely given that most file systems are heavily over-provisioned in
terms of the number of inodes in the system.)
Jaco Kroon [Wed, 23 Aug 2017 17:54:25 +0000 (13:54 -0400)]
e2fsck: optimize out the use region_t in scan_extent_node()
Since extents have a guarantee of being monotonically increasing we
merely need to check that block n+1 starts after block n. This is a
simple enough check and we can perform this by calculating the next
expected logical block instead of using the region usage tracking data
abstraction.
Theodore Ts'o [Wed, 23 Aug 2017 14:57:25 +0000 (10:57 -0400)]
tune2fs: explain why an fsck is needed
Currently tune2fs just says without any explanation, "run fsck -f".
Add a short explanation that a freshly checked file system is required
to reduce user confusion. (We could add even more details, but
hopefully this is enough.)
Theodore Ts'o [Wed, 23 Aug 2017 14:30:09 +0000 (10:30 -0400)]
mke2fs: automatically use 256 byte inodes if project feature enabled
If the inode size is not implicitly requested on the command line, and
it is too small to support the project feature, automatically promote
the inode size to be 256 bytes so that the project feature will work.
Note the previous test to check for a too-small inode size didn't work
because it checked before inode size was set in fs_param. Hence, it
was possible to create file systems with a 128 byte inode and the
project feature enabled.
Theodore Ts'o [Tue, 22 Aug 2017 16:15:26 +0000 (12:15 -0400)]
debian: remove support for pre-multiarch versions of Debian
All versions of Debian after Wheezy support Multiarch, so we can
simply the Debian control.in and rules file by removing support for
older versions of Debian without Multiarch support.
Theodore Ts'o [Tue, 22 Aug 2017 15:23:21 +0000 (11:23 -0400)]
libext2fs: avoid potential out-of-bounds write if pread/pread64 fails
In unix_io.c's raw_read_block(), if the initial attempt to call
pread/pread64 fails because the offset is insane, the variable
"actual" is left at -1, and then when lseek fails, the cleanup
function will try to clear (as an out-of-bounds write) a single byte
before the buffer. Fix this.
Addresses-Debian-Bug: #871539
Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reported-by: Jakub Wilk <jwilk@jwilk.net>
Theodore Ts'o [Tue, 22 Aug 2017 14:37:10 +0000 (10:37 -0400)]
debian: remove support libuuid/libblkid packages
Remove support for util-linux prior to 2.16, when e2fsprogs provided
its own copy of libuuid and libblkid. This is only needed for Debian
distributions prior to Wheezy, which is no longer supported.
Theodore Ts'o [Tue, 22 Aug 2017 04:54:15 +0000 (00:54 -0400)]
libsupport: don't try accessing the project quota for 128 byte inodes
If the file system has 128 byte inode, it's not possible to access its
project quota id, since the inode is too small. So prevent a
potential out-of-bounds read in get_qid().
The problem was found by valgrind and American Fuzzy Lop.
Theodore Ts'o [Tue, 22 Aug 2017 01:20:38 +0000 (21:20 -0400)]
e2fsck: in ask_yn() fall back to English yes/no characters
In the case of missing translations, if the translation for y/n is
missing due to fuzzy transactions, such that user is told to use
<y/n>, those characters will work correctly.
Theodore Ts'o [Mon, 14 Aug 2017 23:52:39 +0000 (19:52 -0400)]
e2fsck: add optimization for large, fragmented sparse files
The code which checks for overlapping logical blocks in an extent tree
is O(h*e) in time, where h is the number of holes in the file, and e
is the number of extents in the file. So a file with a large number
of holes can take e2fsck a long time process. Optimize this taking
advantage of the fact the vast majority of the time, region_allocate()
is called with increasing logical block numbers, so we are almost
always append onto the end of the region list.
Theodore Ts'o [Mon, 14 Aug 2017 01:07:21 +0000 (21:07 -0400)]
mke2fs: fix UI problem caused by fuzzy translations
When the original message was changed from "(y, n)" to "(y, N)", this
caused the translations to be marked as "fuzzy". For those
translations that use a different characters for yes and no --- for
example, German, which uses j and n for "ja" and "nein" --- not having
the translation can cause user confusion since the user will type 'y',
and it will be interpreted as "No", since mke2fs is expecting that the
user will type some other character, such as 'j' or 'J' for "Ja" in
the German locale.
Theodore Ts'o [Sun, 13 Aug 2017 18:45:27 +0000 (14:45 -0400)]
libsupport: fix 32-bit quota test failures
On 32-bit platform some of the util_dqblk structures have a type of
long long. So we need to use %lld and casts to make sure the right
thing happens on both 32-bit and 64-bit platforms.
Theodore Ts'o [Fri, 4 Aug 2017 06:01:43 +0000 (02:01 -0400)]
Remove special mips libraries from Debian build
These libraries were needed to support arcboot, which is obsolete and
no longer part of Debian. So drop these non-standard, legacy special
libraries that were only built on the mips platform.
Theodore Ts'o [Tue, 1 Aug 2017 14:26:11 +0000 (10:26 -0400)]
e2fsck: fix e2fsck -D for encrypted directories
If the directory entry is encrypted there may be embedded NUL
characters; hence, we should use memcmp instead of strncmp when
comparing strings. Otherwise, e2fsck can erroneously report that a
directory have duplicant entries when doing an e2fsck -D check.
libsupport: fix error handling in quota_write_inode
The error return value of quota_file_create() is no longer < 0,
and the error handling in quota_write_inode() is incorrect,
fix these. This also fix a tune2fs segfault that currently
occurs when we add project and quota features to an inode
exhaustion ext4 filesystem.
debugfs: fix "ls -p" to avoid printing garbage after the file name
In commit 68a1de3df3 (debugfs: pretty print encrypted filenames in the
ls command), a change was introduced in debugfs/ls.c which instead of
copying dirent->name and 0-terminating it, dirent->name is used
directly in printf.
However, instead of using the precision to limit the number of
characters output, the code uses the field width. As a result,
characters are output until a 0 is read, which results in garbage
after the file name.
Also fix two other instances of this in debugging messages that aren't
used, but fixing them will avoid potential future copypasta bugs.
Reported-by: Christian Gabriel <ch_gabriel@web.de> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Eric Sandeen [Sun, 23 Jul 2017 22:34:57 +0000 (18:34 -0400)]
tune2fs: edit dire warning about check intervals
Time & mount-count based checks have been off by default for quite some
time now, but the dire warning about disabling them remains in the
tune2fs manpage, which is confusing. We did "strongly consider
the consequences" and disabled it by default, no need to scare the
user about it now. Inform the user of the consequences in a more
measured tone.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Andreas Dilger <adilger@dilger.ca> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
resize2fs: sanity check the free blocks and inode counts
If the free block or free inodes count are larger than the number of
blocks or inodes in the system, request that the file system be
checked. Otherwise it's possible for calcuate_minimum_resize_size()
to hang in an infinite loop.
This problem was found using American Fuzzy Lop.
Reported-by: Adam Buchbinder <abuchbinder@google.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
E2fsck checks block numbers against the block_metadata_map before it
checks to see whether or not the block numbers are valid. So suppress
these harmless warnings.
If the superblock has invalid inode numbers for the user, group, or
project quota inodes, e2fsck should notice and offer to fix things by
zeroing out the invalid superblock field.
libext2fs: fix the s_log_block_size check in ext2fs_open()
The s_log_block_check can fail to detect an invalid value if it is
between UINT_MAX-9 and UINT_MAX, which can lead to ext2fs_open()
crashing with a division by zero error.
This bug was found using American Fuzzy Lop: http://lcamtuf.coredump.cx/afl/
Compiling with -fsanitize=undefined -fsanitize=address causes some
warnings of C code that has undefined behavior according to the C
standard bugs. None of the warnings should cause e2fsprogs
malfunction given a sane compiler running on architectures that Linux
can support. Still, it's better to clean up to code than not.
To fix up a complaint of a negative shift in hash function, update the
very dated hash we had been using for the revoke table with the
current generic hash used by the kernel.
Marc Thomas [Mon, 26 Jun 2017 15:39:47 +0000 (16:39 +0100)]
filefrag: fix GCC7.x compiler warning
../../misc/filefrag.c:591:33: warning: comparison between pointer and
zero character constant [-Wpointer-compare]
for (cpp = argv + optind; *cpp != '\0'; cpp++) {
Signed-off-by: Marc Thomas <marc@dragonfly.plus.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Theodore Ts'o [Mon, 19 Jun 2017 22:39:55 +0000 (18:39 -0400)]
mke2fs: fix hugefile creation so the hugefile(s) are contiguous
Commit 4f868703f6f2: "libext2fs: use fallocate for creating journals
and hugefiles" introduced a regression for the mke2fs hugefile
feature. The problem is that the fallocate library function
intersperses the extent tree metadata blocks with the data blocks, and
this violates the hugefile guarantee that the created files should be
physically contiguous on disk.
Unfortuantely the m_hugefile regression test was flawed, and didn't
pick up the regression.
This commit fixes the regression test so that it detects the problem
before fixing mke2fs, and also fixes the mke2fs hugefile by reverting
the mke2fs hugefile portion of commit 4f868703f6f2.
Theodore Ts'o [Sun, 4 Jun 2017 22:37:31 +0000 (18:37 -0400)]
libext2fs: correctly write up the backup superblocks in big endian systems
This bug has been around since we added support for metadata
checksums, but it was unmasked by commit bf9f3b6d5b ("e2fsck: exit
with exit status 0 if no errors were fixed"). The backup superblocks
are not supposed to have the EXT2_VALID_FS or the NEEDS_RECOVERY bits
set, and earlier 1.43.x versions of e2fsprogs were byte swapping the
shadow superblock each time it was written, so that every other backup
superblock was incorrectly byte swapped.
Fortunately the primary backup superblock was correctly written
(modulo having the VALID_FS bit set when it should not have been set)
so for the most part no one noticed. And very few architectures use
big endian byte ordering these days. (Even IBM has seen the light
with the ppcle architecture. :-)
Fortunately commit bf9f3b6d5b caused f_desc_size_bad and
f_resize_inode to fail on a big endian system, which allowed me to
notice the issue and investigate.
Wang shilong [Tue, 30 May 2017 00:36:51 +0000 (20:36 -0400)]
tune2fs: fix BUGs of tuning project quota
There are several problems for project quota enable/disable:
tune2fs -O ^project did not work, because @clear_ok_features
did not include @EXT4_FEATURE_RO_COMPAT_PROJECT.
update_feature_set() works for -O option, but tune2fs -Q prj/^prj
did not work well, because function handle_quota_options()
did not set and clear @EXT4_FEATURE_RO_COMPAT_PROJECT feature very well.
one warning message is removed, because with project feature
enabled, quota feature will be enabled automatically.
Signed-off-by: Wang Shilong <wshilong@ddn.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
e2fsck: don't flush to device opened in read-only mode
If the e2fsck is called with both the -f and -n options, it will
complete with an exit status of 8 due to an error when trying to flush
the io_channel (which was opened read-only) when built on on Cygwin on
Windows 8.1 and Windows 10. Apparently Cygwin is unhappy when fsync
is called on a file descriptor opened read-only.
Theodore Ts'o [Thu, 25 May 2017 17:11:40 +0000 (13:11 -0400)]
tests: fix expected output for f_detect_junk
The expect files for f_detect_junk had gotten out of sync with the
code base, and since this test is optional (it depends on libmagic
being installed), we hadn't noticed.
Darrick J. Wong [Thu, 25 May 2017 01:56:36 +0000 (21:56 -0400)]
e2fsck: fix sparse bmap to extent conversion
When e2fsck is trying to convert a sparse block-mapped file to an extent
file, we incorrectly merge block mappings that are physically contiguous
but not logically contiguous because of insufficient checking with the
extent we're constructing. Therefore, compare the logical offsets for
contiguity as well.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Eric Biggers [Thu, 25 May 2017 01:48:41 +0000 (21:48 -0400)]
libext2fs: correctly subtract xattr blocks on bigalloc filesystems
ext2fs_inode_data_blocks2() calculates an inode's data block count by
subtracting the external xattr block, if any, from the total blocks.
But on bigalloc filesystems, the xattr "block" is actually a whole
cluster, so ext2fs_inode_data_blocks2() would return a too-large value.
It seems this could have caused several different problems, but the one
I encountered was that xfstest generic/399 failed in the "bigalloc"
config because e2fsck incorrectly considered a symlink on the filesystem
to be corrupted at the end of the test. This happened because e2fsck
incorrectly calculated a nonzero data block count for a "fast" symlink
with an external xattr block and therefore treated it as a "slow"
symlink, which failed validation.
Fix this by updating ext2fs_inode_data_blocks2() to subtract the cluster
size rather than the block size.
Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Eric Whitney [Thu, 25 May 2017 01:34:20 +0000 (21:34 -0400)]
e2fsck: fix multiply-claimed block quota accounting when deleting files
As e2fsck processes each file in pass1, the actual file system quota is
increased by the number of blocks discovered in the file. This can
include both non-multiply-claimed and multiply-claimed blocks, if the
latter exist. However, if a file containing multiply-claimed blocks
is then deleted in pass1b, those blocks are not taken into account when
decreasing the actual quota. In this case, the new quota values written
to the file system by e2fsck overstate the space actually consumed.
And, e2fsck must be run twice on the file system to fully correct
quota.
Fix this by counting multiply-claimed blocks as a debit to quota when
deleting files in pass1b.
Signed-off-by: Eric Whitney <enwlinux@gmail.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Andreas Dilger <adilger@dilger.ca>
Mike Frysinger [Fri, 19 May 2017 17:25:59 +0000 (13:25 -0400)]
include sys/sysmacros.h as needed
The minor/major/makedev macros are not entirely standard. glibc has had
the definitions in sys/sysmacros.h since the start, and wants to move away
from always defining them implicitly via sys/types.h (as this pollutes the
namespace in violation of POSIX). Other C libraries have already dropped
them. Since the configure script already checks for this header, use that
to pull in the header in files that use these macros.
Signed-off-by: Mike Frysinger <vapier@gentoo.org> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Eric Biggers [Mon, 8 May 2017 22:47:57 +0000 (15:47 -0700)]
misc: fix 'zero_hugefiles = false' regression
When mk_hugefiles() was switched to use ext2fs_fallocate(), it was
accidentally changed to ignore the 'zero_hugefiles = false' setting,
which should cause hugefiles to be allocated without initializing their
contents. Fix this by only passing EXT2_FALLOCATE_ZERO_BLOCKS to
ext2fs_fallocate() when zero_hugefiles is true.
Google-Bug-Id: 38037607 Fixes: 4f868703f6f2 ("libext2fs: use fallocate for creating journals and hugefiles") Cc: Darrick J. Wong <darrick.wong@oracle.com> Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
e2fsck: update quota when optimizing the extent tree
If quota is enabled, optimizing the extent tree wouldn't update the
in-memory quota statistics, so that a subsequent e2fsck run would show
that the quota usage statistics on disk were incorrect.
Joe Richey [Mon, 3 Apr 2017 16:52:50 +0000 (16:52 +0000)]
e4crypt: fix error handling for KEYCTL_GET_KEYRING_ID
Due to some interesting behaviour in keyctl (as described in the
comments), we use KEYCTL_GET_KEYRING_ID to translate the special value
of KEY_SPEC_SESSION_KEYRING to a real keyring id. However, how we
currently do this is flawed in two ways.
First, if KEYCTL_GET_KEYRING_ID fails, we don't detect it as it returns
-1 and zero is used for an error value in get_keyring_id. Second, if the
user specifies "-k @s" the translation never runs and the undesireable
behavior occurs.
These are both fixed by doing the translation outside of get_keyring_id.
Signed-off-by: Joe Richey <joerichey@google.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Eric Whitney [Sun, 2 Apr 2017 16:57:05 +0000 (12:57 -0400)]
e2fsck: fix quota accounting to use cluster units
The quota accounting code in e2fsck's pass 1 and pass 3 uses block units
rather than cluster units when recording the allocated space consumed by
files and directories. In pass 1, this causes a large undercount of
actual quota results and test failures for xfstests generic/383, /384,
/385, and /386 on bigalloc file systems. In pass 3, this results in
quota accounting errors when the lost+found directory is either extended
or recreated on a bigalloc file system.
Use clusters rather than blocks when accounting for allocated space,
and correct a related header comment in the quota code.
Note that pass 1b also contains call sites for quota_data_sub() that
also need to be addressed. However, it appears that more than just
unit conversion may be needed, so that will be left to a future patch.
Signed-off-by: Eric Whitney <enwlinux@gmail.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
Eric Biggers [Fri, 17 Mar 2017 22:38:36 +0000 (15:38 -0700)]
libext2fs: apply LDFLAGS when building tst_inline_data
If libext2fs was compiled with an external libblkid pointed to by
LDFLAGS, then linking the tst_inline_data program failed because the
requested linker flags were not used. Fix this by adding $(ALL_LDFLAGS)
to the build rule, as is done for the other test programs.
Fixes: 31253488385a ("libext2fs: add a unit test for inline data") Signed-off-by: Eric Biggers <ebiggers@google.com> Signed-off-by: Theodore Ts'o <tytso@mit.edu>
projects / thirdparty / e2fsprogs.git / log
