Darrick J. Wong [Tue, 13 Oct 2015 23:25:23 +0000 (10:25 +1100)]
xfs_db: trash the block at the top of the cursor stack
Add a new -z option to blocktrash to make it trash the block that's at
the top of the stack, so that we can perform targeted fuzzing. While
we're at it, prevent fuzzing off the end of the buffer and add a -o
parameter so that we can specify an offset to start fuzzing from.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Tue, 13 Oct 2015 23:24:47 +0000 (10:24 +1100)]
xfs_db: enable blocktrash for checksummed filesystems
Disable the write verifiers when we're trashing a block. With this
in place, create a xfs fuzzer script that formats, populates, corrupts,
tries to use, repairs, and tries again to use a crash test xfs image.
Hopefully this will shake out some v5 filesystem bugs.
This allow trashing of log blocks and symlinks, and require the
caller to explicitly ask for trashing of log blocks and super
blocks. Allowing log blocks by default skews the trashing heavily
in favor of (probably unused) log blocks, which doesn't help us with
fuzzing. Furthermore, trashing the superblock results in a time
consuming sector by sector superblock hunt.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Tue, 13 Oct 2015 23:23:38 +0000 (10:23 +1100)]
xfs_io: support reflinking and deduping file ranges
Wire up xfs_io to use the XFS range clone and dedupe ioctls to make
files share data blocks.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Mon, 7 Sep 2015 00:13:41 +0000 (10:13 +1000)]
xfs_repair: release corrupt directory node buffer
If repair encounters a dir node block that fails checksum or
verification, free the buffer before the directory gets rebuilt.
Reported-by: Eric Sandeen <sandeen@sandeen.net> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Carlos Maiolino <cmaiolino@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Brian Foster [Wed, 2 Sep 2015 22:43:23 +0000 (08:43 +1000)]
xfs_repair: update btree ptr when attr node level moves to next buffer
xfs_repair walks the attribute fork btree for files with a significant
number of extended attributes. It creates a cursor, walks the leaf
blocks, and verifies the path from each leaf block back to the root of
the tree. Eryu reports that the following test causes xfs_repair to
report corruption on 512b filesystems:
num_xattrs=577
for ((i = 1; i <= $num_xattrs; i++)); do
name="user.attr_$(printf "%04d" $i)"
setfattr -n $name -v "val_$(printf "%04d" $i)" <file>
done
xfs_repair complains that the block number of the leaf (level 0) does
not match the block number of the level 1 node block entry. This occurs
as soon as the left-most level 1 node block is completely processed and
the cursor is walked to the next level 1 block in the array. The problem
is that while verify_da_path() updates level 1 of the cursor to the next
level 1 buffer, it fails to correctly update the btree pointer to the
entry list of the new buffer. As a result, the child leaf block of the
next node block is incorrectly validated against the entry list of the
previous node block.
Update verify_da_path() to correctly update the btree pointer to the
entry list of the new node block when the cursor is walked forward at
higher (non-leaf) levels.
Reported-by: Eryu Guan <eguan@redhat.com> Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Eric Sandeen <sandeen@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Lucas Stach [Wed, 2 Sep 2015 22:43:16 +0000 (08:43 +1000)]
progs: use CURDIR instead of relative paths for header symlinks
This fixes broken header symlinks when make isn't triggered from the
xfsprogs source location, but as a recursion from another make in a
different directory. This is a common pattern found in cross build
systems.
Signed-off-by: Lucas Stach <dev@lynxeye.de> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Wed, 2 Sep 2015 22:43:07 +0000 (08:43 +1000)]
xfs_repair: force not-so-bad bmbt blocks back through the verifier
If during prefetch we encounter a bmbt block that fails the CRC check
due to corruption in the unused part of the block, force the buffer
back through the non-prefetch verifiers later so that the CRC is
updated. Otherwise, the bad checksum goes unfixed and the kernel will
still flag the bmbt block as invalid.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Check the v5 fields (uuid, blocknr, owner) of attribute blocks for
obvious errors while scanning xattr blocks. If the ownership info
is incorrect, kill the block.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Wed, 2 Sep 2015 22:42:37 +0000 (08:42 +1000)]
xfs_repair: ignore "repaired" flag after we decide to clear xattr block
If in the course of examining extended attribute block contents we
first decide to repair an entry (*repair = 1) but secondly decide to
clear the whole block, set *repair = 0 because the clearing action
only happens if *repair == 0. Put another way, if we're nuking a
block, don't pretend like we've fixed it too.
v2: fix all the paths to clear the attr block if the processing
functions error out.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Wed, 2 Sep 2015 22:42:01 +0000 (08:42 +1000)]
libxfs: clear buffer state flags in libxfs_getbuf and variants
When we're running xfs_repair with prefetch enabled, it's possible
that repair will decide to clear an inode without examining all
metadata blocks owned by that inode. This leaves the unreferenced
prefetched buffers marked UNCHECKED, which will cause a subsequent CRC
error if the block is reallocated to a different structure and read
more than once. Typically this happens when a large directory is
corrupted and lost+found has to grow to accomodate all the
disconnected inodes.
In libxfs_getbuf*(), we're supposed to return an unused buffer which
has a clean state. Unfortunately, things like UNCHECKED can hang
around to cause incorrect verifier errors later, so change those
functions to launder the state bits clean.
v2: Change the function name to reset_buf_state() to reflect what
the function is trying to accomplish.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Wed, 2 Sep 2015 22:41:48 +0000 (08:41 +1000)]
libxfs: fix XFS_WANT_CORRUPTED_* macros to return negative error codes
Since the rest of libxfs returns negative error codes, these two sanity
checking macros ought to have the same applied. While we're at it,
fix a couple more sign errors in the same file.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Wed, 2 Sep 2015 22:41:33 +0000 (08:41 +1000)]
libxfs: verifier should set buffer error when da block has a bad magic number
If xfs_da3_node_read_verify() doesn't recognize the magic number of a
buffer it's just read, set the buffer error to -EFSCORRUPTED so that
the error can be sent up to userspace. Without this patch we'll
notice the bad magic eventually while trying to traverse or change
the block, but we really ought to fail early in the verifier.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Addresses-Coverity-Id: 1298008 Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Tue, 25 Aug 2015 00:10:51 +0000 (10:10 +1000)]
xfsprogs: revert OS X dummy function changes
Commit ff6f019d ("xfsprogs: missing and dummy calls for OS X
support") was committed prematurely. Revert it for now so that
better solutions can be committed cleanly.
On examination of the log via xfs_logprint, none of the symlink
buffers in the log had a bad magic number, nor were any other types
of buffer log format headers mis-identified as symlink buffers.
Tracing was used to find the buffer the kernel was tripping over,
and xfs_db identified it's contents as:
This is a remote attribute buffer, which are notable in that they
are not logged but are instead written synchronously by the remote
attribute code so that they exist on disk before the attribute
transactions are committed to the journal.
The above remote attribute block has an invalid LSN in it - cycle
0xd002000, block 0 - which means when log recovery comes along to
determine if the transaction that writes to the underlying block
should be replayed, it sees a block that has a future LSN and so
does not replay the buffer data in the transaction. Instead, it
validates the buffer magic number and attaches the buffer verifier
to it. It is this buffer magic number check that is failing in the
above assert, indicating that we skipped replay due to the LSN of
the underlying buffer.
The problem here is that the remote attribute buffers cannot have a
valid LSN placed into them, because the transaction that contains
the attribute tree pointer changes and the block allocation that the
attribute data is being written to hasn't yet been committed. Hence
the LSN field in the attribute block is completely unwritten,
thereby leaving the underlying contents of the block in the LSN
field. It could have any value, and hence a future overwrite of the
block by log recovery may or may not work correctly.
Fix this by always writing an invalid LSN to the remote attribute
block, as any buffer in log recovery that needs to write over the
remote attribute should occur. We are protected from having old data
written over the attribute by the fact that freeing the block before
the remote attribute is written will result in the buffer being
marked stale in the log and so all changes prior to the buffer stale
transaction will be cancelled by log recovery.
Hence it is safe to ignore the LSN in the case or synchronously
written, unlogged metadata such as remote attribute blocks, and to
ensure we do that correctly, we need to write an invalid LSN to all
remote attribute blocks to trigger immediate recovery of metadata
that is written over the top.
As a further protection for filesystems that may already have remote
attribute blocks with bad LSNs on disk, change the log recovery code
to always trigger immediate recovery of metadata over remote
attribute blocks.
cc: <stable@vger.kernel.org> Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Jan Kara [Sun, 23 Aug 2015 22:59:12 +0000 (08:59 +1000)]
xfs: Fix uninitialized return value in xfs_alloc_fix_freelist()
xfs_alloc_fix_freelist() can sometimes jump to out_agbp_relse
without ever setting value of 'error' variable which is then
returned. This can happen e.g. when pag->pagf_init is set but AG is
for metadata and we want to allocate user data.
Fix the problem by initializing 'error' to 0, which is the desired
return value when we decide to skip this group.
CC: xfs@oss.sgi.com
Coverity-id: 1309714 Signed-off-by: Jan Kara <jack@suse.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Wed, 29 Jul 2015 01:48:02 +0000 (11:48 +1000)]
xfs: remote attributes need to be considered data
We don't log remote attribute contents, and instead write them
synchronously before we commit the block allocation and attribute
tree update transaction. As a result we are writing to the allocated
space before the allcoation has been made permanent.
As a result, we cannot consider this allocation to be a metadata
allocation. Metadata allocation can take blocks from the free list
and so reuse them before the transaction that freed the block is
committed to disk. This behaviour is perfectly fine for journalled
metadata changes as log recovery will ensure the free operation is
replayed before the overwrite, but for remote attribute writes this
is not the case.
Hence we have to consider the remote attribute blocks to contain
data and allocate accordingly. We do this by dropping the
XFS_BMAPI_METADATA flag from the block allocation. This means the
allocation will not use blocks that are on the busy list without
first ensuring that the freeing transaction has been committed to
disk and the blocks removed from the busy list. This ensures we will
never overwrite a freed block without first ensuring that it is
really free.
Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Wed, 29 Jul 2015 01:51:01 +0000 (11:51 +1000)]
xfs: xfs_bunmapi() does not need XFS_BMAPI_METADATA flag
xfs_bunmapi() doesn't care what type of extent is being freed and
does not look at the XFS_BMAPI_METADATA flag at all. As such we can
remove the XFS_BMAPI_METADATA from all callers that use it.
Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
added these (unsigned char *) casts, but then the _SIZE macros
return "7" - size of a pointer minus one - not the length of
the string. This is harmless in the kernel, because the _SIZE
macros are not used, but as we sync up with userspace, this will
matter.
I don't think the cast is necessary; i.e. assigning the string
literal to an unsigned char *, or passing it to a function
expecting an unsigned char *, should be ok, right?
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Jan Kara [Mon, 24 Aug 2015 01:52:45 +0000 (11:52 +1000)]
libxfs: Fix file type directory corruption for btree directories
Users have occasionally reported that file type for some directory
entries is wrong. This mostly happened after updating libraries some
libraries. After some debugging the problem was traced down to
xfs_dir2_node_replace(). The function uses args->filetype as a file type
to store in the replaced directory entry however it also calls
xfs_da3_node_lookup_int() which will store file type of the current
directory entry in args->filetype. Thus we fail to change file type of a
directory entry to a proper type.
Fix the problem by storing new file type in a local variable before
calling xfs_da3_node_lookup_int().
Reported-by: Giacomo Comes <comes@naic.edu> Signed-off-by: Jan Kara <jack@suse.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 24 Aug 2015 01:52:45 +0000 (11:52 +1000)]
libxfs: remove self-assignment in libxfs/util.c
We don't have percpu counters in userspace, so libxfs plays
tricks. Rather than calling percpu_counter_set() in
xfs_reinit_percpu_counters, we just directly assign
the values in mp->m_sb to the counters in mp.
But this was already handled by #defining the percpu counters
in the mount structure to those in the superblock, i.e.:
so we actually end up with pointless self-assignment.
Define away the xfs_reinit_percpu_counters() function,
because it's a no-op.
Addresses-Coverity-Id: 1298009 Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 24 Aug 2015 01:52:45 +0000 (11:52 +1000)]
xfs_repair: unconditionally free blockmaps when threads complete
blkmap_free() doesn't actually free the block map unless it's
inordinately large; this keeps us from constantly freeing
and re-allocating blockmaps for each inode, which makes sense.
However, once the threads which have allocated these structures
exit, we should actually free them; they can grow up to 2MB
for each of the data and attr maps, for each thread, and not
be freed through the normal blkmap_free() test.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 24 Aug 2015 01:52:45 +0000 (11:52 +1000)]
xfs_repair: call IRELE(ip) after libxfs_trans_iget calls
Commit 260c85e libxfs: dont free xfs_inode until complete
changed the alloc/free convention a bit:
Originally, the xfs_inode are released upon the first
call to xfs_trans_cancel, xfs_trans_commit, or
inode_item_done.
<snip>
This patch does the following:
1) Removes the iput from the transaction completion and
requires that the xfs_inode allocators call IRELE()
when they are done with the pointer.
But that change missed several callers in xfs_repair phase6;
fix that up.
Addresses-Coverity-Id: 1315100
Addresses-Coverity-Id: 1315101 Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 24 Aug 2015 01:52:42 +0000 (11:52 +1000)]
libxfs: fix memory leasks in libxfs_umount()
libxfs_umount was failing to free a handful of resources; fix that
up. Call it from xfs_copy as well, while we're at it; every other
libxfs_mount has a libxfs_umount counterpart, at least on a clean
exit.
[dchinner: fix superblock buffer leak uncovered by adding
libxfs_umount() to xfs_copy. ]
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Sun, 23 Aug 2015 23:21:01 +0000 (09:21 +1000)]
xfs_repair: fix broken EFSBADCRC/EFSCORRUPTED usage with buffer errors
When we encounter CRC or verifier errors, bp->b_error is set to
-EFSBADCRC and -EFSCORRUPTED; note the negative sign. For whatever
reason, repair and db use the positive versions, and therefore fail to
notice the error, so fix all the broken uses.
Note however that the db and repair turn the negative codes returned
by libxfs into positive codes that can be used with strerror.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Eric Sandeen <sandeen@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Sun, 23 Aug 2015 23:21:01 +0000 (09:21 +1000)]
xfs_db: don't crash on a corrupt inode
If the user selects a corrupt inode via the 'inode XXX' command, the
read verifier will fail and the io cursor at the top of the ring will
not have any data attached. When this is the case, we cannot
dereference the NULL pointer or xfs_db will crash. Therefore, check
the buffer pointer before using it.
It's arguable that we ought to retry the read without the verifiers
if the inode is corrupt or fails CRC, since this /is/ a debugging
tool, and maybe you wanted the contents anyway.
[dchinner: fixes xfs/003 on 1k block size failure]
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Sun, 23 Aug 2015 23:21:01 +0000 (09:21 +1000)]
libxfs: readahead of dir3 data blocks should use the read verifier
In the dir3 data block readahead function, use the regular read
verifier to check the block's CRC and spot-check the block contents
instead of calling the spot-checking routine directly. This prevents
corrupted directory data blocks from being read into the kernel, which
can lead to garbage ls output and directory loops (if say one of the
entries contains invalid characters).
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eryu Guan [Sun, 23 Aug 2015 23:21:01 +0000 (09:21 +1000)]
repair: fix wrong logic when validating node magic number
Magic number is wrong only when != XFS_DA_NODE_MAGIC and
!= XFS_DA3_NODE_MAGIC.
This is triggered by shared/002 when testing 512 block size XFS.
Phase 1 - find and verify superblock...
Phase 2 - using internal log
- scan filesystem freespace and inode maps...
- found root inode chunk
Phase 3 - for each AG...
- scan (but don't clear) agi unlinked lists...
- process known inodes and perform inode discovery...
- agno = 0
bad magic number febe in block 64 (108) for directory inode 35
......
Fix it by changing "||" to "&&".
Signed-off-by: Eryu Guan <eguan@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Roger Willcocks [Tue, 18 Aug 2015 07:53:18 +0000 (17:53 +1000)]
libxfs: v3 inodes are only valid on crc-enabled filesystems
xfs_repair was not detecting that version 3 inodes are invalid for
for non-CRC filesystems. The result is specific inode corruptions go
undetected and hence aren't repaired if only the version number is
out of range.
The core of the problem is that the XFS_DINODE_GOOD_VERSION() macro
doesn't know that valid inode versions are dependent on a superblock
version number. Fix this in libxfs, and propagate the new function
out into the rest of xfsprogs to fix the issue.
[dchinner: forward port from 3.2.4 to 4.2.0-rc1, move
xfs_dinode_good_version() to libxfs/xfs_inode-buf.c with all the
other dinode validation functions. ]
Reported-by: Leslie Rhorer <lrhorer@mygrande.net> Signed-off-by: Roger Willcocks <roger@filmlight.ltd.uk> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Jan Kara [Tue, 18 Aug 2015 07:53:17 +0000 (17:53 +1000)]
libxfs: Fix attr leaf block definition
struct xfs_attr_leafblock contains 'entries' array which is declared
with size 1 altough it can in fact contain much more entries. Since this
array is followed by further struct members, gcc (at least in version
4.8.3) thinks that the array has the fixed size of 1 element and thus
optimizes away all accesses beyond the end of array resulting in
non-working code. In particular this problem was seen with
xfsprogs-3.1.8.
Signed-off-by: Jan Kara <jack@suse.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Somehow the tests got written as being constrained on what options
are specified - and in what order! - vs actually testing for
incompatible feature sets.
It's fine to specify both crc & ftype options, as long as it's an
allowed combination, so just test for the incompatible combination
(crc=1 and ftype=0) after all options have been processed.
[dchinner: fix dirftype init value so mkfs default config works]
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Brian Foster [Tue, 18 Aug 2015 07:53:17 +0000 (17:53 +1000)]
libxfs: remove sparse inode mount warning
The sparse inodes experimental feature warning fires multiple times
during mkfs because the warning is emitted as part of the superblock
verifier codepath. The warning is intended as a mount-time warning only
and has been relocated as such in the kernel repo.
Remove the warning from libxfs such that it is not emitted from
userspace.
Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eryu Guan [Tue, 18 Aug 2015 07:53:17 +0000 (17:53 +1000)]
xfsprogs: fix syntax error in include/buildmacros
There's an extra ";" in include/buildmacros, which causes make
install-dev to fail
......
Installing libhandle-install-dev
cd ../libhandle/.libs; ... if [ "x/usr/lib64" != "x/usr/lib64"; ]; ...
/bin/sh: line 0: [: missing `]'
/bin/sh: ]: command not found
......
This was introduced by 02ef543 libhandle: fix installation for symlinked /usr
Signed-off-by: Eryu Guan <eguan@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Jan Tulak [Tue, 18 Aug 2015 07:53:17 +0000 (17:53 +1000)]
xfsprogs: prevent LIST_ macros conflicts
BSD 4.4 added some LIST_ macros into system header files, which
causes "macro redefined" warnings. To ensure we use our own macros,
undefine the system ones at first.
The conflicting macros are LIST_HEAD and LIST_HEAD_INIT
Signed-off-by: Jan Tulak <jtulak@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Jan Tulak [Tue, 18 Aug 2015 07:53:17 +0000 (17:53 +1000)]
xfsprogs: undefined variable fix
Typo fix, which wasn't catch earlier due to #ifdef branching. The
'rmnttomname' does not exists anywhere and looks like a hybrid between
rmntfromname and rmntonname. And because the previous if has has
'fromname' on both arguments of realpath, I choose the same approach
when fixing it.
Signed-off-by: Jan Tulak <jtulak@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Mon, 3 Aug 2015 22:36:44 +0000 (08:36 +1000)]
repair: use sb_meta_uuid for checking of metadata headers
Now that we can change the uuid on v5 filesystems, we always need to
verify the metadata uuid against sb_meta_uuid, not sb_uuid. This
fixes quite a few xfstests failures when UUIDs are changed before
executing tests.
Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Eric Sandeen <sandeen@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 3 Aug 2015 22:36:41 +0000 (08:36 +1000)]
xfs_repair: Fix malloc size of rt_ext_tree_ptr
rt_ext_tree_ptr points to an avl64tree_desc_t, but we malloc memory
according to the size of avltree_desc_t. Oddly, the latter happens
to be larger, so we're ok, but may as well make it correct.
Addresses-Coverity-Id: 1297533 Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 3 Aug 2015 00:45:00 +0000 (10:45 +1000)]
xfsprogs: Add new sb_meta_uuid field, update userspace tools to manipulate it
This adds a new superblock field, sb_meta_uuid. This allows us to
change the use-visible UUID on crc-enabled filesytems from userspace
if desired, by copying the existing UUID to the new location for
metadata comparisons. If this is done, an incompat flag must be
set to prevent older filesystems from mounting the filesystem, but
the original UUID can be restored, and the incompat flag removed,
with a new xfs_db / xfs_admin UUID command, "restore."
Much of this patch mirrors the kernel patch in simply renaming
the field used for metadata uuid comparison; other bits:
* Teach xfs_db to print the new meta_uuid field
* Allow xfs_db to generate a new UUID for CRC-enabled filesystems
* Allow xfs_db to revert to the original UUID and clear the flag
* Fix up xfs_copy to work with CRC-enabled filesystems
* Update the xfs_admin manpage to show the UUID "restore" command
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Theodore Ts'o [Mon, 3 Aug 2015 00:17:02 +0000 (10:17 +1000)]
xfsprogs: use "unsigned short" instead of ushort
Android's bionic libc doesn't define ushort. There isn't a real
benefit (other than perhaps conciseness) to use ushort over "unsigned
short", and it's only used in a handful of files in xfsprogs. So
change over to using unsigned short everywhere.
Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Theodore Ts'o [Mon, 3 Aug 2015 00:16:48 +0000 (10:16 +1000)]
xfsprogs: avoid use of si_tid in struct xlog_split_item
In Android's bionic libc (as well as the Linux kernel's
include/uapi/asm-generic/siginfo.h), si_tid is a #define to provide
backwards compatibility for the timerid in the siginfo structure.
This breaks the compile of logprint/log_misc.c. Change this to be
si_xtid in order to avoid a namespace collision
Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Theodore Ts'o [Mon, 3 Aug 2015 00:16:21 +0000 (10:16 +1000)]
xfsprogs: define and use BUILD_CC in configure.ac for cross compilation
In order to support cross-compilation, we need to build gen_crc32table
using the C compiler targetted for the build platform, since it is run
as part of the build process.
Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Theodore Ts'o [Mon, 3 Aug 2015 00:16:01 +0000 (10:16 +1000)]
xfsprogs: pull in libgen.h to get prototype for basename()
The function prototype for basename() is in <libgen.h>, per Posix.
Without the the function prototype, the build will throw errors due to
the missing prototype.
On glibc, using libgen.h will force the use of Poxis's basename(),
instead of glibc's basename() with GNU extensions. However, xfsprogs
doesn't depend on any of the GNU extensions, so this is fine.
Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Theodore Ts'o [Mon, 3 Aug 2015 00:15:45 +0000 (10:15 +1000)]
xfsprogs: define NBBY if not defined by the system header files
Android's bionic libc doesn't define NBBY; this isn't a standard
define, and since all modern/sane platforms have 8 bits per byte, use
this as a default.
Signed-off-by: Theodore Ts'o <tytso@mit.edu> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Jan Tulak [Mon, 3 Aug 2015 00:06:21 +0000 (10:06 +1000)]
xfsprogs: Don't Make .po files with gettext disabled
"po" target is added only if gettext binary is found.
Without this patch, Make tried to build the target even
with --enable-gettext=no configure option, which led
to a failing build.
Signed-off-by: Jan Tulak <jtulak@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Jan Tulak [Mon, 3 Aug 2015 00:05:35 +0000 (10:05 +1000)]
xfsprogs: Search path for utilities unified
Currently, when autoconf is checking for an utility, every utility has
its own pathes defined independently. Unify it in a single variable
used for (almost) all utilities.
Also, add /opt/local/bin to the path.
Signed-off-by: Jan Tulak <jtulak@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Jan Tulak [Mon, 3 Aug 2015 00:05:08 +0000 (10:05 +1000)]
xfsprogs: blkid is now mandatory
Because blkid is here for a long time, I hereby propose a patch for
removing support for NOT having blkid. The current support through
set of #ifdef is prone to errors like
making a patch just in one of the branches, and according to a
recent talk between Christoph and Eric, it is not necessary to keep
it supported.
Remove code for checking ENABLE_BLKID, and the code when
ENABLE_BLKID is not defined. The only use of libdisk was in the
removed code, so remove libdisk too. It makes blkid required for
compilation.
[dchinner: also remove include/volume.h and include/dvh.h as
suggested by Christoph during review. ]
Signed-off-by: Jan Tulak <jtulak@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
platform_defs.h is a generated header file, which causes all kinds
of problems when installed on multiarch systems, and requires
workarounds in distribution packages. Instead move the small parts
of it needed in the installed xfs.h into xfs.h and keep
platform_defs.h private to xfsprogs.
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
xfsprogs: move __u*/__s* typedefs to per-port headers
Currently we have to install the autoconf-generated platform_defs.h
to get the defintions for these. But they are clearly a feature
of Linux vs non-Linux platforms so move them to the per-port headers
instead.
Note: in the long run it might be a good idea to just the standard
uint*_t/int*_t types instead.
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
We don't need the xfs/ prefix for local includes if we just add the
libxfs directory to the include path. Once that is done we only
need to link the installed headers into include/xfs.
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
xfsprogs: only install *format.h headers in install-qa
Now that we've properly split up the headers we don't need to install all
the libxfs-internal headers for xfstests. Just install the three headers
defining the on-disk format and xfs_arch.h which is required to compile
them instead.
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Fri, 31 Jul 2015 04:44:52 +0000 (14:44 +1000)]
build: create include/xfs before installing headers
Currently the install-headers rule from include/Makefile creates
include/xfs, but there is no guarantee that it will be the first
directory that make executes that rule in. Hence other directories
can race with the creation on include/xfs and fail.
Move the creation of include/xfs to occur before running the
install_headers rules on the subdirectories to avoid any possible
races with creation.
Brian Foster [Fri, 31 Jul 2015 01:12:44 +0000 (11:12 +1000)]
xfs: check min blks for random debug mode sparse allocations
The inode allocator enables random sparse inode chunk allocations in
DEBUG mode to facilitate testing. Sparse inode allocations are not
always possible, however, depending on the fs geometry. For example,
there is no possibility for a sparse inode allocation on filesystems
where the block size is large enough to fit one or more inode chunks
within a single block.
Fix up the DEBUG mode sparse inode allocation logic to trigger random
sparse allocations only when the geometry of the fs allows it.
Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
George Wang [Fri, 31 Jul 2015 01:12:44 +0000 (11:12 +1000)]
xfs: use percpu_counter_read_positive for mp->m_icount
Function percpu_counter_read just return the current counter, which can be
negative. This will cause the checking of "allocated inode
counts <= m_maxicount" false positive. Use percpu_counter_read_positive can
solve this problem, and be consistent with the purpose to introduce percpu
mechanism to xfs.
Signed-off-by: George Wang <xuw2015@gmail.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Fri, 31 Jul 2015 01:11:56 +0000 (11:11 +1000)]
xfs: clean up XFS_MIN_FREELIST macros
We no longer calculate the minimum freelist size from the on-disk
AGF, so we don't need the macros used for this. That means the
nested macros can be cleaned up, and turn this into an actual
function so the logic is clear and concise. This will make it much
easier to add support for the rmap btree when the time comes.
This also gets rid of the XFS_AG_MAXLEVELS macro used by these
freelist macros as it is simply a wrapper around a single variable.
Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Fri, 31 Jul 2015 01:10:56 +0000 (11:10 +1000)]
xfs: sanitise error handling in xfs_alloc_fix_freelist
The error handling is currently an inconsistent mess as every error
condition handles return values and releasing buffers individually.
Clean this up by using gotos and a sane error label stack.
Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Fri, 31 Jul 2015 01:09:56 +0000 (11:09 +1000)]
xfs: factor out free space extent length check
The longest extent length checks in xfs_alloc_fix_freelist() are now
essentially identical. Factor them out into a helper function, so we
know they are checking exactly the same thing before and after we
lock the AGF.
Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Fri, 31 Jul 2015 01:08:56 +0000 (11:08 +1000)]
xfs: xfs_alloc_fix_freelist() can use incore perag structures
At the moment, xfs_alloc_fix_freelist() uses a mix of per-ag based
access and agf buffer based access to freelist and space usage
information. However, once the AGF buffer is locked inside this
function, it is guaranteed that both the in-memory and on-disk
values are identical. xfs_alloc_fix_freelist() doesn't modify the
values in the structures directly, so it is a read-only user of the
infomration, and hence can use the per-ag structure exclusively for
determining what it should do.
This opens up an avenue for cleaning up a lot of duplicated logic
whose only difference is the structure it gets the data from, and in
doing so removes a lot of needless byte swapping overhead when
fixing up the free list.
Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Fri, 31 Jul 2015 01:07:56 +0000 (11:07 +1000)]
xfs: xfs_attr_inactive leaves inconsistent attr fork state behind
xfs_attr_inactive() is supposed to clean up the attribute fork when
the inode is being freed. While it removes attribute fork extents,
it completely ignores attributes in local format, which means that
there can still be active attributes on the inode after
xfs_attr_inactive() has run.
This leads to problems with concurrent inode writeback - the in-core
inode attribute fork is removed without locking on the assumption
that nothing will be attempting to access the attribute fork after a
call to xfs_attr_inactive() because it isn't supposed to exist on
disk any more.
To fix this, make xfs_attr_inactive() completely remove all traces
of the attribute fork from the inode, regardless of it's state.
Further, also remove the in-core attribute fork structure safely so
that there is nothing further that needs to be done by callers to
clean up the attribute fork. This means we can remove the in-core
and on-disk attribute forks atomically.
Also, on error simply remove the in-memory attribute fork. There's
nothing that can be done with it once we have failed to remove the
on-disk attribute fork, so we may as well just blow it away here
anyway.
cc: <stable@vger.kernel.org> # 3.12 to 4.0 Reported-by: Waiman Long <waiman.long@hp.com> Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Brian Foster [Fri, 31 Jul 2015 01:06:56 +0000 (11:06 +1000)]
xfs: always log the inode on unwritten extent conversion
The fsync() requirements for crash consistency on XFS are to flush file
data and force any in-core inode updates to the log. We currently check
whether the inode is pinned to identify whether the log needs to be
forced, since a non-zero pin count generally represents an inode that
has transactions awaiting a flush to the on-disk log.
This is not sufficient in all cases, however. Reports of xfstests test
generic/311 failures on ppc64/s390x hosts have identified failures to
fsync outstanding inode modifications due to the inode not being pinned
at the time of the fsync. This occurs because certain bmap updates can
complete by logging bmapbt buffers but without ever dirtying (and thus
pinning) the core inode. The following is a specific incarnation of this
problem:
In short, the unwritten extent conversion for the last write is lost
despite the fact that an fsync executed before the filesystem was
shutdown. Note that this is impossible to reproduce on v5 supers due to
unconditional time callbacks for di_changecount and highly difficult to
reproduce on CONFIG_HZ=1000 kernels due to those same callbacks
frequently updating cmtime prior to the bmap update. CONFIG_HZ=100
reduces timer granularity enough to increase the odds that time updates
are skipped and allows this to reproduce within a handful of attempts.
To deal with this problem, unconditionally log the core in the unwritten
extent conversion path. Fix up logflags after the extent conversion to
keep the extent update code consistent with the other extent update
helpers. This fixup is not necessary for the other (hole, delay) extent
helpers because they execute in the block allocation codepath, which
already logs the inode for other reasons (e.g., for di_nblocks).
Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Clearly indicating that the extent length is greater than MAXEXTLEN,
which is 2097151. A prior trace point shows the allocation was an
exact size match and that a length greater than MAXEXTLEN was asked
for:
We don't see this problem with extent size hints through the IO path
because we can't do single IOs large enough to trigger MAXEXTLEN
allocation. fallocate(), OTOH, is not limited in it's allocation
sizes and so needs help here.
The issue is that the extent size hint alignment is rounding up the
extent size past MAXEXTLEN, because xfs_bmapi_write() is not taking
into account extent size hints when calculating the maximum extent
length to allocate. xfs_bmapi_reserve_delalloc() is already doing
this, but direct extent allocation is not.
Unfortunately, the calculation in xfs_bmapi_reserve_delalloc() is
wrong, and it works only because delayed allocation extents are not
limited in size to MAXEXTLEN in the in-core extent tree. hence this
calculation does not work for direct allocation, and the delalloc
code needs fixing. This may, in fact be the underlying bug that
occassionally causes transaction overruns in delayed allocation
extent conversion, so now we know it's wrong we should fix it, too.
Many thanks to Brian Foster for finding this problem during review
of this patch.
Hence the fix, after much code reading, is to allow
xfs_bmap_extsize_align() to align partial extents when full
alignment would extend the alignment past MAXEXTLEN. We can safely
do this because all callers have higher layer allocation loops that
already handle short allocations, and so will simply run another
allocation to cover the remainder of the requested allocation range
that we ignored during alignment. The advantage of this approach is
that it also removes the need for callers to do anything other than
limit their requests to MAXEXTLEN - they don't really need to be
aware of extent size hints at all.
Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Brian Foster [Fri, 31 Jul 2015 01:03:56 +0000 (11:03 +1000)]
repair: helper to transition inode blocks to inode state
The state of each block in an inode chunk transitions from free state to
inode state as we process physical inodes on disk. We take care to
detect invalid transitions and warn the user if multiply claimed blocks
are detected.
This block of code is a largish switch statement that is executed twice
due to the implementation details of the inode processing loop. Factor
it into a new helper.
Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Brian Foster [Fri, 31 Jul 2015 01:03:53 +0000 (11:03 +1000)]
repair: helper to import on-disk inobt records to in-core trees
In the common case, the in-core inode state from the on-disk inobt
records is imported from the inobt and validated against the finobt (if
one exists). When both trees exist along with some form of corruption,
it's possible to find inodes in the finobt not tracked by the inobt.
While this is unexpected, we attempt to repair by importing the inodes
from the finobt.
The associated code in the finobt scan function mirrors the associated
code in the inobt scan function. Factor this into a separate helper that
can be called by either tree scan.
Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Brian Foster [Fri, 31 Jul 2015 01:03:52 +0000 (11:03 +1000)]
repair: helper for inode chunk alignment and start/end ino number verification
The inobt scan code executes different routines for processing inobt
records and finobt records. While some verification differs between the
trees, much of it is the same. One such example of this is the inode
record alignment and start/end inode number verification. The only
difference between the inobt and finobt verification is the error
message that is generated as a result of failure.
Factor out these alignment checks into a new helper that takes an enum
parameter that identifies which tree is undergoing the scan. Use a new
string array for this function and subsequent common inobt scan helpers
to convert the enum to the name of the tree for the purposes of
including in any resulting warning messages.
Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Brian Foster [Fri, 31 Jul 2015 01:03:51 +0000 (11:03 +1000)]
repair: access helpers for on-disk inobt record freecount
The on-disk inobt record has two formats depending on whether sparse
inode support is enabled or not. If so, the freecount field is a single
byte and does not require byte-conversion. Otherwise, it is a 4-byte
field and does.
Create the inorec_[get|set]_freecount() helpers to abstract this detail
away from the core repair code.
Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Brian Foster [Fri, 31 Jul 2015 01:03:49 +0000 (11:03 +1000)]
metadump: support sparse inode records
xfs_metadump currently uses mp->m_ialloc_blks sized buffers to copy
inode chunks. If a filesystem supports sparse inodes, some clusters
within inode chunks can point to arbitrary data. If the buffer used to
read inodes includes these sparse clusters, inode read verification
fails and prints filesystem corruption warnings.
Update copy_inode_chunks() to support using a cluster sized buffer to
read a full inode chunk in multiple iterations if sparse inodes is
enabled. For each cluster read, check whether the first inode in the
cluster is sparse and skip the cluster if so. This is safe because
sparse records are allocated at cluster granularity.
Signed-off-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
projects / thirdparty / xfsprogs-dev.git / log
