Darrick J. Wong [Tue, 25 Oct 2016 22:14:31 +0000 (15:14 -0700)]
libxfs: add fsxattr flags and fields for cowextsize
Add the cowextsize field and flag to each platform's struct fsxattr
definitions. We can compile these definitions into the xfsprogs
utilities if we don't pick them up from the system headers, such as on
kernels prior to 4.9.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
If the deferred ops transaction roll fails, we need to abort the intent
items if we haven't already logged a done item for it, regardless of
whether or not the deferred ops has had a transaction committed. Dave
found this while running generic/388.
Move the tracepoint to make it easier to track object lifetimes.
Reported-by: Dave Chinner <david@fromorbit.com> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Split out two helpers for deleting delayed or real extents from the COW fork.
This allows to call them directly from xfs_reflink_cow_end_io once that
function is refactored to iterate the extent tree. It will also allow
to reuse the delalloc deletion from xfs_bunmapi in the future.
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
This helpers allows to trim an extent to a subset of it's original range
while making sure the block numbers in it remain valid,
In the future xfs_trim_extent and xfs_bmapi_trim_map should probably be
merged in some form.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
[hch: split from a previous patch from Darrick, moved around and added
support for "raw" delayed extents"] Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
The function xfs_calc_dquots_per_chunk takes a parameter in units
of basic blocks. The kernel seems to get the units wrong, but
userspace got 'fixed' by commenting out the unnecessary conversion.
Fix both.
cc: <stable@vger.kernel.org> Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Eric Sandeen <sandeen@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
f65306ea xfs: map an inode's offset to an exact physical block
added a pointless error0: target; remove it.
Addresses-Coverity-Id: 1373865 Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Bill O'Donnell <billodo@redhat.com> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Remove redundant ifp = ifp statement, it does nothing. Found with
static analysis by CoverityScan.
Signed-off-by: Colin Ian King <colin.king@canonical.com> Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
The error handling in xfs_refcount_recover_cow_leftovers is confused
and can potentially leak memory, so rework it to release resources
correctly on error.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reported-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Implement swapext for filesystems that have reverse mapping. Back in
the reflink patches, we augmented the bmap code with a 'REMAP' flag
that updates only the bmbt and doesn't touch the allocator and
implemented log redo items for those two operations. Now we can
rewrite extent swapping as a (looong) series of remap operations.
This is far less efficient than the fork swapping method implemented
in the past, so we only switch this on for rmap.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Create an error injection point that enables us to simulate being
critically low on per-AG block reservations. This should enable us to
simulate this specific ENOSPC condition so that we can test falling back
to a regular file copy.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
When it's possible for reverse mappings to overlap (data fork extents
of files on reflink filesystems), use the interval query function to
find the left neighbor of an extent we're trying to add; and be
careful to use the lookup functions to update the neighbors and/or
add new extents.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Prior to the introduction of reflink, allocating a block and mapping
it into a file was performed in a single transaction with a single
block reservation, and the allocator was supposed to find enough
blocks to allocate the extent and any BMBT blocks that might be
necessary (unless we're low on space).
However, due to the way copy on write works, allocation and mapping
have been split into two transactions, which means that we must be
able to handle the case where we allocate an extent for CoW but that
AG runs out of free space before the blocks can be mapped into a file,
and the mapping requires a new BMBT block. When this happens, look in
one of the other AGs for a BMBT block instead of taking the FS down.
The same applies to the functions that convert a data fork to extents
and later btree format.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
To gracefully handle the situation where a CoW operation turns a
single refcount extent into a lot of tiny ones and then run out of
space when a tree split has to happen, use the per-AG reserved block
pool to pre-allocate all the space we'll ever need for a maximal
btree. For a 4K block size, this only costs an overhead of 0.3% of
available disk space.
When reflink is enabled, we have an unfortunate problem with rmap --
since we can share a block billions of times, this means that the
reverse mapping btree can expand basically infinitely. When an AG is
so full that there are no free blocks with which to expand the rmapbt,
the filesystem will shut down hard.
This is rather annoying to the user, so use the AG reservation code to
reserve a "reasonable" amount of space for rmap. We'll prevent
reflinks and CoW operations if we think we're getting close to
exhausting an AG's free space rather than shutting down, but this
permanent reservation should be enough for "most" users. Hopefully.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
[hch@lst.de: ensure that we invalidate the freed btree buffer] Signed-off-by: Christoph Hellwig <hch@lst.de>
Create a per-inode extent size allocator hint for copy-on-write. This
hint is separate from the existing extent size hint so that CoW can
take advantage of the fragmentation-reducing properties of extent size
hints without disabling delalloc for regular writes.
The extent size hint that's fed to the allocator during a copy on
write operation is the greater of the cowextsize and regular extsize
hint.
During reflink, if we're sharing the entire source file to the entire
destination file and the destination file doesn't already have a
cowextsize hint, propagate the source file's cowextsize hint to the
destination file.
Furthermore, zero the bulkstat buffer prior to setting the fields
so that we don't copy kernel memory contents into userspace.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Due to the way the CoW algorithm in XFS works, there's an interval
during which blocks allocated to handle a CoW can be lost -- if the FS
goes down after the blocks are allocated but before the block
remapping takes place. This is exacerbated by the cowextsz hint --
allocated reservations can sit around for a while, waiting to get
used.
Since the refcount btree doesn't normally store records with refcount
of 1, we can use it to record these in-progress extents. In-progress
blocks cannot be shared because they're not user-visible, so there
shouldn't be any conflicts with other programs. This is a better
solution than holding EFIs during writeback because (a) EFIs can't be
relogged currently, (b) even if they could, EFIs are bound by
available log space, which puts an unnecessary upper bound on how much
CoW we can have in flight, and (c) we already have a mechanism to
track blocks.
At mount time, read the refcount records and free anything we find
with a refcount of 1 because those were in-progress when the FS went
down.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Create a helper method to remove extents from the CoW fork without
any of the side effects (rmapbt/bmbt updates) of the regular extent
deletion routine. We'll eventually use this to clear out the CoW fork
during ioend processing.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Modify xfs_bmap_add_extent_delay_real() so that we can convert delayed
allocation extents in the CoW fork to real allocations, and wire this
up all the way back to xfs_iomap_write_allocate(). In a subsequent
patch, we'll modify the writepage handler to call this.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Allow the creation of delayed allocation extents in the CoW fork. In
a subsequent patch we'll wire up iomap_begin to actually do this via
reflink helper functions.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Return the range of file blocks that bunmapi didn't free. This hint
is used by CoW and reflink to figure out what part of an extent
actually got freed so that it can set up the appropriate atomic
remapping of just the freed range.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Pass BMAPI_ flags from bunmapi into bmap_del_extent and extend
BMAPI_REMAP (which means "don't touch the allocator or the quota
accounting") to apply to bunmapi as well. This will be used to
implement the unmap operation, which will be used by swapext.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Teach the bmap routine to know how to map a range of file blocks to a
specific range of physical blocks, instead of simply allocating fresh
blocks. This enables reflink to map a file to blocks that are already
in use.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Provide a mechanism for higher levels to create BUI/BUD items, submit
them to the log, and a stub function to deal with recovered BUI items.
These parts will be connected to the rmapbt in a later patch.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Create bmbt update intent/done log items to record redo information in
the log. Because we roll transactions multiple times for reflink
operations, we also have to track the status of the metadata updates
that will be recorded in the post-roll transactions in case we crash
before committing the final transaction. This mechanism enables log
recovery to finish what was already started.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Provide a mechanism for higher levels to create CUI/CUD items, submit
them to the log, and a stub function to deal with recovered CUI items.
These parts will be connected to the refcountbt in a later patch.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Create refcount update intent/done log items to record redo
information in the log. Because we need to roll transactions between
updating the bmbt mapping and updating the reverse mapping, we also
have to track the status of the metadata updates that will be recorded
in the post-roll transactions, just in case we crash before committing
the final transaction. This mechanism enables log recovery to finish
what was already started.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
Implement the generic btree operations required to manipulate refcount
btree blocks. The implementation is similar to the bmapbt, though it
will only allocate and free blocks from the AG.
Since the refcount root and level fields are separate from the
existing roots and levels array, they need a separate logging flag.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
[hch: fix logging of AGF refcount btree fields] Signed-off-by: Christoph Hellwig <hch@lst.de>
Every time we allocate or free a data extent, we might need to split
the refcount btree. Reserve some blocks in the transaction to handle
this possibility. Even though the deferred refcount code can roll a
transaction to avoid overloading the transaction, we can still exceed
the reservation.
Certain pathological workloads (1k blocks, no cowextsize hint, random
directio writes), cause a perfect storm wherein a refcount adjustment
of a large range of blocks causes full tree splits in two separate
extents in two separate refcount tree blocks; allocating new refcount
tree blocks causes rmap btree splits; and all the allocation activity
causes the freespace btrees to split, blowing the reservation.
(Reproduced by generic/167 over NFS atop XFS)
Signed-off-by: Christoph Hellwig <hch@lst.de>
[darrick.wong@oracle.com: add commit message] Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Start constructing the refcount btree implementation by establishing
the on-disk format and everything needed to read, write, and
manipulate the refcount btree blocks.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Christoph Hellwig <hch@lst.de>
Since XFS reserves a small amount of space in each AG as the minimum
free space needed for an operation, save some more space in case we
touch the refcount btree.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de>
When adding a new remote attribute, we write the attribute to the
new extent before the allocation transaction is committed. This
means we cannot reuse busy extents as that violates crash
consistency semantics. Hence we currently treat remote attribute
extent allocation like userdata because it has the same overwrite
ordering constraints as userdata.
Unfortunately, this also allows the allocator to incorrectly apply
extent size hints to the remote attribute extent allocation. This
results in interesting failures, such as transaction block
reservation overruns and in-memory inode attribute fork corruption.
To fix this, we need to separate the busy extent reuse configuration
from the userdata configuration. This changes the definition of
XFS_BMAPI_METADATA slightly - it now means that allocation is
metadata and reuse of busy extents is acceptible due to the metadata
ordering semantics of the journal. If this flag is not set, it
means the allocation is that has unordered data writeback, and hence
busy extent reuse is not allowed. It no longer implies the
allocation is for user data, just that the data write will not be
strictly ordered. This matches the semantics for both user data
and remote attribute block allocation.
As such, This patch changes the "userdata" field to a "datatype"
field, and adds a "no busy reuse" flag to the field.
When we detect an unordered data extent allocation, we immediately set
the no reuse flag. We then set the "user data" flags based on the
inode fork we are allocating the extent to. Hence we only set
userdata flags on data fork allocations now and consider attribute
fork remote extents to be an unordered metadata extent.
The result is that remote attribute extents now have the expected
allocation semantics, and the data fork allocation behaviour is
completely unchanged.
It should be noted that there may be other ways to fix this (e.g.
use ordered metadata buffers for the remote attribute extent data
write) but they are more invasive and difficult to validate both
from a design and implementation POV. Hence this patch takes the
simple, obvious route to fixing the problem...
Reported-and-tested-by: Ross Zwisler <ross.zwisler@linux.intel.com> Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Currently xfs_iomap_write_delay does up to lookups in the inode
extent tree, which is rather costly especially with the new iomap
based write path and small write sizes.
But it turns out that the low-level xfs_bmap_search_extents gives us
all the information we need in the regular delalloc buffered write
path:
- it will return us an extent covering the block we are looking up
if it exists. In that case we can simply return that extent to
the caller and are done
- it will tell us if we are beyoned the last current allocated
block with an eof return parameter. In that case we can create a
delalloc reservation and use the also returned information about
the last extent in the file as the hint to size our delalloc
reservation.
- it can tell us that we are writing into a hole, but that there is
an extent beyoned this hole. In this case we can create a
delalloc reservation that covers the requested size (possible
capped to the next existing allocation).
All that can be done in one single routine instead of bouncing up
and down a few layers. This reduced the CPU overhead of the block
mapping routines and also simplified the code a lot.
Signed-off-by: Christoph Hellwig <hch@lst.de> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
One unfortunate quirk of the reference count and reverse mapping
btrees -- they can expand in size when blocks are written to *other*
allocation groups if, say, one large extent becomes a lot of tiny
extents. Since we don't want to start throwing errors in the middle
of CoWing, we need to reserve some blocks to handle future expansion.
The transaction block reservation counters aren't sufficient here
because we have to have a reserve of blocks in every AG, not just
somewhere in the filesystem.
Therefore, create two per-AG block reservation pools. One feeds the
AGFL so that rmapbt expansion always succeeds, and the other feeds all
other metadata so that refcountbt expansion never fails.
Use the count of how many reserved blocks we need to have on hand to
create a virtual reservation in the AG. Through selective clamping of
the maximum length of allocation requests and of the length of the
longest free extent, we can make it look like there's less free space
in the AG unless the reservation owner is asking for blocks.
In other words, play some accounting tricks in-core to make sure that
we always have blocks available. On the plus side, there's nothing to
clean up if we crash, which is contrast to the strategy that the rough
draft used (actually removing extents from the freespace btrees).
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
new file mode 100644
index 000000000000..e3ae0f2b4294
When xfs_defer_finish calls ->finish_item, it's possible that
(refcount) won't be able to finish all the work in a single
transaction. When this happens, the ->finish_item handler should
shorten the log done item's list count, update the work item to
reflect where work should continue, and return -EAGAIN so that
defer_finish knows to retain the pending item on the pending list,
roll the transaction, and restart processing where we left off.
Plumb in the code and document how this mechanism is supposed to work.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Provide a helper method to count the number of blocks in a short form
btree. The refcount and rmap btrees need to know the number of blocks
already in use to set up their per-AG block reservations during mount.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Use variable length array declarations for RUI log items,
and replace the open coded sizeof formulae with a single function.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Christoph reports slab corruption when a deferred refcount update
aborts during _defer_finish(). The cause of this was broken log item
state tracking in xfs_defer_pending -- upon an abort,
_defer_trans_abort() will call abort_intent on all intent items,
including the ones that have already had a done item attached.
This is incorrect because each intent item has 2 refcount: the first
is released when the intent item is committed to the log; and the
second is released when the _done_ item is committed to the log, or
by the intent creator if there is no done item. In other words, once
we log the done item, responsibility for releasing the intent item's
second refcount is transferred to the done item and /must not/ be
performed by anything else.
The dfp_committed flag should have been tracking whether or not we had
a done item so that _defer_trans_abort could decide if it needs to
abort the intent item, but due to a thinko this was not the case. Rip
it out and track the done item directly so that we do the right thing
w.r.t. intent item freeing.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reported-by: Christoph Hellwig <hch@infradead.org> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
From inspection, the superblock sb_inprogress check is done in the
verifier and triggered only for the primary superblock via a
"bp->b_bn == XFS_SB_DADDR" check.
Unfortunately, the primary superblock is an uncached buffer, and
hence it is configured by xfs_buf_read_uncached() with:
bp->b_bn = XFS_BUF_DADDR_NULL; /* always null for uncached buffers */
And so this check never triggers. Fix it.
cc: <stable@vger.kernel.org> Signed-off-by: Dave Chinner <dchinner@redhat.com> Reviewed-by: Brian Foster <bfoster@redhat.com> Reviewed-by: Christoph Hellwig <hch@lst.de> Signed-off-by: Dave Chinner <david@fromorbit.com>
Because we are mangling the diff source/destination locations, we
have to add prefixes to them to get them to apply cleanly as -p1
patches. This is all fine until we create or remove a file and
the the src/dest is /dev/null. Applying a prefix here causes
the diff to be malformed and it won't apply.
Add another hack to work around this limitation of filterdiff when
reformatting the diff into readable format.
Dave Chinner [Mon, 24 Oct 2016 22:04:33 +0000 (09:04 +1100)]
libxfs_apply: filter commits from libxfs only
When pulling commits from the kernel, it's easy to specify a commit
range such as "v4.8..for-next" to indicate we want to pull all
commits for libxfs since the 4.8 kernel release. Unfortunately,
this pull commits from all over the kernel tree, not just
fs/xfs/libxfs.
Filter the commit list retrieval to limit the commits to those touch
fs/xfs/libxfs so that we only attempt to apply the realtively small
number of relevant commits.
Eric Sandeen [Sun, 2 Oct 2016 23:56:00 +0000 (10:56 +1100)]
xfs_io: fix inode command with "-n" for bogus inode
If we ask for the next allocated inode after a number for which
no other inode exists, the bulkstat returns success, but with
count == 0. If we ignore this fact, we print a garbage result
from bstat.bs_ino in this case, so fix it.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Sun, 2 Oct 2016 23:50:21 +0000 (10:50 +1100)]
xfs_io: refactor inode command
The inode_f function is a bit convoluted; the default
find-last-inode case appears at the end, there are several return
points, we print the same basic information using 2 different
variables in 2 different locations depending on the mode we're in,
the "inode not found" was a printf & exit in the middle of the
function, etc.
Move the default case up to the top so it's more obvious, not
buried.
Make a new var, result_ino, which holds whatever we want to print
regardless of the mode, and then handle all the output at the end.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Sun, 2 Oct 2016 23:47:47 +0000 (10:47 +1100)]
xfs_io: move inode command arg handling to top
As it stands, collecting the inode number and testing args validity
is all tangled up; for example the test for "-n" having no inode is
buried in an else after a large code block which handles something
else.
Get inode number argument collection and testing out of the way
before doing anything else.
Clean up the error message if a non-numeric inode arg is given.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Sun, 2 Oct 2016 23:46:03 +0000 (10:46 +1100)]
xfs_io: factor out new get_last_inode() helper
The inode command by default finds the last allocated inode in the
filesystem via bulkstat, and this specific function is open-coded
after other cases are handled, leading to a fairly long inode_f
function and confusing code flow.
Clean it up by factoring it into a new function, more refactoring
will follow.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Sun, 2 Oct 2016 23:42:22 +0000 (10:42 +1100)]
xfs_repair: add freesp btree block overflow to the free space
If we overestimate the number of blocks needed to rebuild the free
space btrees to the point that we have more blocks than fit in the
AGFL, save those blocks and reinsert them into the free space at
the end of phase 5. Previously, the overflow blocks would simply
be lost.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Sun, 2 Oct 2016 23:40:32 +0000 (10:40 +1100)]
xfs_repair: fix bogosity when rmapping new AGFL blocks
When repair rebuilds the AGFL, the blocks can come either from the
in-core free space tree or they can come as a result of overestimating
the number of blocks needed to rebuild the on-disk free space btree.
The code in here was trying to only create rmap records for AGFL blocks
that did /not/ come from free space btree rebuild overestimation, but
was totally broken. The initial and check conditions were totally wrong
if there was any overflow. Remove a stray debug printf too.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eryu Guan [Sun, 2 Oct 2016 23:36:14 +0000 (10:36 +1100)]
libxcmd: fix counting of xfs entries in fs_table_insert
Commit bb80e3d6cd04 ("libxcmd: populate fs table with xfs entries
first, foreign entries last") adds a new counter "xfs_fs_count" and
increases the counter when inserting an XFS entry.
But it missed a counter when fs_count is zero (inserting the first
path) and the entry has no FS_FOREIGN bit set, i.e. the first XFS
entry doesn't increase xfs_fs_count.
This results in args_command() mess and infinite loop in xfs/244
when testing v4 XFS (xfs/244 notrun on v5 XFS, but this bug still
reproduces on v5 XFS). e.g.
Eric Sandeen [Thu, 22 Sep 2016 23:16:52 +0000 (09:16 +1000)]
xfs_copy: Fix meta UUID handling on multiple copies
Zorro reported that when making multiple copies of a V5
filesystem with xfs_copy while generating new UUIDs, all
but the first copy were corrupt.
Upon inspection, the corruption was related to incorrect UUIDs;
the original UUID, as stamped into every metadata structure,
was not preserved in the sb_meta_uuid field of the superblock
on any but the first copy.
This happened because sb_update_uuid was using the UUID present in
the ag_hdr structure as the unchanging meta-uuid which is to match
existing structures, but it also /updates/ that UUID with the
new identifying UUID present in tcarg. So the newly-generated
UUIDs moved transitively from tcarg->uuid to ag_hdr->xfs_sb->sb_uuid
to ag_hdr->xfs_sb->sb_meta_uuid each time the function got called.
Fix this by looking instead to the unchanging, original UUID
present in the xfs_sb_t we are given, which reflects the original
filesystem's metadata UUID, and copy /that/ UUID into each target
filesystem's meta_uuid field.
Most of this patch is changing comments and re-ordering tests
to match; the functional change is to simply use the *sb rather
than the *ag_hdr to identify the proper metadata UUID.
Reported-and-tested-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Thu, 22 Sep 2016 23:16:47 +0000 (09:16 +1000)]
xfs_repair: fix segfault from uninitialized tp in mv_orphanage
After 9074815 xfs: better xfs_trans_alloc interface, mv_orphanage
was passing an uninitialized *tp into libxfs_dir_lookup, because
the trans_alloc() call which was present prior to the call got
removed in that commit.
This ultimately led to testing an uninit tp var:
Conditional jump or move depends on uninitialised value(s)
at 0x434D01: libxfs_trans_read_buf_map (trans.c:554)
by 0x45152E: libxfs_da_read_buf (xfs_da_btree.c:2610)
by 0x456ACB: xfs_dir3_block_read (xfs_dir2_block.c:136)
by 0x4570A8: xfs_dir2_block_lookup_int (xfs_dir2_block.c:675)
by 0x457DB7: xfs_dir2_block_lookup (xfs_dir2_block.c:623)
by 0x455F54: libxfs_dir_lookup (xfs_dir2.c:399)
by 0x421C46: mv_orphanage (phase6.c:1095)
by 0x4222C2: check_for_orphaned_inodes (phase6.c:3108)
by 0x423ABD: phase6 (phase6.c:3287)
by 0x42E4B2: main (xfs_repair.c:933)
and ended with a segfault as we tried to use that tp when
searching for the buffer in xfs_trans_buf_item_match():
I think simply passing in NULL for this tp is sufficient to fix
this; we'll just go read the buffer from disk in
libxfs_trans_read_buf_map rather than trying to find it in an
existing transaction.
Reported-by: Consigliere <admin@russenmafia.at> Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 19 Sep 2016 22:49:41 +0000 (08:49 +1000)]
libxfs: factor mount checks into helper function
platform_check_ismounted switched to a getmntent() loop after
ustat disappeared on some new platforms.
We also use a similar mechanism for determining the
ro/rw-mounted status of a device in platform_check_iswritable.
Because the loops are essentially the same, factor them into a
single helper which accepts a VERBOSE flag to print info if the
device is found in the checked-for state, and a WRITABLE flag
which only checks specifically for a mounted and /writable/ device.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 19 Sep 2016 22:48:54 +0000 (08:48 +1000)]
mkfs.xfs: clarify ftype defaults in manpage
When CRCs were made default, a few leftovers related to its
prior non-default status remained in the manpage, in the ftype
section. Clean those up, stating the correct default for this
feature.
Reported-by: Chris Murphy <chris@cmurf.com> Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 19 Sep 2016 06:07:55 +0000 (16:07 +1000)]
xfs_io: allow chattr & chproj on foreign filesystems
Now that FS_IOC_FSSETXATTR is a generic vfs call, these
functions can be used on non-xfs filesystems, and this is
needed for generic project quota testing.
Eric Sandeen [Mon, 19 Sep 2016 06:06:57 +0000 (16:06 +1000)]
xfs_quota: un-flag non-foreign-capable commands
The off command calls XFS_QUOTAOFF / Q_XQUOTAOFF, which calls
quota_disable in the kernel, which returns ENOSYS if the
->quota_enable quota op doesn't exist - and it does not exist
on any non-xfs filesystems.
We could get clever if we wanted it, and send Q_QUOTAOFF
instead for foreign filesystems, but for now it's broken
so just remove the flag.
The free command relies on XFS_IOC_FSGEOMETRY_V1, so unflag it
as well.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Bill O'Donnell <billodo@redhat.com Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 19 Sep 2016 06:06:36 +0000 (16:06 +1000)]
xfs_quota: Enable 3 more foreign commands
Enable restore, limit, and timer.
Unsupported commands remain, for lack of kernel support, generally:
warn, quot,, enable, disable, and remove.
xfs_quota> report
User quota on /mnt/test2/git/xfsprogs/mnt (/dev/loop0)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 13 0 0 00 [--------]
xfs_quota> restore -f quotadump
xfs_quota> report
User quota on /mnt/test2/git/xfsprogs/mnt (/dev/loop0)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 13 0 0 00 [--------]
testuser 0 16384 32768 00 [--------]
fsgqa 0 102400 112640 00 [--------]
xfs_quota> limit bsoft=200m fsgqa
xfs_quota> report
User quota on /mnt/test2/git/xfsprogs/mnt (/dev/loop0)
Blocks
User ID Used Soft Hard Warn/Grace
---------- --------------------------------------------------
root 13 0 0 00 [--------]
testuser 0 16384 32768 00 [--------]
fsgqa 0 204800 112640 00 [--------]
xfs_quota> state -u
User quota state on /mnt/test2/git/xfsprogs/mnt (/dev/loop0)
Accounting: ON
Enforcement: ON
Inode: #12 (16 blocks, 1 extents)
Blocks grace time: [7 days]
Inodes grace time: [7 days]
xfs_quota> timer -b 3days
xfs_quota> state -u
User quota state on /mnt/test2/git/xfsprogs/mnt (/dev/loop0)
Accounting: ON
Enforcement: ON
Inode: #12 (16 blocks, 1 extents)
Blocks grace time: [3 days]
Inodes grace time: [7 days]
Realtime Blocks grace time: [--------]
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Bill O'Donnell <billodo@redhat.com Signed-off-by: Dave Chinner <david@fromorbit.com>
Bill O'Donnell [Mon, 19 Sep 2016 06:05:45 +0000 (16:05 +1000)]
xfs_quota: add case for foreign fs, disabled regardless of foreign_allowed
Some commands are disallowed for foreign filesystems,
regardless of whether or not the -f flag is thrown.
Add a case for this condition and improve commenting
and output messaging accordingly in init_check_command.
Signed-off-by: Bill O'Donnell <billodo@redhat.com> Reviewed-by: Eric Sandeen <sandeen@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Bill O'Donnell [Mon, 19 Sep 2016 06:02:41 +0000 (16:02 +1000)]
xfs_quota: print and path output formatting: maintain reverse compatibility
This patch adjusts the formatting of the xfs_quota print and
path outputs, in order to maintain reverse compatability:
when -f flag isn't used, need to keep the output same as in
previous version.
Signed-off-by: Bill O'Donnell <billodo@redhat.com> Reviewed-by: Eric Sandeen <sandeen@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Bill O'Donnell [Mon, 19 Sep 2016 06:02:22 +0000 (16:02 +1000)]
libxcmd: populate fs table with xfs entries first, foreign entries last
Commits b20b6c2 and 29647c8 modified xfs_quota for use on
non-XFS filesystems. Modifications to fs_initialise_mounts
(paths.c) resulted in an xfstest fail (xfs/261), due to foreign
fs paths being picked up first from the fs table. The xfs_quota
print command then complained about not being able to print the
foreign paths, instead of previous behavior (quiet).
This patch restores correct behavior, sorting the table so that
xfs entries are first, followed by foreign fs entries. The patch
maintains the order of xfs entries and foreign entries in the
same order as mtab entries. Then, in functions which print all
paths we can simply break at the first foreign path if the -f
switch is not specified.
Signed-off-by: Bill O'Donnell <billodo@redhat.com> Reviewed-by: Eric Sandeen <sandeen@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 19 Sep 2016 06:01:14 +0000 (16:01 +1000)]
xfs_repair: exit with status 2 if log dirtiness is unknown
This new case is mostly like the known dirty log case; the log
is corrupt, dirtiness cannot be determined, and a mount/umount
cycle or an xfs_repair -L is required.
So exit with status 2 here as well.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Zorro Lang <zlang@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Hou Tao [Mon, 19 Sep 2016 06:00:04 +0000 (16:00 +1000)]
xfs_logprint: remove the printing of transaction type
THe kernel stopped using meaningful types in transaction headers
when delayed logging was introduced. Since then the only transaction
type that reaches the journal is a "Checkpoint" type. Since then,
we've effectivey broken the transaction type printing for newer
kernels, and the current kernels don't even have transaction types
internally. Hence this logprint function is stale, broken, and
causing us problems.
This patch removes the transaction type parsing. If a user needs
this information from logprint, we can still build a binary from a
prior version that correctly decoded the transaction type (e.g.
3.2.1) for that purpose.
Signed-off-by: Hou Tao <houtao1@huawei.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Eric Sandeen [Mon, 19 Sep 2016 05:53:52 +0000 (15:53 +1000)]
libxfs: move iswritable "fatal" decision to caller
Simplify platform_check_iswritable by moving the
"fatal" decision up to the (one) caller. In other words,
simply return whether mounted+writable is true, and
return 1 if so. Caller decides what to do with that info
based on /its/ "fatal" argument.
Signed-off-by: Eric Sandeen <sandeen@redhat.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Dave Chinner [Thu, 8 Sep 2016 22:16:29 +0000 (08:16 +1000)]
xfsprogs: Release 4.8.0-rc1
Update all the necessary files for a 4.8.0-rc1 release. Also,
replace all the mailing list contact details with the new list
address, and the project website with xfs.org.
Darrick J. Wong [Thu, 8 Sep 2016 00:22:28 +0000 (10:22 +1000)]
xfs_db: pass the inode cluster offset when copying inodes
In copy_inode_chunk, we try to determine whether or not an inode is
free as part of copying the inode records. The macros involved in
testing ir_free require both the inode record and the offset of an
inode within that chunk. Prior to sparse inode support, the loop
index "i" was also the inode chunk offset; however, when sparse
support was added, "i" became the inode offset within a cluster and
"ioff" became the inode cluster offset within an inode chunk.
Therefore, it is necessary to pass "ioff + i" to do the free-ness
calculation correctly.
This was discovered while trying to take metadumps of fs images for
scrub testing.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Darrick J. Wong [Thu, 8 Sep 2016 00:22:28 +0000 (10:22 +1000)]
xfs_repair: don't crash on ENOSPC rebuilding a btree
During btree rebuilding, the cursor setup function checks ext_ptr to
report ENOSPC problems when it grabs the first extent for the btree.
However, subsequent grabs for free space don't check ext_ptr and so we
segfault if there's no space. Therefore, move the ENOSPC check into
the loop so that we always complain about insufficient space instead
of just crashing.
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com> Reviewed-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
Felix Janda [Thu, 8 Sep 2016 00:22:28 +0000 (10:22 +1000)]
libxfs/linux.c: Replace use of ustat by stat
ustat has been used to check whether a device file is mounted.
The function is deprecated and not supported by uclibc and musl.
Now do the check using the *mntent functions.
Based on patch by Natanael Copa <ncopa@alpinelinux.org>.
Signed-off-by: Felix Janda <felix.janda@posteo.de> Reviewed-by: Eric Sandeen <sandeen@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
doesn't write to array index 32 - it incorrectly writes to
/somewhere/ in the entire array range.
The issue here is that the write_struct() code assumes that the
object it is printing always a structure member and any array
indexes will be exposed as children of the parent type. This works
just fine for structures with internal arrays, but when the type
being decoded is an array, we get a direct reference to the offset
to be written in the parent object.
Hence we need to take into account the array index returned by the
parent object parsing when calculating the size of the region to be
modified rather than using fcount() as that results in the size
always being set to the size of the entire array and the
modification being written to the wrong place.
Signed-off-by: Dave Chinner <dchinner@redhat.com> Signed-off-by: Dave Chinner <david@fromorbit.com>
added a "dquot_buf" to the iocur to specify when we were operating
on a dquot and thus handle dquot CRC updates - but nothing ever
actually set dquot_buf to a non-zero value.
Without doing so, we don't recalculate the dquot crc when
changing contents of a dquot: