]> git.ipfire.org Git - thirdparty/xfsprogs-dev.git/log
thirdparty/xfsprogs-dev.git
8 years agoxfs_repair: check for mergeable refcount records
Darrick J. Wong [Tue, 25 Oct 2016 22:14:36 +0000 (15:14 -0700)] 
xfs_repair: check for mergeable refcount records

Make sure there aren't adjacent refcount records that could be merged;
this is a sign that the refcount tree algorithms aren't working
correctly.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_repair: use range query when while checking rmaps
Darrick J. Wong [Tue, 25 Oct 2016 22:14:35 +0000 (15:14 -0700)] 
xfs_repair: use range query when while checking rmaps

For shared extents, we ought to use a range query on the rmapbt to
find the corresponding rmap.  However, most of the time the observed
rmap will be an exact match for the rmapbt rmap, in which case we
could have used the (much faster) regular lookup.  Therefore, try the
regular lookup first and resort to the range lookup if that doesn't
get us what we want.  This can cut the run time of the rmap check of
xfs_repair in half.

Theoretically, the only reason why an observed rmap wouldn't be an
exact match for an rmapbt rmap is because we modified some file on
account of a metadata error.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_repair: check the CoW extent size hint
Darrick J. Wong [Tue, 25 Oct 2016 22:14:35 +0000 (15:14 -0700)] 
xfs_repair: check the CoW extent size hint

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_repair: complain about copy-on-write leftovers
Darrick J. Wong [Tue, 25 Oct 2016 22:14:35 +0000 (15:14 -0700)] 
xfs_repair: complain about copy-on-write leftovers

Complain about leftover CoW allocations that are hanging off the
refcount btree.  These are cleaned out at mount time, but we could be
louder about flagging down evidence of trouble.

Since these extents aren't "owned" by anything, we'll free them up by
reconstructing the free space btrees.

v2: When we're processing rmap records, we inadvertently forgot to
handle the CoW owner, so the leftover CoW staging blocks got marked as
file data.  These blocks will just get freed later, so mark them
"CoW".  When we process the refcountbt, complain about leftovers if
the type is unknown or "CoW".

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_repair: rebuild the refcount btree
Darrick J. Wong [Tue, 25 Oct 2016 22:14:35 +0000 (15:14 -0700)] 
xfs_repair: rebuild the refcount btree

Rebuild the refcount btree with the reference count data we assembled
during phase 4.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_repair: check the refcount btree against our observed reference counts when -n
Darrick J. Wong [Tue, 25 Oct 2016 22:14:35 +0000 (15:14 -0700)] 
xfs_repair: check the refcount btree against our observed reference counts when -n

Check the observed reference counts against whatever's in the refcount
btree for discrepancies.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_repair: fix inode reflink flags
Darrick J. Wong [Tue, 25 Oct 2016 22:14:35 +0000 (15:14 -0700)] 
xfs_repair: fix inode reflink flags

While we're computing reference counts, record which inodes actually
share blocks with other files and fix the flags as necessary.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_repair: record reflink inode state
Darrick J. Wong [Tue, 25 Oct 2016 22:14:34 +0000 (15:14 -0700)] 
xfs_repair: record reflink inode state

Record the state of the per-inode reflink flag, so that we can
compare against the rmap data and update the flags accordingly.
Clear the (reflink) state if we clear the inode.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_repair: process reverse-mapping data into refcount data
Darrick J. Wong [Tue, 25 Oct 2016 22:14:34 +0000 (15:14 -0700)] 
xfs_repair: process reverse-mapping data into refcount data

Take all the reverse-mapping data we've acquired and use it to generate
reference count data.  This data is used in phase 5 to rebuild the
refcount btree.

v2: Update to reflect separation of rmap_irec flags.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_repair: handle multiple owners of data blocks
Darrick J. Wong [Tue, 25 Oct 2016 22:14:34 +0000 (15:14 -0700)] 
xfs_repair: handle multiple owners of data blocks

If reflink is enabled, don't freak out if there are multiple owners of
a given block; that's just a sign that each of those owners are
reflink files.

v2: owner and offset are unsigned types, so use those for inorder
comparison.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_repair: check the existing refcount btree
Darrick J. Wong [Tue, 25 Oct 2016 22:14:34 +0000 (15:14 -0700)] 
xfs_repair: check the existing refcount btree

Spot-check the refcount btree for obvious errors, and mark the
refcount btree blocks as such.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_repair: fix get_agino_buf to avoid corrupting inodes
Darrick J. Wong [Tue, 25 Oct 2016 22:14:34 +0000 (15:14 -0700)] 
xfs_repair: fix get_agino_buf to avoid corrupting inodes

The inode buffering code tries to read inodes in units of chunks,
which are the larger of 8K or 1 FSB.  Each chunk gets its own xfs_buf,
which means that get_agino_buf must calculate the disk address of the
chunk and feed that to libxfs_readbuf in order to find the inode data
correctly.  The current code simply grabs the chunk for the start
inode and indexes from that, which corrupts memory because the start
inode and the target inode could be in different inode chunks.  That
causes the assert in rmap.c to blow when we clear the reflink flag.

(Also fix some minor errors in the debugging printfs.)

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoman: document the inode cowextsize flags & fields
Darrick J. Wong [Tue, 25 Oct 2016 22:14:34 +0000 (15:14 -0700)] 
man: document the inode cowextsize flags & fields

Document the new copy-on-write extent size fields and inode flags
available in struct fsxattr.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_logprint: support bmap redo items
Darrick J. Wong [Tue, 25 Oct 2016 22:14:33 +0000 (15:14 -0700)] 
xfs_logprint: support bmap redo items

Print block mapping update redo items.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_logprint: support refcount redo items
Darrick J. Wong [Tue, 25 Oct 2016 22:14:33 +0000 (15:14 -0700)] 
xfs_logprint: support refcount redo items

Print reference count update redo items.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_logprint: support cowextsize reporting in log contents
Darrick J. Wong [Tue, 25 Oct 2016 22:14:33 +0000 (15:14 -0700)] 
xfs_logprint: support cowextsize reporting in log contents

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_io: try to unshare copy-on-write blocks via fallocate
Darrick J. Wong [Tue, 25 Oct 2016 22:14:33 +0000 (15:14 -0700)] 
xfs_io: try to unshare copy-on-write blocks via fallocate

Wire up the "unshare" flag to the xfs_io fallocate command.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_io: provide long-format help for falloc
Darrick J. Wong [Tue, 25 Oct 2016 22:14:33 +0000 (15:14 -0700)] 
xfs_io: provide long-format help for falloc

Provide long-format help for falloc so that users can learn about
the command.

Note for xfstest writers: If you need to check that a particular
fallocate mode works (-c/-i/-p/-u) on a given filesystem, you should
call _require_xfs_io_command with the falloc subcommand directly, (i.e.
_require_xfs_io_command funshare) because the subcommands are
special-cased to actually try the command.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_io: support injecting the 'per-AG reservation critically low' error
Darrick J. Wong [Tue, 25 Oct 2016 22:14:33 +0000 (15:14 -0700)] 
xfs_io: support injecting the 'per-AG reservation critically low' error

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_io: add refcount+bmap error injection types
Darrick J. Wong [Tue, 25 Oct 2016 22:14:33 +0000 (15:14 -0700)] 
xfs_io: add refcount+bmap error injection types

Add refcount and bmap deferred finish to the types of errors we can
inject.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_io: get and set the CoW extent size hint
Darrick J. Wong [Tue, 25 Oct 2016 22:14:32 +0000 (15:14 -0700)] 
xfs_io: get and set the CoW extent size hint

Enable administrators to get or set the CoW extent size hint.
Report the hint when we run stat.  This also requires some
autoconf magic to detect whether or not fsx_cowextsize exists.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
v2: Use the internal fsxattr override to guarantee that the cowextsize
field always exists inside of whatever struct fsxattr is.

8 years agolibxfs: add autoconf mechanism to override system header fsxattr
Darrick J. Wong [Tue, 25 Oct 2016 22:14:32 +0000 (15:14 -0700)] 
libxfs: add autoconf mechanism to override system header fsxattr

By default, libxfs will use the kernel/system headers to define struct
fsxattr.  Unfortunately, this creates a problem for developers who are
writing new features but building xfsprogs on a stable system, because
the stable kernel's headers don't reflect the new feature.  In this
case, we want to be able to use the internal fsxattr definition while
the kernel headers catch up, so provide some configure magic to allow
further patches to force the use of the internal definition.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
---
v2: Remove the user-visible configure option but leave the fsxattr
override ability so that subsequent patches can trigger it if
necessary.

8 years agoxfs_io: bmap should support querying CoW fork, shared blocks
Darrick J. Wong [Tue, 25 Oct 2016 22:14:32 +0000 (15:14 -0700)] 
xfs_io: bmap should support querying CoW fork, shared blocks

Teach the bmap command to report shared and delayed allocation
extents, and to be able to query the CoW fork.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_growfs: report the presence of the reflink feature
Darrick J. Wong [Tue, 25 Oct 2016 22:14:32 +0000 (15:14 -0700)] 
xfs_growfs: report the presence of the reflink feature

Report the presence of the reflink feature in xfs_info.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_db: print one array element per line
Darrick J. Wong [Tue, 25 Oct 2016 22:14:32 +0000 (15:14 -0700)] 
xfs_db: print one array element per line

Print one array element per line so that the debugger output isn't
a gigantic pile of screen snow.

Before (inobt):

xfs_db> p recs
recs[1-55] = [startino,holemask,count,freecount,free]
1:[128,0,64,0,0] 2:[4288,0xff,32,0,0xffffffff] 3:[4352,0,64,0,0]
4:[4416,0,64,10,0x1f0003e000000000] 5:[4480,0,64,17,0xc00e1803c2007840]

After:

xfs_db> p recs
recs[1-55] = [startino,holemask,count,freecount,free]
1:[128,0,64,0,0]
2:[4288,0xff,32,0,0xffffffff]
3:[4352,0,64,0,0]
4:[4416,0,64,10,0x1f0003e000000000]
5:[4480,0,64,17,0xc00e1803c2007840]

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_db: deal with the CoW extent size hint
Darrick J. Wong [Tue, 25 Oct 2016 22:14:32 +0000 (15:14 -0700)] 
xfs_db: deal with the CoW extent size hint

Display the CoW extent hint size when dumping inodes.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_db: metadump should copy the refcount btree too
Darrick J. Wong [Tue, 25 Oct 2016 22:14:32 +0000 (15:14 -0700)] 
xfs_db: metadump should copy the refcount btree too

Teach metadump to copy the refcount btree.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_db: add support for checking the refcount btree
Darrick J. Wong [Tue, 25 Oct 2016 22:14:31 +0000 (15:14 -0700)] 
xfs_db: add support for checking the refcount btree

Do some basic checks of the refcount btree.  xfs_repair will have to
check that the reference counts match the various bmbt mappings.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs_db: dump refcount btree data
Darrick J. Wong [Tue, 25 Oct 2016 22:14:31 +0000 (15:14 -0700)] 
xfs_db: dump refcount btree data

Add the ability to walk and dump the refcount btree in xfs_db.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agolibxfs: add fsxattr flags and fields for cowextsize
Darrick J. Wong [Tue, 25 Oct 2016 22:14:31 +0000 (15:14 -0700)] 
libxfs: add fsxattr flags and fields for cowextsize

Add the cowextsize field and flag to each platform's struct fsxattr
definitions.  We can compile these definitions into the xfsprogs
utilities if we don't pick them up from the system headers, such as on
kernels prior to 4.9.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agolibxfs: free the CoW fork from an inode
Darrick J. Wong [Tue, 25 Oct 2016 22:14:31 +0000 (15:14 -0700)] 
libxfs: free the CoW fork from an inode

Clean up the CoW fork, should there ever be one.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agolibxfs: plumb in bmap deferred op log items
Darrick J. Wong [Tue, 25 Oct 2016 22:14:31 +0000 (15:14 -0700)] 
libxfs: plumb in bmap deferred op log items

Add a deferred op handler for block mapping actions.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agolibxfs: plumb in refcount deferred op log items
Darrick J. Wong [Tue, 25 Oct 2016 22:14:31 +0000 (15:14 -0700)] 
libxfs: plumb in refcount deferred op log items

Add a deferred op handler for refcount update actions.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agolibxfs: add xfs_refcount.h to the standard include list
Darrick J. Wong [Tue, 25 Oct 2016 22:14:30 +0000 (15:14 -0700)] 
libxfs: add xfs_refcount.h to the standard include list

Pick up the definitions in xfs_refcount.h for all compilation units.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agolibxfs: initialize the in-core mount context for refcount btrees
Darrick J. Wong [Tue, 25 Oct 2016 22:14:30 +0000 (15:14 -0700)] 
libxfs: initialize the in-core mount context for refcount btrees

Initialize the refcount btree maxlevel field of the mount context.
This helps us to detect overly tall trees.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_buflock: handling parsing errors more gracefully
Darrick J. Wong [Tue, 25 Oct 2016 22:14:30 +0000 (15:14 -0700)] 
xfs_buflock: handling parsing errors more gracefully

Skip ftrace output lines that don't parse.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs_logprint: fix up the RUI printing code to reflect new format
Darrick J. Wong [Tue, 25 Oct 2016 22:14:30 +0000 (15:14 -0700)] 
xfs_logprint: fix up the RUI printing code to reflect new format

We changed the RUI format to use a variable length array, so update
the logprint code to reflect that.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: defer should abort intent items if the trans roll fails libxfs-4.9-sync
Darrick J. Wong [Tue, 25 Oct 2016 02:00:12 +0000 (13:00 +1100)] 
xfs: defer should abort intent items if the trans roll fails

Source kernel commit: b77428b12b55437b28deae738d9ce8b2e0663b55

If the deferred ops transaction roll fails, we need to abort the intent
items if we haven't already logged a done item for it, regardless of
whether or not the deferred ops has had a transaction committed.  Dave
found this while running generic/388.

Move the tracepoint to make it easier to track object lifetimes.

Reported-by: Dave Chinner <david@fromorbit.com>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: remove xfs_bunmapi_cow
Christoph Hellwig [Tue, 25 Oct 2016 01:59:49 +0000 (12:59 +1100)] 
xfs: remove xfs_bunmapi_cow

Source kernel commit: 64e6428ddd00f864e3ca105f914a2b6920c2bc41

Since no one uses it anymore.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: refactor xfs_bunmapi_cow
Christoph Hellwig [Tue, 25 Oct 2016 01:59:46 +0000 (12:59 +1100)] 
xfs: refactor xfs_bunmapi_cow

Source kernel commit: fa5c836ca8eb5bad6316ddfc066acbc4e2485356

Split out two helpers for deleting delayed or real extents from the COW fork.
This allows to call them directly from xfs_reflink_cow_end_io once that
function is refactored to iterate the extent tree.  It will also allow
to reuse the delalloc deletion from xfs_bunmapi in the future.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: add xfs_trim_extent
Darrick J. Wong [Tue, 25 Oct 2016 01:47:36 +0000 (12:47 +1100)] 
xfs: add xfs_trim_extent

Source kernel commit: 0a0af28cad9a43d90f13c2047bd8ee3d4cffb7f3

This helpers allows to trim an extent to a subset of it's original range
while making sure the block numbers in it remain valid,

In the future xfs_trim_extent and xfs_bmapi_trim_map should probably be
merged in some form.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
[hch: split from a previous patch from Darrick, moved around and added
support for "raw" delayed extents"]
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agolibxfs: clean up _calc_dquots_per_chunk
Darrick J. Wong [Tue, 25 Oct 2016 01:47:36 +0000 (12:47 +1100)] 
libxfs: clean up _calc_dquots_per_chunk

Source kernel commit: 58d789678546d46d7bbd809dd7dab417c0f23655

The function xfs_calc_dquots_per_chunk takes a parameter in units
of basic blocks.  The kernel seems to get the units wrong, but
userspace got 'fixed' by commenting out the unnecessary conversion.
Fix both.

cc: <stable@vger.kernel.org>
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Eric Sandeen <sandeen@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: remove pointless error goto in xfs_bmap_remap_alloc
Eric Sandeen [Tue, 25 Oct 2016 01:47:14 +0000 (12:47 +1100)] 
xfs: remove pointless error goto in xfs_bmap_remap_alloc

Source kernel commit: fe23759eaf2f6540de20c1623f066aad967ff9c9

The commit:

f65306ea xfs: map an inode's offset to an exact physical block

added a pointless error0: target; remove it.

Addresses-Coverity-Id: 1373865
Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Bill O'Donnell <billodo@redhat.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: add some 'static' annotations
Eric Biggers [Tue, 25 Oct 2016 01:47:14 +0000 (12:47 +1100)] 
xfs: add some 'static' annotations

Source kernel commit: f1b8243c55ca6fd2a3898e2f586b8cfcfff684bb

sparse reported that several variables and a function were not
forward-declared anywhere and therefore should be 'static'.

Found with sparse by running 'make C=2 CF=-D__CHECK_ENDIAN__ fs/xfs/'

Signed-off-by: Eric Biggers <ebiggers@google.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: remove redundant assignment of ifp
Colin Ian King [Tue, 25 Oct 2016 01:47:14 +0000 (12:47 +1100)] 
xfs: remove redundant assignment of ifp

Source kernel commit: 1d55a4bfd080ff4c6c96acfccfb7cdd2615ed6c2

Remove redundant ifp = ifp statement, it does nothing. Found with
static analysis by CoverityScan.

Signed-off-by: Colin Ian King <colin.king@canonical.com>
Reviewed-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: rework refcount cow recovery error handling
Darrick J. Wong [Tue, 25 Oct 2016 01:47:13 +0000 (12:47 +1100)] 
xfs: rework refcount cow recovery error handling

Source kernel commit: 6f97077ff6ef28e0f3b361b6ba9c95a222ef384b

The error handling in xfs_refcount_recover_cow_leftovers is confused
and can potentially leak memory, so rework it to release resources
correctly on error.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reported-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: implement swapext for rmap filesystems
Darrick J. Wong [Tue, 25 Oct 2016 01:47:13 +0000 (12:47 +1100)] 
xfs: implement swapext for rmap filesystems

Source kernel commit: 1f08af52e7c981e9877796a2d90b0e0f08666945

Implement swapext for filesystems that have reverse mapping.  Back in
the reflink patches, we augmented the bmap code with a 'REMAP' flag
that updates only the bmbt and doesn't touch the allocator and
implemented log redo items for those two operations.  Now we can
rewrite extent swapping as a (looong) series of remap operations.

This is far less efficient than the fork swapping method implemented
in the past, so we only switch this on for rmap.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: recognize the reflink feature bit
Darrick J. Wong [Tue, 25 Oct 2016 01:46:51 +0000 (12:46 +1100)] 
xfs: recognize the reflink feature bit

Source kernel commit: e54b5bf9d7aeb92d92c7f5115035e6a851d0f0c5

Add the reflink feature flag to the set of recognized feature flags.
This enables users to write to reflink filesystems.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: simulate per-AG reservations being critically low
Darrick J. Wong [Tue, 25 Oct 2016 01:46:51 +0000 (12:46 +1100)] 
xfs: simulate per-AG reservations being critically low

Source kernel commit: a35eb41519ab8db90e87d375ee9362d6e080ca4c

Create an error injection point that enables us to simulate being
critically low on per-AG block reservations.  This should enable us to
simulate this specific ENOSPC condition so that we can test falling back
to a regular file copy.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: don't mix reflink and DAX mode for now
Darrick J. Wong [Tue, 25 Oct 2016 01:46:51 +0000 (12:46 +1100)] 
xfs: don't mix reflink and DAX mode for now

Source kernel commit: 4f435ebe7d0422af61cdcddbbcc659888645a1e1

Since we don't have a strategy for handling both DAX and reflink,
for now we'll just prohibit both being set at the same time.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: check for invalid inode reflink flags
Darrick J. Wong [Tue, 25 Oct 2016 01:46:51 +0000 (12:46 +1100)] 
xfs: check for invalid inode reflink flags

Source kernel commit: c8e156ac336d82f67d7adc014404a2251e9dad09

We don't support sharing blocks on the realtime device.  Flag inodes
with the reflink or cowextsize flags set when the reflink feature is
disabled.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: convert unwritten status of reverse mappings for shared files
Darrick J. Wong [Tue, 25 Oct 2016 01:46:51 +0000 (12:46 +1100)] 
xfs: convert unwritten status of reverse mappings for shared files

Source kernel commit: 3f165b334e51477d2b33ac1c81b39927514daab7

Provide a function to convert an unwritten extent to a real one and
vice versa when shared extents are possible.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: use interval query for rmap alloc operations on shared files
Darrick J. Wong [Tue, 25 Oct 2016 01:46:22 +0000 (12:46 +1100)] 
xfs: use interval query for rmap alloc operations on shared files

Source kernel commit: ceeb9c832eeca5c1c2efc54a38f67283ccb60288

When it's possible for reverse mappings to overlap (data fork extents
of files on reflink filesystems), use the interval query function to
find the left neighbor of an extent we're trying to add; and be
careful to use the lookup functions to update the neighbors and/or
add new extents.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: add shared rmap map/unmap/convert log item types
Darrick J. Wong [Tue, 25 Oct 2016 01:43:48 +0000 (12:43 +1100)] 
xfs: add shared rmap map/unmap/convert log item types

Source kernel commit: 0e07c039bac5f6ce7e3bc512ab9efb4aaa76da94

Wire up some rmap log redo item type codes to map, unmap, or convert
shared data block extents.  The actual log item recovery comes in a
later patch.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: increase log reservations for reflink
Darrick J. Wong [Tue, 25 Oct 2016 01:43:48 +0000 (12:43 +1100)] 
xfs: increase log reservations for reflink

Source kernel commit: 80de462e090c2c346ca6ec6344b326e81e8cef84

Increase the log reservations to handle the increased rolling that
happens at the end of copy-on-write operations.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: try other AGs to allocate a BMBT block
Darrick J. Wong [Tue, 25 Oct 2016 01:43:48 +0000 (12:43 +1100)] 
xfs: try other AGs to allocate a BMBT block

Source kernel commit: 90e2056d76adc7894a019f5289d259de58065e13

Prior to the introduction of reflink, allocating a block and mapping
it into a file was performed in a single transaction with a single
block reservation, and the allocator was supposed to find enough
blocks to allocate the extent and any BMBT blocks that might be
necessary (unless we're low on space).

However, due to the way copy on write works, allocation and mapping
have been split into two transactions, which means that we must be
able to handle the case where we allocate an extent for CoW but that
AG runs out of free space before the blocks can be mapped into a file,
and the mapping requires a new BMBT block.  When this happens, look in
one of the other AGs for a BMBT block instead of taking the FS down.

The same applies to the functions that convert a data fork to extents
and later btree format.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: preallocate blocks for worst-case btree expansion
Darrick J. Wong [Tue, 25 Oct 2016 01:43:48 +0000 (12:43 +1100)] 
xfs: preallocate blocks for worst-case btree expansion

Source kernel commit: 84d6961910ea7b3ae8d8338f5b4df25dea68cee9

To gracefully handle the situation where a CoW operation turns a
single refcount extent into a lot of tiny ones and then run out of
space when a tree split has to happen, use the per-AG reserved block
pool to pre-allocate all the space we'll ever need for a maximal
btree.  For a 4K block size, this only costs an overhead of 0.3% of
available disk space.

When reflink is enabled, we have an unfortunate problem with rmap --
since we can share a block billions of times, this means that the
reverse mapping btree can expand basically infinitely.  When an AG is
so full that there are no free blocks with which to expand the rmapbt,
the filesystem will shut down hard.

This is rather annoying to the user, so use the AG reservation code to
reserve a "reasonable" amount of space for rmap.  We'll prevent
reflinks and CoW operations if we think we're getting close to
exhausting an AG's free space rather than shutting down, but this
permanent reservation should be enough for "most" users.  Hopefully.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
[hch@lst.de: ensure that we invalidate the freed btree buffer]
Signed-off-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: create a separate cow extent size hint for the allocator
Darrick J. Wong [Tue, 25 Oct 2016 01:43:42 +0000 (12:43 +1100)] 
xfs: create a separate cow extent size hint for the allocator

Source kernel commit: f7ca35227253dc8244fd908140b06010e67a31e5

Create a per-inode extent size allocator hint for copy-on-write.  This
hint is separate from the existing extent size hint so that CoW can
take advantage of the fragmentation-reducing properties of extent size
hints without disabling delalloc for regular writes.

The extent size hint that's fed to the allocator during a copy on
write operation is the greater of the cowextsize and regular extsize
hint.

During reflink, if we're sharing the entire source file to the entire
destination file and the destination file doesn't already have a
cowextsize hint, propagate the source file's cowextsize hint to the
destination file.

Furthermore, zero the bulkstat buffer prior to setting the fields
so that we don't copy kernel memory contents into userspace.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: teach get_bmapx about shared extents and the CoW fork
Darrick J. Wong [Tue, 25 Oct 2016 01:42:12 +0000 (12:42 +1100)] 
xfs: teach get_bmapx about shared extents and the CoW fork

Source kernel commit: f86f403794b1446b68afb3c233d4c0bc0e93b654

Teach xfs_getbmapx how to report shared extents and CoW fork contents
accurately in the bmap output by querying the refcount btree
appropriately.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: store in-progress CoW allocations in the refcount btree
Darrick J. Wong [Tue, 25 Oct 2016 01:41:50 +0000 (12:41 +1100)] 
xfs: store in-progress CoW allocations in the refcount btree

Source kernel commit: 174edb0e46e520230791a1a894397b7c824cefc4

Due to the way the CoW algorithm in XFS works, there's an interval
during which blocks allocated to handle a CoW can be lost -- if the FS
goes down after the blocks are allocated but before the block
remapping takes place.  This is exacerbated by the cowextsz hint --
allocated reservations can sit around for a while, waiting to get
used.

Since the refcount btree doesn't normally store records with refcount
of 1, we can use it to record these in-progress extents.  In-progress
blocks cannot be shared because they're not user-visible, so there
shouldn't be any conflicts with other programs.  This is a better
solution than holding EFIs during writeback because (a) EFIs can't be
relogged currently, (b) even if they could, EFIs are bound by
available log space, which puts an unnecessary upper bound on how much
CoW we can have in flight, and (c) we already have a mechanism to
track blocks.

At mount time, read the refcount records and free anything we find
with a refcount of 1 because those were in-progress when the FS went
down.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: support removing extents from CoW fork
Darrick J. Wong [Tue, 25 Oct 2016 01:38:27 +0000 (12:38 +1100)] 
xfs: support removing extents from CoW fork

Source kernel commit: 4862cfe825c0087c14452b362e708a35da675f5e

Create a helper method to remove extents from the CoW fork without
any of the side effects (rmapbt/bmbt updates) of the regular extent
deletion routine.  We'll eventually use this to clear out the CoW fork
during ioend processing.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: support allocating delayed extents in CoW fork
Darrick J. Wong [Tue, 25 Oct 2016 01:38:27 +0000 (12:38 +1100)] 
xfs: support allocating delayed extents in CoW fork

Source kernel commit: 60b4984fc3924bff292ec46b95a3e98b34b8e259

Modify xfs_bmap_add_extent_delay_real() so that we can convert delayed
allocation extents in the CoW fork to real allocations, and wire this
up all the way back to xfs_iomap_write_allocate().  In a subsequent
patch, we'll modify the writepage handler to call this.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: support bmapping delalloc extents in the CoW fork
Darrick J. Wong [Tue, 25 Oct 2016 01:37:50 +0000 (12:37 +1100)] 
xfs: support bmapping delalloc extents in the CoW fork

Source kernel commit: be51f8119c2f5e27437d2c4271f6419f3b8e609f

Allow the creation of delayed allocation extents in the CoW fork.  In
a subsequent patch we'll wire up iomap_begin to actually do this via
reflink helper functions.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: introduce the CoW fork
Darrick J. Wong [Tue, 25 Oct 2016 01:37:28 +0000 (12:37 +1100)] 
xfs: introduce the CoW fork

Source kernel commit: 3993baeb3c52f497d243a4a3b5510df97b22596b

Introduce a new in-core fork for storing copy-on-write delalloc
reservations and allocated extents that are in the process of being
written out.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: don't allow reflinked dir/dev/fifo/socket/pipe files
Darrick J. Wong [Tue, 25 Oct 2016 01:37:28 +0000 (12:37 +1100)] 
xfs: don't allow reflinked dir/dev/fifo/socket/pipe files

Source kernel commit: 11715a21bc3035440b853a0334685f1a55ca8c3c

Only non-rt files can be reflinked, so check that when we load an
inode.  Also, don't leak the attr fork if there's a failure.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: add reflink feature flag to geometry
Darrick J. Wong [Tue, 25 Oct 2016 01:37:28 +0000 (12:37 +1100)] 
xfs: add reflink feature flag to geometry

Source kernel commit: f0ec1b8ef11df0a51954df7e3ff3ca4aadb0d34b

Report the reflink feature in the XFS geometry so that xfs_info and
friends know the filesystem has this feature.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: return work remaining at the end of a bunmapi operation
Darrick J. Wong [Tue, 25 Oct 2016 01:37:28 +0000 (12:37 +1100)] 
xfs: return work remaining at the end of a bunmapi operation

Source kernel commit: 4453593be6c54e7581467e80f4a2757be098a3a2

Return the range of file blocks that bunmapi didn't free.  This hint
is used by CoW and reflink to figure out what part of an extent
actually got freed so that it can set up the appropriate atomic
remapping of just the freed range.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: implement deferred bmbt map/unmap operations
Darrick J. Wong [Tue, 25 Oct 2016 01:37:20 +0000 (12:37 +1100)] 
xfs: implement deferred bmbt map/unmap operations

Source kernel commit: 9f3afb57d5f1e7145986132106c6ca91f8136cc2

Implement deferred versions of the inode block map/unmap functions.
These will be used in subsequent patches to make reflink operations
atomic.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: pass bmapi flags through to bmap_del_extent
Darrick J. Wong [Tue, 25 Oct 2016 01:31:59 +0000 (12:31 +1100)] 
xfs: pass bmapi flags through to bmap_del_extent

Source kernel commit: 4847acf868bb426455c8b703c80ed5fc5e2ee556

Pass BMAPI_ flags from bunmapi into bmap_del_extent and extend
BMAPI_REMAP (which means "don't touch the allocator or the quota
accounting") to apply to bunmapi as well.  This will be used to
implement the unmap operation, which will be used by swapext.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: map an inode's offset to an exact physical block
Darrick J. Wong [Tue, 25 Oct 2016 01:31:37 +0000 (12:31 +1100)] 
xfs: map an inode's offset to an exact physical block

Source kernel commit: f65306ea5246ef3ff68a6abf85f5a73a04903366

Teach the bmap routine to know how to map a range of file blocks to a
specific range of physical blocks, instead of simply allocating fresh
blocks.  This enables reflink to map a file to blocks that are already
in use.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: log bmap intent items
Darrick J. Wong [Tue, 25 Oct 2016 01:30:19 +0000 (12:30 +1100)] 
xfs: log bmap intent items

Source kernel commit: 77d61fe45e720577a2cc0e9580fbc57d8faa7232

Provide a mechanism for higher levels to create BUI/BUD items, submit
them to the log, and a stub function to deal with recovered BUI items.
These parts will be connected to the rmapbt in a later patch.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: create bmbt update intent log items
Darrick J. Wong [Tue, 25 Oct 2016 01:29:43 +0000 (12:29 +1100)] 
xfs: create bmbt update intent log items

Source kernel commit: 6413a01420c2fbf03b3d059795f541caeb962e86

Create bmbt update intent/done log items to record redo information in
the log.  Because we roll transactions multiple times for reflink
operations, we also have to track the status of the metadata updates
that will be recorded in the post-roll transactions in case we crash
before committing the final transaction.  This mechanism enables log
recovery to finish what was already started.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: introduce reflink utility functions
Darrick J. Wong [Tue, 25 Oct 2016 01:29:22 +0000 (12:29 +1100)] 
xfs: introduce reflink utility functions

Source kernel commit: 350a27a6a65cc5dd2ba1b220e8641993414816d2

These functions will be used by the other reflink functions to find
the maximum length of a range of shared blocks.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.coM>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: reserve AG space for the refcount btree root
Darrick J. Wong [Tue, 25 Oct 2016 01:26:50 +0000 (12:26 +1100)] 
xfs: reserve AG space for the refcount btree root

Source kernel commit: d0e853f3600cd2a3f7c4a067dc38155c77c51df9

Reduce the max AG usable space size so that we always have space for
the refcount btree root.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: adjust refcount when unmapping file blocks
Darrick J. Wong [Tue, 25 Oct 2016 01:26:48 +0000 (12:26 +1100)] 
xfs: adjust refcount when unmapping file blocks

Source kernel commit: 62aab20f08758b1b171a73a54e0c72dd12beb980

When we're unmapping blocks from a reflinked file, decrease the
refcount of the affected blocks and free the extents that are no
longer in use.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: connect refcount adjust functions to upper layers
Darrick J. Wong [Tue, 25 Oct 2016 01:24:46 +0000 (12:24 +1100)] 
xfs: connect refcount adjust functions to upper layers

Source kernel commit: 33ba6129208475ec3aeffe6e9dad9f9afe022405

Plumb in the upper level interface to schedule and finish deferred
refcount operations via the deferred ops mechanism.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: adjust refcount of an extent of blocks in refcount btree
Darrick J. Wong [Tue, 25 Oct 2016 01:24:45 +0000 (12:24 +1100)] 
xfs: adjust refcount of an extent of blocks in refcount btree

Source kernel commit: 3172725814f9a689d6e8b3c7979b66403abf5dae

Provide functions to adjust the reference counts for an extent of
physical blocks stored in the refcount btree.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: log refcount intent items
Darrick J. Wong [Tue, 25 Oct 2016 01:20:28 +0000 (12:20 +1100)] 
xfs: log refcount intent items

Source kernel commit: f997ee2137175f5b2bd7ced52acf1ca51f04f420

Provide a mechanism for higher levels to create CUI/CUD items, submit
them to the log, and a stub function to deal with recovered CUI items.
These parts will be connected to the refcountbt in a later patch.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: create refcount update intent log items
Darrick J. Wong [Tue, 25 Oct 2016 01:20:28 +0000 (12:20 +1100)] 
xfs: create refcount update intent log items

Source kernel commit: baf4bcacb715cebd412b2f4bb69989ef24496523

Create refcount update intent/done log items to record redo
information in the log.  Because we need to roll transactions between
updating the bmbt mapping and updating the reverse mapping, we also
have to track the status of the metadata updates that will be recorded
in the post-roll transactions, just in case we crash before committing
the final transaction.  This mechanism enables log recovery to finish
what was already started.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: add refcount btree operations
Darrick J. Wong [Tue, 25 Oct 2016 01:20:26 +0000 (12:20 +1100)] 
xfs: add refcount btree operations

Source kernel commit: bdf28630b72154e5766cbad5874576b6f22e7237

Implement the generic btree operations required to manipulate refcount
btree blocks.  The implementation is similar to the bmapbt, though it
will only allocate and free blocks from the AG.

Since the refcount root and level fields are separate from the
existing roots and levels array, they need a separate logging flag.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
[hch: fix logging of AGF refcount btree fields]
Signed-off-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: account for the refcount btree in the alloc/free log reservation
Darrick J. Wong [Tue, 25 Oct 2016 01:01:06 +0000 (12:01 +1100)] 
xfs: account for the refcount btree in the alloc/free log reservation

Source kernel commit: f310bd2ecd37b17bf0042c9d1595329057970eb6

Every time we allocate or free a data extent, we might need to split
the refcount btree.  Reserve some blocks in the transaction to handle
this possibility.  Even though the deferred refcount code can roll a
transaction to avoid overloading the transaction, we can still exceed
the reservation.

Certain pathological workloads (1k blocks, no cowextsize hint, random
directio writes), cause a perfect storm wherein a refcount adjustment
of a large range of blocks causes full tree splits in two separate
extents in two separate refcount tree blocks; allocating new refcount
tree blocks causes rmap btree splits; and all the allocation activity
causes the freespace btrees to split, blowing the reservation.

(Reproduced by generic/167 over NFS atop XFS)

Signed-off-by: Christoph Hellwig <hch@lst.de>
[darrick.wong@oracle.com: add commit message]
Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs: define the on-disk refcount btree format
Darrick J. Wong [Tue, 25 Oct 2016 01:00:53 +0000 (12:00 +1100)] 
xfs: define the on-disk refcount btree format

Source kernel commit: 1946b91cee4fc8ae25450673e4d4f35e9b462e9e

Start constructing the refcount btree implementation by establishing
the on-disk format and everything needed to read, write, and
manipulate the refcount btree blocks.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: refcount btree add more reserved blocks
Darrick J. Wong [Tue, 25 Oct 2016 00:49:22 +0000 (11:49 +1100)] 
xfs: refcount btree add more reserved blocks

Source kernel commit: af30dfa14411e9df0e69c6e46e8c6c467b88229d

Since XFS reserves a small amount of space in each AG as the minimum
free space needed for an operation, save some more space in case we
touch the refcount btree.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: introduce refcount btree definitions
Darrick J. Wong [Tue, 25 Oct 2016 00:48:45 +0000 (11:48 +1100)] 
xfs: introduce refcount btree definitions

Source kernel commit: 46eeb521b95247170d2db773bb4cc8fb3de1d85c

Add new per-AG refcount btree definitions to the per-AG structures.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Christoph Hellwig <hch@lst.de>
8 years agoxfs: remote attribute blocks aren't really userdata
Dave Chinner [Tue, 25 Oct 2016 00:46:35 +0000 (11:46 +1100)] 
xfs: remote attribute blocks aren't really userdata

Source kernel commit: 292378edcb408c652e841fdc867fc14f8b4995fa

When adding a new remote attribute, we write the attribute to the
new extent before the allocation transaction is committed. This
means we cannot reuse busy extents as that violates crash
consistency semantics. Hence we currently treat remote attribute
extent allocation like userdata because it has the same overwrite
ordering constraints as userdata.

Unfortunately, this also allows the allocator to incorrectly apply
extent size hints to the remote attribute extent allocation. This
results in interesting failures, such as transaction block
reservation overruns and in-memory inode attribute fork corruption.

To fix this, we need to separate the busy extent reuse configuration
from the userdata configuration. This changes the definition of
XFS_BMAPI_METADATA slightly - it now means that allocation is
metadata and reuse of busy extents is acceptible due to the metadata
ordering semantics of the journal. If this flag is not set, it
means the allocation is that has unordered data writeback, and hence
busy extent reuse is not allowed. It no longer implies the
allocation is for user data, just that the data write will not be
strictly ordered. This matches the semantics for both user data
and remote attribute block allocation.

As such, This patch changes the "userdata" field to a "datatype"
field, and adds a "no busy reuse" flag to the field.
When we detect an unordered data extent allocation, we immediately set
the no reuse flag. We then set the "user data" flags based on the
inode fork we are allocating the extent to. Hence we only set
userdata flags on data fork allocations now and consider attribute
fork remote extents to be an unordered metadata extent.

The result is that remote attribute extents now have the expected
allocation semantics, and the data fork allocation behaviour is
completely unchanged.

It should be noted that there may be other ways to fix this (e.g.
use ordered metadata buffers for the remote attribute extent data
write) but they are more invasive and difficult to validate both
from a design and implementation POV. Hence this patch takes the
simple, obvious route to fixing the problem...

Reported-and-tested-by: Ross Zwisler <ross.zwisler@linux.intel.com>
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: rewrite and optimize the delalloc write path
Christoph Hellwig [Tue, 25 Oct 2016 00:46:35 +0000 (11:46 +1100)] 
xfs: rewrite and optimize the delalloc write path

Source kernel commit: 51446f5ba44874db4d2a93a6eb61b133e5ec1b3e

Currently xfs_iomap_write_delay does up to lookups in the inode
extent tree, which is rather costly especially with the new iomap
based write path and small write sizes.

But it turns out that the low-level xfs_bmap_search_extents gives us
all the information we need in the regular delalloc buffered write
path:

- it will return us an extent covering the block we are looking up
if it exists.  In that case we can simply return that extent to
the caller and are done
- it will tell us if we are beyoned the last current allocated
block with an eof return parameter.  In that case we can create a
delalloc reservation and use the also returned information about
the last extent in the file as the hint to size our delalloc
reservation.
- it can tell us that we are writing into a hole, but that there is
an extent beyoned this hole.  In this case we can create a
delalloc reservation that covers the requested size (possible
capped to the next existing allocation).

All that can be done in one single routine instead of bouncing up
and down a few layers.  This reduced the CPU overhead of the block
mapping routines and also simplified the code a lot.

Signed-off-by: Christoph Hellwig <hch@lst.de>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: set up per-AG free space reservations
Darrick J. Wong [Tue, 25 Oct 2016 00:46:32 +0000 (11:46 +1100)] 
xfs: set up per-AG free space reservations

Source kernel commit: 3fd129b63fd062a0d8f5d55994a6e98896c20fa7

One unfortunate quirk of the reference count and reverse mapping
btrees -- they can expand in size when blocks are written to *other*
allocation groups if, say, one large extent becomes a lot of tiny
extents.  Since we don't want to start throwing errors in the middle
of CoWing, we need to reserve some blocks to handle future expansion.
The transaction block reservation counters aren't sufficient here
because we have to have a reserve of blocks in every AG, not just
somewhere in the filesystem.

Therefore, create two per-AG block reservation pools.  One feeds the
AGFL so that rmapbt expansion always succeeds, and the other feeds all
other metadata so that refcountbt expansion never fails.

Use the count of how many reserved blocks we need to have on hand to
create a virtual reservation in the AG.  Through selective clamping of
the maximum length of allocation requests and of the length of the
longest free extent, we can make it look like there's less free space
in the AG unless the reservation owner is asking for blocks.

In other words, play some accounting tricks in-core to make sure that
we always have blocks available.  On the plus side, there's nothing to
clean up if we crash, which is contrast to the strategy that the rough
draft used (actually removing extents from the freespace btrees).

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
new file mode 100644
index 000000000000..e3ae0f2b4294

8 years agoxfs: defer should allow ->finish_item to request a new transaction
Darrick J. Wong [Tue, 25 Oct 2016 00:26:53 +0000 (11:26 +1100)] 
xfs: defer should allow ->finish_item to request a new transaction

Source kernel commit: 385d655861d221bb43ae69a9cfa9adbefe31ad00

When xfs_defer_finish calls ->finish_item, it's possible that
(refcount) won't be able to finish all the work in a single
transaction.  When this happens, the ->finish_item handler should
shorten the log done item's list count, update the work item to
reflect where work should continue, and return -EAGAIN so that
defer_finish knows to retain the pending item on the pending list,
roll the transaction, and restart processing where we left off.

Plumb in the code and document how this mechanism is supposed to work.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
8 years agoxfs: count the blocks in a btree
Darrick J. Wong [Tue, 25 Oct 2016 00:26:53 +0000 (11:26 +1100)] 
xfs: count the blocks in a btree

Source kernel commit: c611cc0360cd924448c23ccd70ce8be703fcb4a6

Provide a helper method to count the number of blocks in a short form
btree.  The refcount and rmap btrees need to know the number of blocks
already in use to set up their per-AG block reservations during mount.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: create a standard btree size calculator code
Darrick J. Wong [Tue, 25 Oct 2016 00:26:53 +0000 (11:26 +1100)] 
xfs: create a standard btree size calculator code

Source kernel commit: 4ed3f68792f6a9c21a290ae777565e7562a09653

Create a helper to generate AG btree height calculator functions.
This will be used (much) later when we get to the refcount btree.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: remove xfs_btree_bigkey
Darrick J. Wong [Tue, 25 Oct 2016 00:26:51 +0000 (11:26 +1100)] 
xfs: remove xfs_btree_bigkey

Source kernel commit: a1d46cffaf40e04acb0ecab14980ece3ef1ab933

Remove the xfs_btree_bigkey mess and simply make xfs_btree_key big enough
to hold both keys in-core.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: convert RUI log formats to use variable length arrays
Darrick J. Wong [Tue, 25 Oct 2016 00:26:50 +0000 (11:26 +1100)] 
xfs: convert RUI log formats to use variable length arrays

Source kernel commit: cd00158ce34d6e2c42d8892e8499779b8ac1d2bf

Use variable length array declarations for RUI log items,
and replace the open coded sizeof formulae with a single function.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: track log done items directly in the deferred pending work item
Darrick J. Wong [Tue, 25 Oct 2016 00:26:47 +0000 (11:26 +1100)] 
xfs: track log done items directly in the deferred pending work item

Source kernel commit: ea78d80866ce375defb2fdd1c8a3aafec95e0f85

Christoph reports slab corruption when a deferred refcount update
aborts during _defer_finish().  The cause of this was broken log item
state tracking in xfs_defer_pending -- upon an abort,
_defer_trans_abort() will call abort_intent on all intent items,
including the ones that have already had a done item attached.

This is incorrect because each intent item has 2 refcount: the first
is released when the intent item is committed to the log; and the
second is released when the _done_ item is committed to the log, or
by the intent creator if there is no done item.  In other words, once
we log the done item, responsibility for releasing the intent item's
second refcount is transferred to the done item and /must not/ be
performed by anything else.

The dfp_committed flag should have been tracking whether or not we had
a done item so that _defer_trans_abort could decide if it needs to
abort the intent item, but due to a thinko this was not the case.  Rip
it out and track the done item directly so that we do the right thing
w.r.t. intent item freeing.

Signed-off-by: Darrick J. Wong <darrick.wong@oracle.com>
Reported-by: Christoph Hellwig <hch@infradead.org>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs: fix superblock inprogress check
Dave Chinner [Tue, 25 Oct 2016 00:26:46 +0000 (11:26 +1100)] 
xfs: fix superblock inprogress check

Source kernel commit: f3d7ebdeb2c297bd26272384e955033493ca291c

From inspection, the superblock sb_inprogress check is done in the
verifier and triggered only for the primary superblock via a
"bp->b_bn == XFS_SB_DADDR" check.

Unfortunately, the primary superblock is an uncached buffer, and
hence it is configured by xfs_buf_read_uncached() with:

bp->b_bn = XFS_BUF_DADDR_NULL;  /* always null for uncached buffers */

And so this check never triggers. Fix it.

cc: <stable@vger.kernel.org>
Signed-off-by: Dave Chinner <dchinner@redhat.com>
Reviewed-by: Brian Foster <bfoster@redhat.com>
Reviewed-by: Christoph Hellwig <hch@lst.de>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agolibxfs_apply: filterdiff can't handle /dev/null properly
Dave Chinner [Mon, 24 Oct 2016 22:52:21 +0000 (09:52 +1100)] 
libxfs_apply: filterdiff can't handle /dev/null properly

Because we are mangling the diff source/destination locations, we
have to add prefixes to them to get them to apply cleanly as -p1
patches. This is all fine until we create or remove a file and
the the src/dest is /dev/null. Applying a prefix here causes
the diff to be malformed and it won't apply.

Add another hack to work around this limitation of filterdiff when
reformatting the diff into readable format.

Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agolibxfs_apply: filter commits from libxfs only
Dave Chinner [Mon, 24 Oct 2016 22:04:33 +0000 (09:04 +1100)] 
libxfs_apply: filter commits from libxfs only

When pulling commits from the kernel, it's easy to specify a commit
range such as "v4.8..for-next" to indicate we want to pull all
commits for libxfs since the 4.8 kernel release. Unfortunately,
this pull commits from all over the kernel tree, not just
fs/xfs/libxfs.

Filter the commit list retrieval to limit the commits to those touch
fs/xfs/libxfs so that we only attempt to apply the realtively small
number of relevant commits.

Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfsprogs: Release 4.8.0 v4.8.0
Dave Chinner [Mon, 17 Oct 2016 03:17:48 +0000 (14:17 +1100)] 
xfsprogs: Release 4.8.0

Update all the necessary files for a 4.8.0 release.

Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfsprogs: Release 4.8.0-rc3 v4.8.0-rc3
Dave Chinner [Mon, 3 Oct 2016 03:25:45 +0000 (14:25 +1100)] 
xfsprogs: Release 4.8.0-rc3

Update all the necessary files for a 4.8.0-rc3 release.

Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs_io: fix inode command with "-n" for bogus inode
Eric Sandeen [Sun, 2 Oct 2016 23:56:00 +0000 (10:56 +1100)] 
xfs_io: fix inode command with "-n" for bogus inode

If we ask for the next allocated inode after a number for which
no other inode exists, the bulkstat returns success, but with
count == 0.  If we ignore this fact, we print a garbage result
from bstat.bs_ino in this case, so fix it.

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>
8 years agoxfs_io: refactor inode command
Eric Sandeen [Sun, 2 Oct 2016 23:50:21 +0000 (10:50 +1100)] 
xfs_io: refactor inode command

The inode_f function is a bit convoluted; the default
find-last-inode case appears at the end, there are several return
points, we print the same basic information using 2 different
variables in 2 different locations depending on the mode we're in,
the "inode not found" was a printf & exit in the middle of the
function, etc.

Move the default case up to the top so it's more obvious, not
buried.

Make a new var, result_ino, which holds whatever we want to print
regardless of the mode, and then handle all the output at the end.

Signed-off-by: Eric Sandeen <sandeen@redhat.com>
Reviewed-by: Dave Chinner <dchinner@redhat.com>
Signed-off-by: Dave Chinner <david@fromorbit.com>