5 External (third-party metadata) reshape differs from native-metadata
6 reshape in three key ways:
8 1.1 Format specific constraints
10 In the native case reshape is limited by what is implemented in the
11 generic reshape routine (Grow_reshape()) and what is supported by the
12 kernel. There are exceptional cases where Grow_reshape() may block
13 operations when it knows that the kernel implementation is broken, but
14 otherwise the kernel is relied upon to be the final arbiter of what
15 reshape operations are supported.
17 In the external case the kernel, and the generic checks in
18 Grow_reshape(), become the super-set of what reshapes are possible. The
19 metadata format may not support, or have yet to implement a given
20 reshape type. The implication for Grow_reshape() is that it must query
21 the metadata handler and effect changes in the metadata before the new
22 geometry is posted to the kernel. The ->reshape_super method allows
23 Grow_reshape() to validate the requested operation and post the metadata
28 Native metadata reshape is always performed at the array scope (no
29 metadata relationship with sibling arrays on the same disks). External
30 reshape, depending on the format, may not allow the number of member
31 disks to be changed in a subarray unless the change is simultaneously
32 applied to all subarrays in the container. For example the imsm format
33 requires all member disks to be a member of all subarrays, so a 4-disk
34 raid5 in a container that also houses a 4-disk raid10 array could not be
35 reshaped to 5 disks as the imsm format does not support a 5-disk raid10
36 representation. This requires the ->reshape_super method to check the
37 contents of the array and ask the user to run the reshape at container
38 scope (if both subarrays are agreeable to the change), or report an
39 error in the case where one subarray cannot support the change.
41 1.3 Monitoring / checkpointing
43 Reshape, unlike rebuild/resync, requires strict checkpointing to survive
44 interrupted reshape operations. For example when expanding a raid5
45 array the first few stripes of the array will be overwritten in a
46 destructive manner. When restarting the reshape process we need to know
47 the exact location of the last successfully written stripe, and we need
48 to restore the data in any partially overwritten stripe. Native
49 metadata stores this backup data in the unused portion of spares that
50 are being promoted to array members, or in an external backup file
51 (located on a non-involved block device).
53 The kernel is in charge of recording checkpoints of reshape progress,
54 but mdadm is delegated the task of managing the backup space which
56 1/ Identifying what data will be overwritten in the next unit of reshape
58 2/ Suspending access to that region so that a snapshot of the data can
59 be transferred to the backup space.
60 3/ Allowing the kernel to reshape the saved region and setting the
61 boundary for the next backup.
63 In the external reshape case we want to preserve this mdadm
64 'reshape-manager' arrangement, but have a third actor, mdmon, to
65 consider. It is tempting to give the role of managing reshape to mdmon,
66 but that is counter to its role as a monitor, and conflicts with the
67 existing capabilities and role of mdadm to manage the progress of
68 reshape. For clarity the external reshape implementation maintains the
69 role of mdmon as a (mostly) passive recorder of raid events, and mdadm
70 treats it as it would the kernel in the native reshape case (modulo
71 needing to send explicit metadata update messages and checking that
72 mdmon took the expected action).
74 External reshape can use the generic md backup file as a fallback, but in the
75 optimal/firmware-compatible case the reshape-manager will use the metadata
76 specific areas for managing reshape. The implementation also needs to spawn a
77 reshape-manager per subarray when the reshape is being carried out at the
78 container level. For these two reasons the ->manage_reshape() method is
79 introduced. This method in addition to base tasks mentioned above:
80 1/ Spawns a manager per-subarray, when necessary
81 2/ Uses either generic routines in Grow.c for md-style backup file
82 support, or uses the metadata-format specific location for storing
84 This aims to avoid a "midlayer mistake"[1] and lets the metadata handler
85 optionally take advantage of generic infrastructure in Grow.c
87 2 Details for specific reshape requests
89 There are quite a few moving pieces spread out across md, mdadm, and mdmon for
90 the support of external reshape, and there are several different types of
91 reshape that need to be comprehended by the implementation. A rundown of
92 these details follows.
94 2.0 General provisions:
96 Obtain an exclusive open on the container to make sure we are not
97 running concurrently with a Create() event.
99 2.1 Freezing sync_action
103 1/ mdadm::Grow_reshape(): checks if mdmon is running and optionally
104 initializes st->update_tail
105 2/ mdadm::Grow_reshape() calls ->reshape_super() to check that the size change
106 is allowed (being performed at subarray scope / enough room) prepares a
108 3/ mdadm::Grow_reshape(): flushes the metadata update (via
109 flush_metadata_update(), or ->sync_metadata())
110 4/ mdadm::Grow_reshape(): post the new size to the kernel
113 2.3 Reshape level (simple-takeover)
115 "simple-takeover" implies the level change can be satisfied without touching
118 1/ mdadm::Grow_reshape(): checks if mdmon is running and optionally
119 initializes st->update_tail
120 2/ mdadm::Grow_reshape() calls ->reshape_super() to check that the level change
121 is allowed (being performed at subarray scope) prepares a
123 2a/ raid10 --> raid0: degrade all mirror legs prior to calling
125 3/ mdadm::Grow_reshape(): flushes the metadata update (via
126 flush_metadata_update(), or ->sync_metadata())
127 4/ mdadm::Grow_reshape(): post the new level to the kernel
129 2.4 Reshape chunk, layout
131 2.5 Reshape raid disks (grow)
133 1/ mdadm::Grow_reshape(): unconditionally initializes st->update_tail
134 because only redundant raid levels can modify the number of raid disks
135 2/ mdadm::Grow_reshape(): calls ->reshape_super() to check that the level
136 change is allowed (being performed at proper scope / permissible
137 geometry / proper spares available in the container) prepares a metadata
139 3/ mdadm::Grow_reshape(): Converts each subarray in the container to the
140 raid level that can perform the reshape and starts mdmon.
141 4/ mdadm::Grow_reshape(): Pushes the update to mdmon...
142 4a/ mdmon::process_update(): marks the array as reshaping
143 4b/ mdmon::manage_member(): adds the spares (without assigning a slot)
144 5/ mdadm::Grow_reshape(): Notes that mdmon has assigned spares and invokes
146 5/ mdadm::<format>->manage_reshape(): (for each subarray) sets sync_max to
147 zero, starts the reshape, and pings mdmon
148 5a/ mdmon::read_and_act(): notices that reshape has started and notifies
149 the metadata handler to record the slots chosen by the kernel
150 6/ mdadm::<format>->manage_reshape(): saves data that will be overwritten by
151 the kernel to either the backup file or the metadata specific location,
152 advances sync_max, waits for reshape, ping mdmon, repeat.
153 6a/ mdmon::read_and_act(): records checkpoints
154 7/ mdadm::<format>->manage_reshape(): Once reshape completes changes the raid
155 level back to the nominal raid level (if necessary)
157 FIXME: native metadata does not have the capability to record the original
158 raid level in reshape-restart case because the kernel always records current
159 raid level to the metadata, whereas external metadata can masquerade at an
160 alternate level based on the reshape state.
162 2.6 Reshape raid disks (shrink)
168 [1]: Linux kernel design patterns - part 3, Neil Brown http://lwn.net/Articles/336262/