3 md \- Multiple Device driver aka Linux Software Raid
11 driver provides virtual devices that are created from one or more
12 independent underlying devices. This array of devices often contains
13 redundancy, and hence the acronym RAID which stands for a Redundant
14 Array of Independent Devices.
19 4 (striped array with parity device),
20 5 (striped array with distributed parity information),
21 6 (striped array with distributed dual redundancy information), and
22 10 (striped and mirrored).
23 If some number of underlying devices fails while using one of these
24 levels, the array will continue to function; this number is one for
25 RAID levels 4 and 5, two for RAID level 6, and all but one (N-1) for
26 RAID level 1, and dependant of configuration for level 10.
29 also supports a number of pseudo RAID (non-redundant) configurations
30 including RAID0 (striped array), LINEAR (catenated array),
31 MULTIPATH (a set of different interfaces to the same device),
32 and FAULTY (a layer over a single device into which errors can be injected).
35 Each device in an array may have a
37 which records information about the structure and state of the array.
38 This allows the array to be reliably re-assembled after a shutdown.
40 From Linux kernel version 2.6.10,
42 provides support for two different formats of this superblock, and
43 other formats can be added. Prior to this release, only one format is
46 The common format - known as version 0.90 - has
47 a superblock that is 4K long and is written into a 64K aligned block that
48 starts at least 64K and less than 128K from the end of the device
49 (i.e. to get the address of the superblock round the size of the
50 device down to a multiple of 64K and then subtract 64K).
51 The available size of each device is the amount of space before the
52 super block, so between 64K and 128K is lost when a device in
53 incorporated into an MD array.
54 This superblock stores multi-byte fields in a processor-dependant
55 manner, so arrays cannot easily be moved between computers with
58 The new format - known as version 1 - has a superblock that is
59 normally 1K long, but can be longer. It is normally stored between 8K
60 and 12K from the end of the device, on a 4K boundary, though
61 variations can be stored at the start of the device (version 1.1) or 4K from
62 the start of the device (version 1.2).
63 This superblock format stores multibyte data in a
64 processor-independant format and has supports upto hundreds of
65 component devices (version 0.90 only supports 28).
67 The superblock contains, among other things:
70 The manner in which the devices are arranged into the array
71 (linear, raid0, raid1, raid4, raid5, raid10, multipath).
74 a 128 bit Universally Unique Identifier that identifies the array that
75 this device is part of.
77 .SS ARRAYS WITHOUT SUPERBLOCKS
78 While it is usually best to create arrays with superblocks so that
79 they can be assembled reliably, there are some circumstances where an
80 array without superblocks in preferred. This include:
85 driver only supported Linear and Raid0 configurations and did not use
86 a superblock (which is less critical with these configurations).
87 While such arrays should be rebuilt with superblocks if possible,
89 continues to support them.
92 Being a largely transparent layer over a different device, the FAULTY
93 personality doesn't gain anything from having a superblock.
96 It is often possible to detect devices which are different paths to
97 the same storage directly rather than having a distinctive superblock
98 written to the device and searched for on all paths. In this case,
99 a MULTIPATH array with no superblock makes sense.
102 In some configurations it might be desired to create a raid1
103 configuration that does use a superblock, and to maintain the state of
104 the array elsewhere. While not encouraged, this is supported.
108 A linear array simply catenates the available space on each
109 drive together to form one large virtual drive.
111 One advantage of this arrangement over the more common RAID0
112 arrangement is that the array may be reconfigured at a later time with
113 an extra drive and so the array is made bigger without disturbing the
114 data that is on the array. However this cannot be done on a live
117 If a chunksize is given with a LINEAR array, the usable space on each
118 device is rounded down to a multiple of this chunksize.
122 A RAID0 array (which has zero redundancy) is also known as a
124 A RAID0 array is configured at creation with a
126 which must be a power of two, and at least 4 kibibytes.
128 The RAID0 driver assigns the first chunk of the array to the first
129 device, the second chunk to the second device, and so on until all
130 drives have been assigned one chunk. This collection of chunks forms
133 Further chunks are gathered into stripes in the same way which are
134 assigned to the remaining space in the drives.
136 If devices in the array are not all the same size, then once the
137 smallest device has been exhausted, the RAID0 driver starts
138 collecting chunks into smaller stripes that only span the drives which
139 still have remaining space.
144 A RAID1 array is also known as a mirrored set (though mirrors tend to
145 provide reflected images, which RAID1 does not) or a plex.
147 Once initialised, each device in a RAID1 array contains exactly the
148 same data. Changes are written to all devices in parallel. Data is
149 read from any one device. The driver attempts to distribute read
150 requests across all devices to maximise performance.
152 All devices in a RAID1 array should be the same size. If they are
153 not, then only the amount of space available on the smallest device is
154 used. Any extra space on other devices is wasted.
158 A RAID4 array is like a RAID0 array with an extra device for storing
159 parity. This device is the last of the active devices in the
160 array. Unlike RAID0, RAID4 also requires that all stripes span all
161 drives, so extra space on devices that are larger than the smallest is
164 When any block in a RAID4 array is modified the parity block for that
165 stripe (i.e. the block in the parity device at the same device offset
166 as the stripe) is also modified so that the parity block always
167 contains the "parity" for the whole stripe. i.e. its contents is
168 equivalent to the result of performing an exclusive-or operation
169 between all the data blocks in the stripe.
171 This allows the array to continue to function if one device fails.
172 The data that was on that device can be calculated as needed from the
173 parity block and the other data blocks.
177 RAID5 is very similar to RAID4. The difference is that the parity
178 blocks for each stripe, instead of being on a single device, are
179 distributed across all devices. This allows more parallelism when
180 writing as two different block updates will quite possibly affect
181 parity blocks on different devices so there is less contention.
183 This also allows more parallelism when reading as read requests are
184 distributed over all the devices in the array instead of all but one.
188 RAID6 is similar to RAID5, but can handle the loss of any \fItwo\fP
189 devices without data loss. Accordingly, it requires N+2 drives to
190 store N drives worth of data.
192 The performance for RAID6 is slightly lower but comparable to RAID5 in
193 normal mode and single disk failure mode. It is very slow in dual
194 disk failure mode, however.
198 RAID10 provides a combination of RAID1 and RAID0, and sometimes known
199 as RAID1+0. Every datablock is duplicated some number of times, and
200 the resulting collection of datablocks are distributed over multiple
203 When configuring a RAID10 array it is necessary to specify the number
204 of replicas of each data block that are required (this will normally
205 be 2) and whether the replicas should be 'near' or 'far'.
207 When 'near' replicas are chosen, the multiple copies of a given chunk
208 are laid out consecutively across the stripes of the array, so the two
209 copies of a datablock will likely be at the same offset on two
212 When 'far' replicas are chosen, the multiple copies of a given chunk
213 are laid out quite distant from each other. The first copy of all
214 data blocks will be striped across the early part of all drives in
215 RAID0 fashion, and then the next copy of all blocks will be striped
216 across a later section of all drives, always ensuring that all copies
217 of any given block are on different drives.
219 The 'far' arrangement can give sequential read performance equal to
220 that of a RAID0 array, but at the cost of degraded write performance.
222 It should be noted that the number of devices in a RAID10 array need
223 not be a multiple of the number of replica of each data block, those
224 there must be at least as many devices as replicas.
226 If, for example, an array is created with 5 devices and 2 replicas,
227 then space equivalent to 2.5 of the devices will be available, and
228 every block will be stored on two different devices.
230 Finally, it is possible to have an array with both 'near' and 'far'
231 copies. If and array is configured with 2 near copies and 2 far
232 copies, then there will be a total of 4 copies of each block, each on
233 a different drive. This is an artifact of the implementation and is
234 unlikely to be of real value.
238 MULTIPATH is not really a RAID at all as there is only one real device
239 in a MULTIPATH md array. However there are multiple access points
240 (paths) to this device, and one of these paths might fail, so there
241 are some similarities.
243 A MULTIPATH array is composed of a number of logically different
244 devices, often fibre channel interfaces, that all refer the the same
245 real device. If one of these interfaces fails (e.g. due to cable
246 problems), the multipath driver will attempt to redirect requests to
250 The FAULTY md module is provided for testing purposes. A faulty array
251 has exactly one component device and is normally assembled without a
252 superblock, so the md array created provides direct access to all of
253 the data in the component device.
255 The FAULTY module may be requested to simulate faults to allow testing
256 of other md levels or of filesystems. Faults can be chosen to trigger
257 on read requests or write requests, and can be transient (a subsequent
258 read/write at the address will probably succeed) or persistant
259 (subsequent read/write of the same address will fail). Further, read
260 faults can be "fixable" meaning that they persist until a write
261 request at the same address.
263 Fault types can be requested with a period. In this case the fault
264 will recur repeatedly after the given number of requests of the
265 relevant type. For example if persistent read faults have a period of
266 100, then every 100th read request would generate a fault, and the
267 faulty sector would be recorded so that subsequent reads on that
268 sector would also fail.
270 There is a limit to the number of faulty sectors that are remembered.
271 Faults generated after this limit is exhausted are treated as
274 The list of faulty sectors can be flushed, and the active list of
275 failure modes can be cleared.
279 When changes are made to a RAID1, RAID4, RAID5, RAID6, or RAID10 array
280 there is a possibility of inconsistency for short periods of time as
281 each update requires are least two block to be written to different
282 devices, and these writes probably wont happen at exactly the same
283 time. Thus if a system with one of these arrays is shutdown in the
284 middle of a write operation (e.g. due to power failure), the array may
287 To handle this situation, the md driver marks an array as "dirty"
288 before writing any data to it, and marks it as "clean" when the array
289 is being disabled, e.g. at shutdown. If the md driver finds an array
290 to be dirty at startup, it proceeds to correct any possibly
291 inconsistency. For RAID1, this involves copying the contents of the
292 first drive onto all other drives. For RAID4, RAID5 and RAID6 this
293 involves recalculating the parity for each stripe and making sure that
294 the parity block has the correct data. For RAID10 it involves copying
295 one of the replicas of each block onto all the others. This process,
296 known as "resynchronising" or "resync" is performed in the background.
297 The array can still be used, though possibly with reduced performance.
299 If a RAID4, RAID5 or RAID6 array is degraded (missing at least one
300 drive) when it is restarted after an unclean shutdown, it cannot
301 recalculate parity, and so it is possible that data might be
302 undetectably corrupted. The 2.4 md driver
304 alert the operator to this condition. The 2.5 md driver will fail to
305 start an array in this condition without manual intervention.
309 If the md driver detects any error on a device in a RAID1, RAID4,
310 RAID5, RAID6, or RAID10 array, it immediately disables that device
311 (marking it as faulty) and continues operation on the remaining
312 devices. If there is a spare drive, the driver will start recreating
313 on one of the spare drives the data what was on that failed drive,
314 either by copying a working drive in a RAID1 configuration, or by
315 doing calculations with the parity block on RAID4, RAID5 or RAID6, or
316 by finding a copying originals for RAID10.
318 While this recovery process is happening, the md driver will monitor
319 accesses to the array and will slow down the rate of recovery if other
320 activity is happening, so that normal access to the array will not be
321 unduly affected. When no other activity is happening, the recovery
322 process proceeds at full speed. The actual speed targets for the two
323 different situations can be controlled by the
327 control files mentioned below.
329 .SS BITMAP WRITE-INTENT LOGGING
333 supports a bitmap based write-intent log. If configured, the bitmap
334 is used to record which blocks of the array may be out of sync.
335 Before any write request is honoured, md will make sure that the
336 corresponding bit in the log is set. After a period of time with no
337 writes to an area of the array, the corresponding bit will be cleared.
339 This bitmap is used for two optimisations.
341 Firstly, after an unclear shutdown, the resync process will consult
342 the bitmap and only resync those blocks that correspond to bits in the
343 bitmap that are set. This can dramatically increase resync time.
345 Secondly, when a drive fails and is removed from the array, md stops
346 clearing bits in the intent log. If that same drive is re-added to
347 the array, md will notice and will only recover the sections of the
348 drive that are covered by bits in the intent log that are set. This
349 can allow a device to be temporarily removed and reinserted without
350 causing an enormous recovery cost.
352 The intent log can be stored in a file on a separate device, or it can
353 be stored near the superblocks of an array which has superblocks.
355 Subsequent versions of Linux will support hot-adding of bitmaps to
358 In 2.6.13, intent bitmaps are only supported with RAID1. Other levels
365 will support WRITE-BEHIND on RAID1 arrays.
367 This allows certain devices in the array to be flagged as
369 MD will only read from such devices if there is no
372 If a write-intent bitmap is also provided, write requests to
373 write-mostly devices will be treated as write-behind requests and md
374 will not wait for writes to those requests to complete before
375 reporting the write as complete to the filesystem.
377 This allows for a RAID1 with WRITE-BEHIND to be used to mirror data
378 over a slow link to a remove computer (providing the link isn't too
379 slow). The extra latency of the remote link will not slow down normal
380 operations, but the remote system will still have a reasonably
381 up-to-date copy of all data.
383 .SS KERNEL PARAMETERS
385 The md driver recognised three different kernel parameters.
388 This will disable the normal detection of md arrays that happens at
389 boot time. If a drive is partitioned with MS-DOS style partitions,
390 then if any of the 4 main partitions has a partition type of 0xFD,
391 then that partition will normally be inspected to see if it is part of
392 an MD array, and if any full arrays are found, they are started. This
393 kernel paramenter disables this behaviour.
396 .B raid=partitionable
399 These are available in 2.6 and later kernels only. They indicate that
400 autodetected MD arrays should be created as partitionable arrays, with
401 a different major device number to the original non-partitionable md
402 arrays. The device number is listed as
409 .BI md= n , dev , dev ,...
411 .BI md=d n , dev , dev ,...
412 This tells the md driver to assemble
414 from the listed devices. It is only necessary to start the device
415 holding the root filesystem this way. Other arrays are best started
416 once the system is booted.
420 immediately after the
422 indicates that a partitionable device (e.g.
424 should be created rather than the original non-partitionable device.
427 .BI md= n , l , c , i , dev...
428 This tells the md driver to assemble a legacy RAID0 or LINEAR array
429 without a superblock.
431 gives the md device number,
433 gives the level, 0 for RAID0 or -1 for LINEAR,
435 gives the chunk size as a base-2 logarithm offset by twelve, so 0
436 means 4K, 1 means 8K.
438 is ignored (legacy support).
443 Contains information about the status of currently running array.
445 .B /proc/sys/dev/raid/speed_limit_min
446 A readable and writable file that reflects the current goal rebuild
447 speed for times when non-rebuild activity is current on an array.
448 The speed is in Kibibytes per second, and is a per-device rate, not a
449 per-array rate (which means that an array with more disc will shuffle
450 more data for a given speed). The default is 100.
453 .B /proc/sys/dev/raid/speed_limit_max
454 A readable and writable file that reflects the current goal rebuild
455 speed for times when no non-rebuild activity is current on an array.
456 The default is 100,000.
projects / thirdparty / mdadm.git / blob