]>
Commit | Line | Data |
---|---|---|
519561f7 NB |
1 | .\" Copyright Neil Brown and others. |
2 | .\" This program is free software; you can redistribute it and/or modify | |
3 | .\" it under the terms of the GNU General Public License as published by | |
4 | .\" the Free Software Foundation; either version 2 of the License, or | |
5 | .\" (at your option) any later version. | |
6 | .\" See file COPYING in distribution for details. | |
56eb10c0 NB |
7 | .TH MD 4 |
8 | .SH NAME | |
93e790af | 9 | md \- Multiple Device driver aka Linux Software RAID |
56eb10c0 NB |
10 | .SH SYNOPSIS |
11 | .BI /dev/md n | |
12 | .br | |
13 | .BI /dev/md/ n | |
e0fe762a N |
14 | .br |
15 | .BR /dev/md/ name | |
56eb10c0 NB |
16 | .SH DESCRIPTION |
17 | The | |
18 | .B md | |
19 | driver provides virtual devices that are created from one or more | |
e0d19036 | 20 | independent underlying devices. This array of devices often contains |
02b76eea NB |
21 | redundancy and the devices are often disk drives, hence the acronym RAID |
22 | which stands for a Redundant Array of Independent Disks. | |
56eb10c0 NB |
23 | .PP |
24 | .B md | |
599e5a36 NB |
25 | supports RAID levels |
26 | 1 (mirroring), | |
27 | 4 (striped array with parity device), | |
28 | 5 (striped array with distributed parity information), | |
29 | 6 (striped array with distributed dual redundancy information), and | |
30 | 10 (striped and mirrored). | |
31 | If some number of underlying devices fails while using one of these | |
98c6faba NB |
32 | levels, the array will continue to function; this number is one for |
33 | RAID levels 4 and 5, two for RAID level 6, and all but one (N-1) for | |
93e790af | 34 | RAID level 1, and dependent on configuration for level 10. |
56eb10c0 NB |
35 | .PP |
36 | .B md | |
e0d19036 | 37 | also supports a number of pseudo RAID (non-redundant) configurations |
570c0542 NB |
38 | including RAID0 (striped array), LINEAR (catenated array), |
39 | MULTIPATH (a set of different interfaces to the same device), | |
40 | and FAULTY (a layer over a single device into which errors can be injected). | |
56eb10c0 | 41 | |
e0fe762a | 42 | .SS MD METADATA |
bcbb92d4 | 43 | Each device in an array may have some |
e0fe762a N |
44 | .I metadata |
45 | stored in the device. This metadata is sometimes called a | |
46 | .BR superblock . | |
47 | The metadata records information about the structure and state of the array. | |
570c0542 | 48 | This allows the array to be reliably re-assembled after a shutdown. |
56eb10c0 | 49 | |
570c0542 NB |
50 | From Linux kernel version 2.6.10, |
51 | .B md | |
e0fe762a | 52 | provides support for two different formats of metadata, and |
570c0542 NB |
53 | other formats can be added. Prior to this release, only one format is |
54 | supported. | |
55 | ||
b3f1c093 | 56 | The common format \(em known as version 0.90 \(em has |
570c0542 | 57 | a superblock that is 4K long and is written into a 64K aligned block that |
11a3e71d | 58 | starts at least 64K and less than 128K from the end of the device |
56eb10c0 NB |
59 | (i.e. to get the address of the superblock round the size of the |
60 | device down to a multiple of 64K and then subtract 64K). | |
11a3e71d | 61 | The available size of each device is the amount of space before the |
56eb10c0 NB |
62 | super block, so between 64K and 128K is lost when a device in |
63 | incorporated into an MD array. | |
93e790af | 64 | This superblock stores multi-byte fields in a processor-dependent |
570c0542 NB |
65 | manner, so arrays cannot easily be moved between computers with |
66 | different processors. | |
67 | ||
b3f1c093 | 68 | The new format \(em known as version 1 \(em has a superblock that is |
570c0542 NB |
69 | normally 1K long, but can be longer. It is normally stored between 8K |
70 | and 12K from the end of the device, on a 4K boundary, though | |
71 | variations can be stored at the start of the device (version 1.1) or 4K from | |
72 | the start of the device (version 1.2). | |
e0fe762a | 73 | This metadata format stores multibyte data in a |
93e790af | 74 | processor-independent format and supports up to hundreds of |
570c0542 | 75 | component devices (version 0.90 only supports 28). |
56eb10c0 | 76 | |
e0fe762a | 77 | The metadata contains, among other things: |
56eb10c0 NB |
78 | .TP |
79 | LEVEL | |
11a3e71d | 80 | The manner in which the devices are arranged into the array |
956a13fb | 81 | (LINEAR, RAID0, RAID1, RAID4, RAID5, RAID10, MULTIPATH). |
56eb10c0 NB |
82 | .TP |
83 | UUID | |
84 | a 128 bit Universally Unique Identifier that identifies the array that | |
93e790af | 85 | contains this device. |
56eb10c0 | 86 | |
e0fe762a | 87 | .PP |
2a940e36 NB |
88 | When a version 0.90 array is being reshaped (e.g. adding extra devices |
89 | to a RAID5), the version number is temporarily set to 0.91. This | |
90 | ensures that if the reshape process is stopped in the middle (e.g. by | |
91 | a system crash) and the machine boots into an older kernel that does | |
92 | not support reshaping, then the array will not be assembled (which | |
93 | would cause data corruption) but will be left untouched until a kernel | |
94 | that can complete the reshape processes is used. | |
95 | ||
e0fe762a | 96 | .SS ARRAYS WITHOUT METADATA |
570c0542 | 97 | While it is usually best to create arrays with superblocks so that |
93e790af SW |
98 | they can be assembled reliably, there are some circumstances when an |
99 | array without superblocks is preferred. These include: | |
570c0542 NB |
100 | .TP |
101 | LEGACY ARRAYS | |
11a3e71d NB |
102 | Early versions of the |
103 | .B md | |
956a13fb | 104 | driver only supported LINEAR and RAID0 configurations and did not use |
570c0542 NB |
105 | a superblock (which is less critical with these configurations). |
106 | While such arrays should be rebuilt with superblocks if possible, | |
11a3e71d | 107 | .B md |
570c0542 NB |
108 | continues to support them. |
109 | .TP | |
110 | FAULTY | |
111 | Being a largely transparent layer over a different device, the FAULTY | |
112 | personality doesn't gain anything from having a superblock. | |
113 | .TP | |
114 | MULTIPATH | |
115 | It is often possible to detect devices which are different paths to | |
116 | the same storage directly rather than having a distinctive superblock | |
117 | written to the device and searched for on all paths. In this case, | |
118 | a MULTIPATH array with no superblock makes sense. | |
119 | .TP | |
120 | RAID1 | |
956a13fb | 121 | In some configurations it might be desired to create a RAID1 |
93e790af | 122 | configuration that does not use a superblock, and to maintain the state of |
095407fa | 123 | the array elsewhere. While not encouraged for general use, it does |
addc80c4 | 124 | have special-purpose uses and is supported. |
11a3e71d | 125 | |
e0fe762a N |
126 | .SS ARRAYS WITH EXTERNAL METADATA |
127 | ||
128 | From release 2.6.28, the | |
129 | .I md | |
130 | driver supports arrays with externally managed metadata. That is, | |
1e49aaa0 | 131 | the metadata is not managed by the kernel but rather by a user-space |
e0fe762a N |
132 | program which is external to the kernel. This allows support for a |
133 | variety of metadata formats without cluttering the kernel with lots of | |
134 | details. | |
135 | .PP | |
136 | .I md | |
137 | is able to communicate with the user-space program through various | |
138 | sysfs attributes so that it can make appropriate changes to the | |
1b17b4e4 | 139 | metadata \- for example to mark a device as faulty. When necessary, |
e0fe762a N |
140 | .I md |
141 | will wait for the program to acknowledge the event by writing to a | |
142 | sysfs attribute. | |
143 | The manual page for | |
144 | .IR mdmon (8) | |
145 | contains more detail about this interaction. | |
146 | ||
147 | .SS CONTAINERS | |
148 | Many metadata formats use a single block of metadata to describe a | |
149 | number of different arrays which all use the same set of devices. | |
150 | In this case it is helpful for the kernel to know about the full set | |
151 | of devices as a whole. This set is known to md as a | |
152 | .IR container . | |
153 | A container is an | |
154 | .I md | |
155 | array with externally managed metadata and with device offset and size | |
156 | so that it just covers the metadata part of the devices. The | |
157 | remainder of each device is available to be incorporated into various | |
158 | arrays. | |
159 | ||
56eb10c0 | 160 | .SS LINEAR |
11a3e71d | 161 | |
956a13fb | 162 | A LINEAR array simply catenates the available space on each |
93e790af | 163 | drive to form one large virtual drive. |
11a3e71d NB |
164 | |
165 | One advantage of this arrangement over the more common RAID0 | |
166 | arrangement is that the array may be reconfigured at a later time with | |
93e790af SW |
167 | an extra drive, so the array is made bigger without disturbing the |
168 | data that is on the array. This can even be done on a live | |
11a3e71d NB |
169 | array. |
170 | ||
599e5a36 NB |
171 | If a chunksize is given with a LINEAR array, the usable space on each |
172 | device is rounded down to a multiple of this chunksize. | |
11a3e71d | 173 | |
56eb10c0 | 174 | .SS RAID0 |
11a3e71d NB |
175 | |
176 | A RAID0 array (which has zero redundancy) is also known as a | |
177 | striped array. | |
e0d19036 | 178 | A RAID0 array is configured at creation with a |
bcbb92d4 | 179 | .B "Chunk Size" |
e0fe762a N |
180 | which must be a power of two (prior to Linux 2.6.31), and at least 4 |
181 | kibibytes. | |
e0d19036 | 182 | |
2d465520 | 183 | The RAID0 driver assigns the first chunk of the array to the first |
e0d19036 | 184 | device, the second chunk to the second device, and so on until all |
e0fe762a | 185 | drives have been assigned one chunk. This collection of chunks forms a |
e0d19036 | 186 | .BR stripe . |
93e790af | 187 | Further chunks are gathered into stripes in the same way, and are |
e0d19036 NB |
188 | assigned to the remaining space in the drives. |
189 | ||
2d465520 NB |
190 | If devices in the array are not all the same size, then once the |
191 | smallest device has been exhausted, the RAID0 driver starts | |
e0d19036 NB |
192 | collecting chunks into smaller stripes that only span the drives which |
193 | still have remaining space. | |
194 | ||
195 | ||
56eb10c0 | 196 | .SS RAID1 |
e0d19036 NB |
197 | |
198 | A RAID1 array is also known as a mirrored set (though mirrors tend to | |
5787fa49 | 199 | provide reflected images, which RAID1 does not) or a plex. |
e0d19036 NB |
200 | |
201 | Once initialised, each device in a RAID1 array contains exactly the | |
202 | same data. Changes are written to all devices in parallel. Data is | |
203 | read from any one device. The driver attempts to distribute read | |
204 | requests across all devices to maximise performance. | |
205 | ||
206 | All devices in a RAID1 array should be the same size. If they are | |
207 | not, then only the amount of space available on the smallest device is | |
93e790af | 208 | used (any extra space on other devices is wasted). |
e0d19036 | 209 | |
3dacb890 IP |
210 | Note that the read balancing done by the driver does not make the RAID1 |
211 | performance profile be the same as for RAID0; a single stream of | |
212 | sequential input will not be accelerated (e.g. a single dd), but | |
213 | multiple sequential streams or a random workload will use more than one | |
214 | spindle. In theory, having an N-disk RAID1 will allow N sequential | |
215 | threads to read from all disks. | |
216 | ||
e0fe762a | 217 | Individual devices in a RAID1 can be marked as "write-mostly". |
1b17b4e4 | 218 | These drives are excluded from the normal read balancing and will only |
e0fe762a N |
219 | be read from when there is no other option. This can be useful for |
220 | devices connected over a slow link. | |
221 | ||
56eb10c0 | 222 | .SS RAID4 |
e0d19036 NB |
223 | |
224 | A RAID4 array is like a RAID0 array with an extra device for storing | |
aa88f531 NB |
225 | parity. This device is the last of the active devices in the |
226 | array. Unlike RAID0, RAID4 also requires that all stripes span all | |
e0d19036 NB |
227 | drives, so extra space on devices that are larger than the smallest is |
228 | wasted. | |
229 | ||
93e790af | 230 | When any block in a RAID4 array is modified, the parity block for that |
e0d19036 NB |
231 | stripe (i.e. the block in the parity device at the same device offset |
232 | as the stripe) is also modified so that the parity block always | |
93e790af | 233 | contains the "parity" for the whole stripe. I.e. its content is |
e0d19036 NB |
234 | equivalent to the result of performing an exclusive-or operation |
235 | between all the data blocks in the stripe. | |
236 | ||
237 | This allows the array to continue to function if one device fails. | |
238 | The data that was on that device can be calculated as needed from the | |
239 | parity block and the other data blocks. | |
240 | ||
56eb10c0 | 241 | .SS RAID5 |
e0d19036 NB |
242 | |
243 | RAID5 is very similar to RAID4. The difference is that the parity | |
244 | blocks for each stripe, instead of being on a single device, are | |
245 | distributed across all devices. This allows more parallelism when | |
93e790af | 246 | writing, as two different block updates will quite possibly affect |
e0d19036 NB |
247 | parity blocks on different devices so there is less contention. |
248 | ||
93e790af | 249 | This also allows more parallelism when reading, as read requests are |
e0d19036 NB |
250 | distributed over all the devices in the array instead of all but one. |
251 | ||
98c6faba NB |
252 | .SS RAID6 |
253 | ||
254 | RAID6 is similar to RAID5, but can handle the loss of any \fItwo\fP | |
255 | devices without data loss. Accordingly, it requires N+2 drives to | |
256 | store N drives worth of data. | |
257 | ||
258 | The performance for RAID6 is slightly lower but comparable to RAID5 in | |
259 | normal mode and single disk failure mode. It is very slow in dual | |
260 | disk failure mode, however. | |
261 | ||
599e5a36 NB |
262 | .SS RAID10 |
263 | ||
93e790af | 264 | RAID10 provides a combination of RAID1 and RAID0, and is sometimes known |
599e5a36 NB |
265 | as RAID1+0. Every datablock is duplicated some number of times, and |
266 | the resulting collection of datablocks are distributed over multiple | |
267 | drives. | |
268 | ||
93e790af | 269 | When configuring a RAID10 array, it is necessary to specify the number |
8dc92b41 CAM |
270 | of replicas of each data block that are required (this will usually |
271 | be\ 2) and whether their layout should be "near", "far" or "offset" | |
272 | (with "offset" being available since Linux\ 2.6.18). | |
273 | ||
274 | .B About the RAID10 Layout Examples: | |
275 | .br | |
276 | The examples below visualise the chunk distribution on the underlying | |
277 | devices for the respective layout. | |
278 | ||
279 | For simplicity it is assumed that the size of the chunks equals the | |
280 | size of the blocks of the underlying devices as well as those of the | |
281 | RAID10 device exported by the kernel (for example \fB/dev/md/\fPname). | |
282 | .br | |
283 | Therefore the chunks\ /\ chunk numbers map directly to the blocks\ /\ | |
284 | block addresses of the exported RAID10 device. | |
285 | ||
286 | Decimal numbers (0,\ 1, 2,\ ...) are the chunks of the RAID10 and due | |
287 | to the above assumption also the blocks and block addresses of the | |
288 | exported RAID10 device. | |
289 | .br | |
290 | Repeated numbers mean copies of a chunk\ /\ block (obviously on | |
291 | different underlying devices). | |
292 | .br | |
293 | Hexadecimal numbers (0x00,\ 0x01, 0x02,\ ...) are the block addresses | |
294 | of the underlying devices. | |
295 | ||
296 | .TP | |
297 | \fB "near" Layout\fP | |
298 | When "near" replicas are chosen, the multiple copies of a given chunk are laid | |
299 | out consecutively ("as close to each other as possible") across the stripes of | |
300 | the array. | |
301 | ||
302 | With an even number of devices, they will likely (unless some misalignment is | |
303 | present) lay at the very same offset on the different devices. | |
304 | .br | |
305 | This is as the "classic" RAID1+0; that is two groups of mirrored devices (in the | |
306 | example below the groups Device\ #1\ /\ #2 and Device\ #3\ /\ #4 are each a | |
307 | RAID1) both in turn forming a striped RAID0. | |
308 | ||
309 | .ne 10 | |
310 | .B Example with 2\ copies per chunk and an even number\ (4) of devices: | |
311 | .TS | |
312 | tab(;); | |
313 | C - - - - | |
314 | C | C | C | C | C | | |
315 | | - | - | - | - | - | | |
316 | | C | C | C | C | C | | |
317 | | C | C | C | C | C | | |
318 | | C | C | C | C | C | | |
319 | | C | C | C | C | C | | |
320 | | C | C | C | C | C | | |
321 | | C | C | C | C | C | | |
322 | | - | - | - | - | - | | |
323 | C C S C S | |
324 | C C S C S | |
325 | C C S S S | |
326 | C C S S S. | |
327 | ; | |
328 | ;Device #1;Device #2;Device #3;Device #4 | |
329 | 0x00;0;0;1;1 | |
330 | 0x01;2;2;3;3 | |
331 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\. | |
332 | :;:;:;:;: | |
333 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\. | |
334 | 0x80;254;254;255;255 | |
335 | ;\\---------v---------/;\\---------v---------/ | |
336 | ;RAID1;RAID1 | |
337 | ;\\---------------------v---------------------/ | |
338 | ;RAID0 | |
339 | .TE | |
340 | ||
341 | .ne 10 | |
342 | .B Example with 2\ copies per chunk and an odd number\ (5) of devices: | |
343 | .TS | |
344 | tab(;); | |
345 | C - - - - - | |
346 | C | C | C | C | C | C | | |
347 | | - | - | - | - | - | - | | |
348 | | C | C | C | C | C | C | | |
349 | | C | C | C | C | C | C | | |
350 | | C | C | C | C | C | C | | |
351 | | C | C | C | C | C | C | | |
352 | | C | C | C | C | C | C | | |
353 | | C | C | C | C | C | C | | |
354 | | - | - | - | - | - | - | | |
355 | C. | |
356 | ; | |
f99a9e15 | 357 | ;Dev #1;Dev #2;Dev #3;Dev #4;Dev #5 |
8dc92b41 CAM |
358 | 0x00;0;0;1;1;2 |
359 | 0x01;2;3;3;4;4 | |
360 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\. | |
361 | :;:;:;:;:;: | |
362 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\. | |
363 | 0x80;317;318;318;319;319 | |
364 | ; | |
365 | .TE | |
366 | ||
367 | .TP | |
368 | \fB "far" Layout\fP | |
369 | When "far" replicas are chosen, the multiple copies of a given chunk | |
370 | are laid out quite distant ("as far as reasonably possible") from each | |
371 | other. | |
372 | ||
373 | First a complete sequence of all data blocks (that is all the data one | |
374 | sees on the exported RAID10 block device) is striped over the | |
375 | devices. Then another (though "shifted") complete sequence of all data | |
376 | blocks; and so on (in the case of more than 2\ copies per chunk). | |
377 | ||
378 | The "shift" needed to prevent placing copies of the same chunks on the | |
379 | same devices is actually a cyclic permutation with offset\ 1 of each | |
380 | of the stripes within a complete sequence of chunks. | |
381 | .br | |
382 | The offset\ 1 is relative to the previous complete sequence of chunks, | |
383 | so in case of more than 2\ copies per chunk one gets the following | |
384 | offsets: | |
385 | .br | |
386 | 1.\ complete sequence of chunks: offset\ =\ \ 0 | |
387 | .br | |
388 | 2.\ complete sequence of chunks: offset\ =\ \ 1 | |
389 | .br | |
390 | 3.\ complete sequence of chunks: offset\ =\ \ 2 | |
391 | .br | |
392 | : | |
393 | .br | |
394 | n.\ complete sequence of chunks: offset\ =\ n-1 | |
395 | ||
396 | .ne 10 | |
397 | .B Example with 2\ copies per chunk and an even number\ (4) of devices: | |
398 | .TS | |
399 | tab(;); | |
400 | C - - - - | |
401 | C | C | C | C | C | | |
402 | | - | - | - | - | - | | |
403 | | C | C | C | C | C | L | |
404 | | C | C | C | C | C | L | |
405 | | C | C | C | C | C | L | |
406 | | C | C | C | C | C | L | |
407 | | C | C | C | C | C | L | |
408 | | C | C | C | C | C | L | |
409 | | C | C | C | C | C | L | |
410 | | C | C | C | C | C | L | |
411 | | C | C | C | C | C | L | |
412 | | C | C | C | C | C | L | |
413 | | C | C | C | C | C | L | |
414 | | C | C | C | C | C | L | |
415 | | - | - | - | - | - | | |
416 | C. | |
417 | ; | |
418 | ;Device #1;Device #2;Device #3;Device #4 | |
419 | ; | |
420 | 0x00;0;1;2;3;\\ | |
421 | 0x01;4;5;6;7;> [#] | |
422 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
423 | :;:;:;:;:;: | |
424 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
425 | 0x40;252;253;254;255;/ | |
426 | 0x41;3;0;1;2;\\ | |
427 | 0x42;7;4;5;6;> [#]~ | |
428 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
429 | :;:;:;:;:;: | |
430 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
431 | 0x80;255;252;253;254;/ | |
432 | ; | |
433 | .TE | |
434 | ||
435 | .ne 10 | |
436 | .B Example with 2\ copies per chunk and an odd number\ (5) of devices: | |
437 | .TS | |
438 | tab(;); | |
439 | C - - - - - | |
440 | C | C | C | C | C | C | | |
441 | | - | - | - | - | - | - | | |
442 | | C | C | C | C | C | C | L | |
443 | | C | C | C | C | C | C | L | |
444 | | C | C | C | C | C | C | L | |
445 | | C | C | C | C | C | C | L | |
446 | | C | C | C | C | C | C | L | |
447 | | C | C | C | C | C | C | L | |
448 | | C | C | C | C | C | C | L | |
449 | | C | C | C | C | C | C | L | |
450 | | C | C | C | C | C | C | L | |
451 | | C | C | C | C | C | C | L | |
452 | | C | C | C | C | C | C | L | |
453 | | C | C | C | C | C | C | L | |
454 | | - | - | - | - | - | - | | |
455 | C. | |
456 | ; | |
f99a9e15 | 457 | ;Dev #1;Dev #2;Dev #3;Dev #4;Dev #5 |
8dc92b41 CAM |
458 | ; |
459 | 0x00;0;1;2;3;4;\\ | |
460 | 0x01;5;6;7;8;9;> [#] | |
461 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
462 | :;:;:;:;:;:;: | |
463 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
464 | 0x40;315;316;317;318;319;/ | |
465 | 0x41;4;0;1;2;3;\\ | |
466 | 0x42;9;5;6;7;8;> [#]~ | |
467 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
468 | :;:;:;:;:;:;: | |
469 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
470 | 0x80;319;315;316;317;318;/ | |
471 | ; | |
472 | .TE | |
473 | ||
474 | With [#]\ being the complete sequence of chunks and [#]~\ the cyclic permutation | |
475 | with offset\ 1 thereof (in the case of more than 2 copies per chunk there would | |
476 | be ([#]~)~,\ (([#]~)~)~,\ ...). | |
477 | ||
478 | The advantage of this layout is that MD can easily spread sequential reads over | |
479 | the devices, making them similar to RAID0 in terms of speed. | |
480 | .br | |
481 | The cost is more seeking for writes, making them substantially slower. | |
482 | ||
483 | .TP | |
484 | \fB"offset" Layout\fP | |
485 | When "offset" replicas are chosen, all the copies of a given chunk are | |
486 | striped consecutively ("offset by the stripe length after each other") | |
487 | over the devices. | |
488 | ||
489 | Explained in detail, <number of devices> consecutive chunks are | |
490 | striped over the devices, immediately followed by a "shifted" copy of | |
491 | these chunks (and by further such "shifted" copies in the case of more | |
492 | than 2\ copies per chunk). | |
493 | .br | |
494 | This pattern repeats for all further consecutive chunks of the | |
495 | exported RAID10 device (in other words: all further data blocks). | |
496 | ||
497 | The "shift" needed to prevent placing copies of the same chunks on the | |
498 | same devices is actually a cyclic permutation with offset\ 1 of each | |
499 | of the striped copies of <number of devices> consecutive chunks. | |
500 | .br | |
501 | The offset\ 1 is relative to the previous striped copy of <number of | |
502 | devices> consecutive chunks, so in case of more than 2\ copies per | |
503 | chunk one gets the following offsets: | |
504 | .br | |
505 | 1.\ <number of devices> consecutive chunks: offset\ =\ \ 0 | |
506 | .br | |
507 | 2.\ <number of devices> consecutive chunks: offset\ =\ \ 1 | |
508 | .br | |
509 | 3.\ <number of devices> consecutive chunks: offset\ =\ \ 2 | |
510 | .br | |
511 | : | |
512 | .br | |
513 | n.\ <number of devices> consecutive chunks: offset\ =\ n-1 | |
514 | ||
515 | .ne 10 | |
516 | .B Example with 2\ copies per chunk and an even number\ (4) of devices: | |
517 | .TS | |
518 | tab(;); | |
519 | C - - - - | |
520 | C | C | C | C | C | | |
521 | | - | - | - | - | - | | |
522 | | C | C | C | C | C | L | |
523 | | C | C | C | C | C | L | |
524 | | C | C | C | C | C | L | |
525 | | C | C | C | C | C | L | |
526 | | C | C | C | C | C | L | |
527 | | C | C | C | C | C | L | |
528 | | C | C | C | C | C | L | |
529 | | C | C | C | C | C | L | |
530 | | C | C | C | C | C | L | |
531 | | - | - | - | - | - | | |
532 | C. | |
533 | ; | |
534 | ;Device #1;Device #2;Device #3;Device #4 | |
535 | ; | |
536 | 0x00;0;1;2;3;) AA | |
537 | 0x01;3;0;1;2;) AA~ | |
538 | 0x02;4;5;6;7;) AB | |
539 | 0x03;7;4;5;6;) AB~ | |
540 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;) \.\.\. | |
541 | :;:;:;:;:; : | |
542 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;) \.\.\. | |
543 | 0x79;251;252;253;254;) EX | |
544 | 0x80;254;251;252;253;) EX~ | |
545 | ; | |
546 | .TE | |
547 | ||
548 | .ne 10 | |
549 | .B Example with 2\ copies per chunk and an odd number\ (5) of devices: | |
550 | .TS | |
551 | tab(;); | |
552 | C - - - - - | |
553 | C | C | C | C | C | C | | |
554 | | - | - | - | - | - | - | | |
555 | | C | C | C | C | C | C | L | |
556 | | C | C | C | C | C | C | L | |
557 | | C | C | C | C | C | C | L | |
558 | | C | C | C | C | C | C | L | |
559 | | C | C | C | C | C | C | L | |
560 | | C | C | C | C | C | C | L | |
561 | | C | C | C | C | C | C | L | |
562 | | C | C | C | C | C | C | L | |
563 | | C | C | C | C | C | C | L | |
564 | | - | - | - | - | - | - | | |
565 | C. | |
566 | ; | |
f99a9e15 | 567 | ;Dev #1;Dev #2;Dev #3;Dev #4;Dev #5 |
8dc92b41 CAM |
568 | ; |
569 | 0x00;0;1;2;3;4;) AA | |
570 | 0x01;4;0;1;2;3;) AA~ | |
571 | 0x02;5;6;7;8;9;) AB | |
572 | 0x03;9;5;6;7;8;) AB~ | |
573 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;) \.\.\. | |
574 | :;:;:;:;:;:; : | |
575 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;) \.\.\. | |
576 | 0x79;314;315;316;317;318;) EX | |
577 | 0x80;318;314;315;316;317;) EX~ | |
578 | ; | |
579 | .TE | |
580 | ||
581 | With AA,\ AB,\ ..., AZ,\ BA,\ ... being the sets of <number of devices> consecutive | |
582 | chunks and AA~,\ AB~,\ ..., AZ~,\ BA~,\ ... the cyclic permutations with offset\ 1 | |
583 | thereof (in the case of more than 2 copies per chunk there would be (AA~)~,\ ... | |
584 | as well as ((AA~)~)~,\ ... and so on). | |
585 | ||
586 | This should give similar read characteristics to "far" if a suitably large chunk | |
587 | size is used, but without as much seeking for writes. | |
588 | .PP | |
589 | ||
b578481c | 590 | |
599e5a36 | 591 | It should be noted that the number of devices in a RAID10 array need |
93e790af | 592 | not be a multiple of the number of replica of each data block; however, |
599e5a36 NB |
593 | there must be at least as many devices as replicas. |
594 | ||
595 | If, for example, an array is created with 5 devices and 2 replicas, | |
596 | then space equivalent to 2.5 of the devices will be available, and | |
597 | every block will be stored on two different devices. | |
598 | ||
8dc92b41 | 599 | Finally, it is possible to have an array with both "near" and "far" |
93e790af | 600 | copies. If an array is configured with 2 near copies and 2 far |
599e5a36 NB |
601 | copies, then there will be a total of 4 copies of each block, each on |
602 | a different drive. This is an artifact of the implementation and is | |
603 | unlikely to be of real value. | |
604 | ||
bf40ab85 | 605 | .SS MULTIPATH |
e0d19036 NB |
606 | |
607 | MULTIPATH is not really a RAID at all as there is only one real device | |
608 | in a MULTIPATH md array. However there are multiple access points | |
609 | (paths) to this device, and one of these paths might fail, so there | |
610 | are some similarities. | |
611 | ||
a9d69660 | 612 | A MULTIPATH array is composed of a number of logically different |
2d465520 NB |
613 | devices, often fibre channel interfaces, that all refer the the same |
614 | real device. If one of these interfaces fails (e.g. due to cable | |
956a13fb | 615 | problems), the MULTIPATH driver will attempt to redirect requests to |
e0fe762a N |
616 | another interface. |
617 | ||
618 | The MULTIPATH drive is not receiving any ongoing development and | |
619 | should be considered a legacy driver. The device-mapper based | |
620 | multipath drivers should be preferred for new installations. | |
e0d19036 | 621 | |
b5e64645 | 622 | .SS FAULTY |
956a13fb | 623 | The FAULTY md module is provided for testing purposes. A FAULTY array |
b5e64645 NB |
624 | has exactly one component device and is normally assembled without a |
625 | superblock, so the md array created provides direct access to all of | |
626 | the data in the component device. | |
627 | ||
628 | The FAULTY module may be requested to simulate faults to allow testing | |
a9d69660 | 629 | of other md levels or of filesystems. Faults can be chosen to trigger |
b5e64645 | 630 | on read requests or write requests, and can be transient (a subsequent |
addc80c4 | 631 | read/write at the address will probably succeed) or persistent |
b5e64645 NB |
632 | (subsequent read/write of the same address will fail). Further, read |
633 | faults can be "fixable" meaning that they persist until a write | |
634 | request at the same address. | |
635 | ||
93e790af | 636 | Fault types can be requested with a period. In this case, the fault |
a9d69660 NB |
637 | will recur repeatedly after the given number of requests of the |
638 | relevant type. For example if persistent read faults have a period of | |
639 | 100, then every 100th read request would generate a fault, and the | |
b5e64645 NB |
640 | faulty sector would be recorded so that subsequent reads on that |
641 | sector would also fail. | |
642 | ||
643 | There is a limit to the number of faulty sectors that are remembered. | |
644 | Faults generated after this limit is exhausted are treated as | |
645 | transient. | |
646 | ||
a9d69660 | 647 | The list of faulty sectors can be flushed, and the active list of |
b5e64645 | 648 | failure modes can be cleared. |
e0d19036 NB |
649 | |
650 | .SS UNCLEAN SHUTDOWN | |
651 | ||
599e5a36 NB |
652 | When changes are made to a RAID1, RAID4, RAID5, RAID6, or RAID10 array |
653 | there is a possibility of inconsistency for short periods of time as | |
93e790af SW |
654 | each update requires at least two block to be written to different |
655 | devices, and these writes probably won't happen at exactly the same | |
599e5a36 NB |
656 | time. Thus if a system with one of these arrays is shutdown in the |
657 | middle of a write operation (e.g. due to power failure), the array may | |
658 | not be consistent. | |
e0d19036 | 659 | |
2d465520 | 660 | To handle this situation, the md driver marks an array as "dirty" |
e0d19036 | 661 | before writing any data to it, and marks it as "clean" when the array |
98c6faba NB |
662 | is being disabled, e.g. at shutdown. If the md driver finds an array |
663 | to be dirty at startup, it proceeds to correct any possibly | |
664 | inconsistency. For RAID1, this involves copying the contents of the | |
665 | first drive onto all other drives. For RAID4, RAID5 and RAID6 this | |
666 | involves recalculating the parity for each stripe and making sure that | |
599e5a36 NB |
667 | the parity block has the correct data. For RAID10 it involves copying |
668 | one of the replicas of each block onto all the others. This process, | |
669 | known as "resynchronising" or "resync" is performed in the background. | |
670 | The array can still be used, though possibly with reduced performance. | |
98c6faba NB |
671 | |
672 | If a RAID4, RAID5 or RAID6 array is degraded (missing at least one | |
93e790af | 673 | drive, two for RAID6) when it is restarted after an unclean shutdown, it cannot |
98c6faba NB |
674 | recalculate parity, and so it is possible that data might be |
675 | undetectably corrupted. The 2.4 md driver | |
e0d19036 | 676 | .B does not |
addc80c4 NB |
677 | alert the operator to this condition. The 2.6 md driver will fail to |
678 | start an array in this condition without manual intervention, though | |
35cc5be4 | 679 | this behaviour can be overridden by a kernel parameter. |
e0d19036 NB |
680 | |
681 | .SS RECOVERY | |
682 | ||
addc80c4 | 683 | If the md driver detects a write error on a device in a RAID1, RAID4, |
599e5a36 NB |
684 | RAID5, RAID6, or RAID10 array, it immediately disables that device |
685 | (marking it as faulty) and continues operation on the remaining | |
93e790af SW |
686 | devices. If there are spare drives, the driver will start recreating |
687 | on one of the spare drives the data which was on that failed drive, | |
599e5a36 NB |
688 | either by copying a working drive in a RAID1 configuration, or by |
689 | doing calculations with the parity block on RAID4, RAID5 or RAID6, or | |
93e790af | 690 | by finding and copying originals for RAID10. |
e0d19036 | 691 | |
addc80c4 NB |
692 | In kernels prior to about 2.6.15, a read error would cause the same |
693 | effect as a write error. In later kernels, a read-error will instead | |
694 | cause md to attempt a recovery by overwriting the bad block. i.e. it | |
695 | will find the correct data from elsewhere, write it over the block | |
696 | that failed, and then try to read it back again. If either the write | |
697 | or the re-read fail, md will treat the error the same way that a write | |
93e790af | 698 | error is treated, and will fail the whole device. |
addc80c4 | 699 | |
2d465520 | 700 | While this recovery process is happening, the md driver will monitor |
e0d19036 NB |
701 | accesses to the array and will slow down the rate of recovery if other |
702 | activity is happening, so that normal access to the array will not be | |
703 | unduly affected. When no other activity is happening, the recovery | |
704 | process proceeds at full speed. The actual speed targets for the two | |
705 | different situations can be controlled by the | |
706 | .B speed_limit_min | |
707 | and | |
708 | .B speed_limit_max | |
709 | control files mentioned below. | |
710 | ||
1cc44574 N |
711 | .SS SCRUBBING AND MISMATCHES |
712 | ||
713 | As storage devices can develop bad blocks at any time it is valuable | |
714 | to regularly read all blocks on all devices in an array so as to catch | |
715 | such bad blocks early. This process is called | |
716 | .IR scrubbing . | |
717 | ||
718 | md arrays can be scrubbed by writing either | |
719 | .I check | |
720 | or | |
721 | .I repair | |
722 | to the file | |
723 | .I md/sync_action | |
724 | in the | |
725 | .I sysfs | |
726 | directory for the device. | |
727 | ||
c93e9d68 | 728 | Requesting a scrub will cause |
1cc44574 N |
729 | .I md |
730 | to read every block on every device in the array, and check that the | |
c93e9d68 N |
731 | data is consistent. For RAID1 and RAID10, this means checking that the copies |
732 | are identical. For RAID4, RAID5, RAID6 this means checking that the | |
733 | parity block is (or blocks are) correct. | |
1cc44574 N |
734 | |
735 | If a read error is detected during this process, the normal read-error | |
736 | handling causes correct data to be found from other devices and to be | |
737 | written back to the faulty device. In many case this will | |
738 | effectively | |
739 | .I fix | |
740 | the bad block. | |
741 | ||
742 | If all blocks read successfully but are found to not be consistent, | |
743 | then this is regarded as a | |
744 | .IR mismatch . | |
745 | ||
746 | If | |
747 | .I check | |
748 | was used, then no action is taken to handle the mismatch, it is simply | |
749 | recorded. | |
750 | If | |
751 | .I repair | |
752 | was used, then a mismatch will be repaired in the same way that | |
753 | .I resync | |
c93e9d68 | 754 | repairs arrays. For RAID5/RAID6 new parity blocks are written. For RAID1/RAID10, |
1cc44574 N |
755 | all but one block are overwritten with the content of that one block. |
756 | ||
757 | A count of mismatches is recorded in the | |
758 | .I sysfs | |
759 | file | |
760 | .IR md/mismatch_cnt . | |
761 | This is set to zero when a | |
c93e9d68 | 762 | scrub starts and is incremented whenever a sector is |
1cc44574 N |
763 | found that is a mismatch. |
764 | .I md | |
765 | normally works in units much larger than a single sector and when it | |
1e49aaa0 | 766 | finds a mismatch, it does not determine exactly how many actual sectors were |
c93e9d68 N |
767 | affected but simply adds the number of sectors in the IO unit that was |
768 | used. So a value of 128 could simply mean that a single 64KB check | |
769 | found an error (128 x 512bytes = 64KB). | |
770 | ||
771 | If an array is created by | |
772 | .I mdadm | |
773 | with | |
1cc44574 N |
774 | .I \-\-assume\-clean |
775 | then a subsequent check could be expected to find some mismatches. | |
776 | ||
777 | On a truly clean RAID5 or RAID6 array, any mismatches should indicate | |
778 | a hardware problem at some level - software issues should never cause | |
779 | such a mismatch. | |
780 | ||
781 | However on RAID1 and RAID10 it is possible for software issues to | |
782 | cause a mismatch to be reported. This does not necessarily mean that | |
783 | the data on the array is corrupted. It could simply be that the | |
784 | system does not care what is stored on that part of the array - it is | |
785 | unused space. | |
786 | ||
787 | The most likely cause for an unexpected mismatch on RAID1 or RAID10 | |
788 | occurs if a swap partition or swap file is stored on the array. | |
789 | ||
790 | When the swap subsystem wants to write a page of memory out, it flags | |
791 | the page as 'clean' in the memory manager and requests the swap device | |
792 | to write it out. It is quite possible that the memory will be | |
793 | changed while the write-out is happening. In that case the 'clean' | |
794 | flag will be found to be clear when the write completes and so the | |
795 | swap subsystem will simply forget that the swapout had been attempted, | |
c93e9d68 | 796 | and will possibly choose a different page to write out. |
1cc44574 | 797 | |
c93e9d68 | 798 | If the swap device was on RAID1 (or RAID10), then the data is sent |
1cc44574 | 799 | from memory to a device twice (or more depending on the number of |
c93e9d68 N |
800 | devices in the array). Thus it is possible that the memory gets changed |
801 | between the times it is sent, so different data can be written to | |
802 | the different devices in the array. This will be detected by | |
1cc44574 N |
803 | .I check |
804 | as a mismatch. However it does not reflect any corruption as the | |
805 | block where this mismatch occurs is being treated by the swap system as | |
806 | being empty, and the data will never be read from that block. | |
807 | ||
808 | It is conceivable for a similar situation to occur on non-swap files, | |
809 | though it is less likely. | |
810 | ||
811 | Thus the | |
812 | .I mismatch_cnt | |
813 | value can not be interpreted very reliably on RAID1 or RAID10, | |
814 | especially when the device is used for swap. | |
815 | ||
816 | ||
599e5a36 NB |
817 | .SS BITMAP WRITE-INTENT LOGGING |
818 | ||
819 | From Linux 2.6.13, | |
820 | .I md | |
821 | supports a bitmap based write-intent log. If configured, the bitmap | |
822 | is used to record which blocks of the array may be out of sync. | |
823 | Before any write request is honoured, md will make sure that the | |
824 | corresponding bit in the log is set. After a period of time with no | |
825 | writes to an area of the array, the corresponding bit will be cleared. | |
826 | ||
827 | This bitmap is used for two optimisations. | |
828 | ||
1afe1167 | 829 | Firstly, after an unclean shutdown, the resync process will consult |
599e5a36 | 830 | the bitmap and only resync those blocks that correspond to bits in the |
1afe1167 | 831 | bitmap that are set. This can dramatically reduce resync time. |
599e5a36 NB |
832 | |
833 | Secondly, when a drive fails and is removed from the array, md stops | |
834 | clearing bits in the intent log. If that same drive is re-added to | |
835 | the array, md will notice and will only recover the sections of the | |
836 | drive that are covered by bits in the intent log that are set. This | |
837 | can allow a device to be temporarily removed and reinserted without | |
838 | causing an enormous recovery cost. | |
839 | ||
840 | The intent log can be stored in a file on a separate device, or it can | |
841 | be stored near the superblocks of an array which has superblocks. | |
842 | ||
93e790af | 843 | It is possible to add an intent log to an active array, or remove an |
addc80c4 | 844 | intent log if one is present. |
599e5a36 NB |
845 | |
846 | In 2.6.13, intent bitmaps are only supported with RAID1. Other levels | |
addc80c4 | 847 | with redundancy are supported from 2.6.15. |
599e5a36 | 848 | |
968d2a33 | 849 | .SS BAD BLOCK LIST |
bf95d0f3 N |
850 | |
851 | From Linux 3.5 each device in an | |
852 | .I md | |
853 | array can store a list of known-bad-blocks. This list is 4K in size | |
854 | and usually positioned at the end of the space between the superblock | |
855 | and the data. | |
856 | ||
857 | When a block cannot be read and cannot be repaired by writing data | |
858 | recovered from other devices, the address of the block is stored in | |
968d2a33 | 859 | the bad block list. Similarly if an attempt to write a block fails, |
bf95d0f3 N |
860 | the address will be recorded as a bad block. If attempting to record |
861 | the bad block fails, the whole device will be marked faulty. | |
862 | ||
863 | Attempting to read from a known bad block will cause a read error. | |
864 | Attempting to write to a known bad block will be ignored if any write | |
865 | errors have been reported by the device. If there have been no write | |
866 | errors then the data will be written to the known bad block and if | |
867 | that succeeds, the address will be removed from the list. | |
868 | ||
869 | This allows an array to fail more gracefully - a few blocks on | |
870 | different devices can be faulty without taking the whole array out of | |
871 | action. | |
872 | ||
968d2a33 | 873 | The list is particularly useful when recovering to a spare. If a few blocks |
bf95d0f3 | 874 | cannot be read from the other devices, the bulk of the recovery can |
968d2a33 | 875 | complete and those few bad blocks will be recorded in the bad block list. |
bf95d0f3 | 876 | |
28f83f6d SL |
877 | .SS RAID456 WRITE JOURNAL |
878 | ||
879 | Due to non-atomicity nature of RAID write operations, interruption of | |
880 | write operations (system crash, etc.) to RAID456 array can lead to | |
881 | inconsistent parity and data loss (so called RAID-5 write hole). | |
882 | ||
883 | To plug the write hole, from Linux 4.4 (to be confirmed), | |
884 | .I md | |
885 | supports write ahead journal for RAID456. When the array is created, | |
886 | an additional journal device can be added to the array through | |
887 | .IR write-journal | |
888 | option. The RAID write journal works similar to file system journals. | |
889 | Before writing to the data disks, md persists data AND parity of the | |
890 | stripe to the journal device. After crashes, md searches the journal | |
891 | device for incomplete write operations, and replay them to the data | |
892 | disks. | |
893 | ||
894 | When the journal device fails, the RAID array is forced to run in | |
895 | read-only mode. | |
896 | ||
599e5a36 NB |
897 | .SS WRITE-BEHIND |
898 | ||
899 | From Linux 2.6.14, | |
900 | .I md | |
addc80c4 | 901 | supports WRITE-BEHIND on RAID1 arrays. |
599e5a36 NB |
902 | |
903 | This allows certain devices in the array to be flagged as | |
904 | .IR write-mostly . | |
905 | MD will only read from such devices if there is no | |
906 | other option. | |
907 | ||
908 | If a write-intent bitmap is also provided, write requests to | |
909 | write-mostly devices will be treated as write-behind requests and md | |
910 | will not wait for writes to those requests to complete before | |
911 | reporting the write as complete to the filesystem. | |
912 | ||
913 | This allows for a RAID1 with WRITE-BEHIND to be used to mirror data | |
8f21823f | 914 | over a slow link to a remote computer (providing the link isn't too |
599e5a36 NB |
915 | slow). The extra latency of the remote link will not slow down normal |
916 | operations, but the remote system will still have a reasonably | |
917 | up-to-date copy of all data. | |
918 | ||
addc80c4 NB |
919 | .SS RESTRIPING |
920 | ||
921 | .IR Restriping , | |
922 | also known as | |
923 | .IR Reshaping , | |
924 | is the processes of re-arranging the data stored in each stripe into a | |
925 | new layout. This might involve changing the number of devices in the | |
93e790af | 926 | array (so the stripes are wider), changing the chunk size (so stripes |
addc80c4 | 927 | are deeper or shallower), or changing the arrangement of data and |
956a13fb | 928 | parity (possibly changing the RAID level, e.g. 1 to 5 or 5 to 6). |
addc80c4 | 929 | |
c64881d7 N |
930 | As of Linux 2.6.35, md can reshape a RAID4, RAID5, or RAID6 array to |
931 | have a different number of devices (more or fewer) and to have a | |
932 | different layout or chunk size. It can also convert between these | |
933 | different RAID levels. It can also convert between RAID0 and RAID10, | |
934 | and between RAID0 and RAID4 or RAID5. | |
935 | Other possibilities may follow in future kernels. | |
addc80c4 NB |
936 | |
937 | During any stripe process there is a 'critical section' during which | |
35cc5be4 | 938 | live data is being overwritten on disk. For the operation of |
956a13fb | 939 | increasing the number of drives in a RAID5, this critical section |
addc80c4 NB |
940 | covers the first few stripes (the number being the product of the old |
941 | and new number of devices). After this critical section is passed, | |
942 | data is only written to areas of the array which no longer hold live | |
b3f1c093 | 943 | data \(em the live data has already been located away. |
addc80c4 | 944 | |
c64881d7 N |
945 | For a reshape which reduces the number of devices, the 'critical |
946 | section' is at the end of the reshape process. | |
947 | ||
addc80c4 NB |
948 | md is not able to ensure data preservation if there is a crash |
949 | (e.g. power failure) during the critical section. If md is asked to | |
950 | start an array which failed during a critical section of restriping, | |
951 | it will fail to start the array. | |
952 | ||
953 | To deal with this possibility, a user-space program must | |
954 | .IP \(bu 4 | |
955 | Disable writes to that section of the array (using the | |
956 | .B sysfs | |
957 | interface), | |
958 | .IP \(bu 4 | |
93e790af | 959 | take a copy of the data somewhere (i.e. make a backup), |
addc80c4 | 960 | .IP \(bu 4 |
93e790af | 961 | allow the process to continue and invalidate the backup and restore |
addc80c4 NB |
962 | write access once the critical section is passed, and |
963 | .IP \(bu 4 | |
93e790af | 964 | provide for restoring the critical data before restarting the array |
addc80c4 NB |
965 | after a system crash. |
966 | .PP | |
967 | ||
968 | .B mdadm | |
93e790af | 969 | versions from 2.4 do this for growing a RAID5 array. |
addc80c4 NB |
970 | |
971 | For operations that do not change the size of the array, like simply | |
972 | increasing chunk size, or converting RAID5 to RAID6 with one extra | |
93e790af SW |
973 | device, the entire process is the critical section. In this case, the |
974 | restripe will need to progress in stages, as a section is suspended, | |
c64881d7 | 975 | backed up, restriped, and released. |
addc80c4 NB |
976 | |
977 | .SS SYSFS INTERFACE | |
93e790af | 978 | Each block device appears as a directory in |
addc80c4 | 979 | .I sysfs |
93e790af | 980 | (which is usually mounted at |
addc80c4 NB |
981 | .BR /sys ). |
982 | For MD devices, this directory will contain a subdirectory called | |
983 | .B md | |
984 | which contains various files for providing access to information about | |
985 | the array. | |
986 | ||
987 | This interface is documented more fully in the file | |
988 | .B Documentation/md.txt | |
989 | which is distributed with the kernel sources. That file should be | |
990 | consulted for full documentation. The following are just a selection | |
991 | of attribute files that are available. | |
992 | ||
993 | .TP | |
994 | .B md/sync_speed_min | |
995 | This value, if set, overrides the system-wide setting in | |
996 | .B /proc/sys/dev/raid/speed_limit_min | |
997 | for this array only. | |
998 | Writing the value | |
93e790af SW |
999 | .B "system" |
1000 | to this file will cause the system-wide setting to have effect. | |
addc80c4 NB |
1001 | |
1002 | .TP | |
1003 | .B md/sync_speed_max | |
1004 | This is the partner of | |
1005 | .B md/sync_speed_min | |
1006 | and overrides | |
1e49aaa0 | 1007 | .B /proc/sys/dev/raid/speed_limit_max |
addc80c4 NB |
1008 | described below. |
1009 | ||
1010 | .TP | |
1011 | .B md/sync_action | |
1012 | This can be used to monitor and control the resync/recovery process of | |
1013 | MD. | |
1014 | In particular, writing "check" here will cause the array to read all | |
1015 | data block and check that they are consistent (e.g. parity is correct, | |
1016 | or all mirror replicas are the same). Any discrepancies found are | |
1017 | .B NOT | |
1018 | corrected. | |
1019 | ||
1020 | A count of problems found will be stored in | |
1021 | .BR md/mismatch_count . | |
1022 | ||
1023 | Alternately, "repair" can be written which will cause the same check | |
1024 | to be performed, but any errors will be corrected. | |
1025 | ||
1026 | Finally, "idle" can be written to stop the check/repair process. | |
1027 | ||
1028 | .TP | |
1029 | .B md/stripe_cache_size | |
1030 | This is only available on RAID5 and RAID6. It records the size (in | |
1031 | pages per device) of the stripe cache which is used for synchronising | |
800053d6 DW |
1032 | all write operations to the array and all read operations if the array |
1033 | is degraded. The default is 256. Valid values are 17 to 32768. | |
addc80c4 | 1034 | Increasing this number can increase performance in some situations, at |
800053d6 DW |
1035 | some cost in system memory. Note, setting this value too high can |
1036 | result in an "out of memory" condition for the system. | |
1037 | ||
1038 | memory_consumed = system_page_size * nr_disks * stripe_cache_size | |
addc80c4 | 1039 | |
a5ee6dfb DW |
1040 | .TP |
1041 | .B md/preread_bypass_threshold | |
1042 | This is only available on RAID5 and RAID6. This variable sets the | |
1043 | number of times MD will service a full-stripe-write before servicing a | |
1044 | stripe that requires some "prereading". For fairness this defaults to | |
800053d6 DW |
1045 | 1. Valid values are 0 to stripe_cache_size. Setting this to 0 |
1046 | maximizes sequential-write throughput at the cost of fairness to threads | |
bcbb92d4 | 1047 | doing small or random writes. |
addc80c4 | 1048 | |
5787fa49 NB |
1049 | .SS KERNEL PARAMETERS |
1050 | ||
addc80c4 | 1051 | The md driver recognised several different kernel parameters. |
5787fa49 NB |
1052 | .TP |
1053 | .B raid=noautodetect | |
1054 | This will disable the normal detection of md arrays that happens at | |
1055 | boot time. If a drive is partitioned with MS-DOS style partitions, | |
1056 | then if any of the 4 main partitions has a partition type of 0xFD, | |
1057 | then that partition will normally be inspected to see if it is part of | |
1058 | an MD array, and if any full arrays are found, they are started. This | |
addc80c4 | 1059 | kernel parameter disables this behaviour. |
5787fa49 | 1060 | |
a9d69660 NB |
1061 | .TP |
1062 | .B raid=partitionable | |
1063 | .TP | |
1064 | .B raid=part | |
1065 | These are available in 2.6 and later kernels only. They indicate that | |
1066 | autodetected MD arrays should be created as partitionable arrays, with | |
1067 | a different major device number to the original non-partitionable md | |
1068 | arrays. The device number is listed as | |
1069 | .I mdp | |
1070 | in | |
1071 | .IR /proc/devices . | |
1072 | ||
addc80c4 NB |
1073 | .TP |
1074 | .B md_mod.start_ro=1 | |
e0fe762a N |
1075 | .TP |
1076 | .B /sys/module/md_mod/parameters/start_ro | |
addc80c4 NB |
1077 | This tells md to start all arrays in read-only mode. This is a soft |
1078 | read-only that will automatically switch to read-write on the first | |
1079 | write request. However until that write request, nothing is written | |
1080 | to any device by md, and in particular, no resync or recovery | |
1081 | operation is started. | |
1082 | ||
1083 | .TP | |
1084 | .B md_mod.start_dirty_degraded=1 | |
e0fe762a N |
1085 | .TP |
1086 | .B /sys/module/md_mod/parameters/start_dirty_degraded | |
addc80c4 NB |
1087 | As mentioned above, md will not normally start a RAID4, RAID5, or |
1088 | RAID6 that is both dirty and degraded as this situation can imply | |
1089 | hidden data loss. This can be awkward if the root filesystem is | |
93e790af | 1090 | affected. Using this module parameter allows such arrays to be started |
addc80c4 NB |
1091 | at boot time. It should be understood that there is a real (though |
1092 | small) risk of data corruption in this situation. | |
a9d69660 | 1093 | |
5787fa49 NB |
1094 | .TP |
1095 | .BI md= n , dev , dev ,... | |
a9d69660 NB |
1096 | .TP |
1097 | .BI md=d n , dev , dev ,... | |
5787fa49 NB |
1098 | This tells the md driver to assemble |
1099 | .B /dev/md n | |
1100 | from the listed devices. It is only necessary to start the device | |
1101 | holding the root filesystem this way. Other arrays are best started | |
1102 | once the system is booted. | |
1103 | ||
a9d69660 NB |
1104 | In 2.6 kernels, the |
1105 | .B d | |
1106 | immediately after the | |
1107 | .B = | |
1108 | indicates that a partitionable device (e.g. | |
1109 | .BR /dev/md/d0 ) | |
1110 | should be created rather than the original non-partitionable device. | |
1111 | ||
5787fa49 NB |
1112 | .TP |
1113 | .BI md= n , l , c , i , dev... | |
1114 | This tells the md driver to assemble a legacy RAID0 or LINEAR array | |
1115 | without a superblock. | |
1116 | .I n | |
1117 | gives the md device number, | |
1118 | .I l | |
dae45415 | 1119 | gives the level, 0 for RAID0 or \-1 for LINEAR, |
5787fa49 NB |
1120 | .I c |
1121 | gives the chunk size as a base-2 logarithm offset by twelve, so 0 | |
1122 | means 4K, 1 means 8K. | |
1123 | .I i | |
1124 | is ignored (legacy support). | |
e0d19036 | 1125 | |
56eb10c0 NB |
1126 | .SH FILES |
1127 | .TP | |
1128 | .B /proc/mdstat | |
1129 | Contains information about the status of currently running array. | |
1130 | .TP | |
1131 | .B /proc/sys/dev/raid/speed_limit_min | |
93e790af | 1132 | A readable and writable file that reflects the current "goal" rebuild |
56eb10c0 NB |
1133 | speed for times when non-rebuild activity is current on an array. |
1134 | The speed is in Kibibytes per second, and is a per-device rate, not a | |
93e790af | 1135 | per-array rate (which means that an array with more disks will shuffle |
e0fe762a | 1136 | more data for a given speed). The default is 1000. |
56eb10c0 NB |
1137 | |
1138 | .TP | |
1139 | .B /proc/sys/dev/raid/speed_limit_max | |
93e790af | 1140 | A readable and writable file that reflects the current "goal" rebuild |
56eb10c0 | 1141 | speed for times when no non-rebuild activity is current on an array. |
e0fe762a | 1142 | The default is 200,000. |
56eb10c0 NB |
1143 | |
1144 | .SH SEE ALSO | |
1145 | .BR mdadm (8), |