]>
Commit | Line | Data |
---|---|---|
519561f7 NB |
1 | .\" Copyright Neil Brown and others. |
2 | .\" This program is free software; you can redistribute it and/or modify | |
3 | .\" it under the terms of the GNU General Public License as published by | |
4 | .\" the Free Software Foundation; either version 2 of the License, or | |
5 | .\" (at your option) any later version. | |
6 | .\" See file COPYING in distribution for details. | |
17645275 | 7 | .if n .pl 1000v |
56eb10c0 NB |
8 | .TH MD 4 |
9 | .SH NAME | |
93e790af | 10 | md \- Multiple Device driver aka Linux Software RAID |
56eb10c0 NB |
11 | .SH SYNOPSIS |
12 | .BI /dev/md n | |
13 | .br | |
14 | .BI /dev/md/ n | |
e0fe762a N |
15 | .br |
16 | .BR /dev/md/ name | |
56eb10c0 NB |
17 | .SH DESCRIPTION |
18 | The | |
19 | .B md | |
20 | driver provides virtual devices that are created from one or more | |
e0d19036 | 21 | independent underlying devices. This array of devices often contains |
02b76eea NB |
22 | redundancy and the devices are often disk drives, hence the acronym RAID |
23 | which stands for a Redundant Array of Independent Disks. | |
56eb10c0 NB |
24 | .PP |
25 | .B md | |
599e5a36 NB |
26 | supports RAID levels |
27 | 1 (mirroring), | |
28 | 4 (striped array with parity device), | |
29 | 5 (striped array with distributed parity information), | |
30 | 6 (striped array with distributed dual redundancy information), and | |
31 | 10 (striped and mirrored). | |
32 | If some number of underlying devices fails while using one of these | |
98c6faba NB |
33 | levels, the array will continue to function; this number is one for |
34 | RAID levels 4 and 5, two for RAID level 6, and all but one (N-1) for | |
93e790af | 35 | RAID level 1, and dependent on configuration for level 10. |
56eb10c0 NB |
36 | .PP |
37 | .B md | |
e0d19036 | 38 | also supports a number of pseudo RAID (non-redundant) configurations |
570c0542 NB |
39 | including RAID0 (striped array), LINEAR (catenated array), |
40 | MULTIPATH (a set of different interfaces to the same device), | |
41 | and FAULTY (a layer over a single device into which errors can be injected). | |
56eb10c0 | 42 | |
e0fe762a | 43 | .SS MD METADATA |
bcbb92d4 | 44 | Each device in an array may have some |
e0fe762a N |
45 | .I metadata |
46 | stored in the device. This metadata is sometimes called a | |
47 | .BR superblock . | |
48 | The metadata records information about the structure and state of the array. | |
570c0542 | 49 | This allows the array to be reliably re-assembled after a shutdown. |
56eb10c0 | 50 | |
570c0542 NB |
51 | From Linux kernel version 2.6.10, |
52 | .B md | |
e0fe762a | 53 | provides support for two different formats of metadata, and |
570c0542 NB |
54 | other formats can be added. Prior to this release, only one format is |
55 | supported. | |
56 | ||
b3f1c093 | 57 | The common format \(em known as version 0.90 \(em has |
570c0542 | 58 | a superblock that is 4K long and is written into a 64K aligned block that |
11a3e71d | 59 | starts at least 64K and less than 128K from the end of the device |
56eb10c0 NB |
60 | (i.e. to get the address of the superblock round the size of the |
61 | device down to a multiple of 64K and then subtract 64K). | |
11a3e71d | 62 | The available size of each device is the amount of space before the |
56eb10c0 NB |
63 | super block, so between 64K and 128K is lost when a device in |
64 | incorporated into an MD array. | |
93e790af | 65 | This superblock stores multi-byte fields in a processor-dependent |
570c0542 NB |
66 | manner, so arrays cannot easily be moved between computers with |
67 | different processors. | |
68 | ||
b3f1c093 | 69 | The new format \(em known as version 1 \(em has a superblock that is |
570c0542 NB |
70 | normally 1K long, but can be longer. It is normally stored between 8K |
71 | and 12K from the end of the device, on a 4K boundary, though | |
72 | variations can be stored at the start of the device (version 1.1) or 4K from | |
73 | the start of the device (version 1.2). | |
e0fe762a | 74 | This metadata format stores multibyte data in a |
93e790af | 75 | processor-independent format and supports up to hundreds of |
570c0542 | 76 | component devices (version 0.90 only supports 28). |
56eb10c0 | 77 | |
e0fe762a | 78 | The metadata contains, among other things: |
56eb10c0 NB |
79 | .TP |
80 | LEVEL | |
11a3e71d | 81 | The manner in which the devices are arranged into the array |
956a13fb | 82 | (LINEAR, RAID0, RAID1, RAID4, RAID5, RAID10, MULTIPATH). |
56eb10c0 NB |
83 | .TP |
84 | UUID | |
85 | a 128 bit Universally Unique Identifier that identifies the array that | |
93e790af | 86 | contains this device. |
56eb10c0 | 87 | |
e0fe762a | 88 | .PP |
2a940e36 NB |
89 | When a version 0.90 array is being reshaped (e.g. adding extra devices |
90 | to a RAID5), the version number is temporarily set to 0.91. This | |
91 | ensures that if the reshape process is stopped in the middle (e.g. by | |
92 | a system crash) and the machine boots into an older kernel that does | |
93 | not support reshaping, then the array will not be assembled (which | |
94 | would cause data corruption) but will be left untouched until a kernel | |
95 | that can complete the reshape processes is used. | |
96 | ||
e0fe762a | 97 | .SS ARRAYS WITHOUT METADATA |
570c0542 | 98 | While it is usually best to create arrays with superblocks so that |
93e790af SW |
99 | they can be assembled reliably, there are some circumstances when an |
100 | array without superblocks is preferred. These include: | |
570c0542 NB |
101 | .TP |
102 | LEGACY ARRAYS | |
11a3e71d NB |
103 | Early versions of the |
104 | .B md | |
956a13fb | 105 | driver only supported LINEAR and RAID0 configurations and did not use |
570c0542 NB |
106 | a superblock (which is less critical with these configurations). |
107 | While such arrays should be rebuilt with superblocks if possible, | |
11a3e71d | 108 | .B md |
570c0542 NB |
109 | continues to support them. |
110 | .TP | |
111 | FAULTY | |
112 | Being a largely transparent layer over a different device, the FAULTY | |
113 | personality doesn't gain anything from having a superblock. | |
114 | .TP | |
115 | MULTIPATH | |
116 | It is often possible to detect devices which are different paths to | |
117 | the same storage directly rather than having a distinctive superblock | |
118 | written to the device and searched for on all paths. In this case, | |
119 | a MULTIPATH array with no superblock makes sense. | |
120 | .TP | |
121 | RAID1 | |
956a13fb | 122 | In some configurations it might be desired to create a RAID1 |
93e790af | 123 | configuration that does not use a superblock, and to maintain the state of |
095407fa | 124 | the array elsewhere. While not encouraged for general use, it does |
addc80c4 | 125 | have special-purpose uses and is supported. |
11a3e71d | 126 | |
e0fe762a N |
127 | .SS ARRAYS WITH EXTERNAL METADATA |
128 | ||
129 | From release 2.6.28, the | |
130 | .I md | |
131 | driver supports arrays with externally managed metadata. That is, | |
1e49aaa0 | 132 | the metadata is not managed by the kernel but rather by a user-space |
e0fe762a N |
133 | program which is external to the kernel. This allows support for a |
134 | variety of metadata formats without cluttering the kernel with lots of | |
135 | details. | |
136 | .PP | |
137 | .I md | |
138 | is able to communicate with the user-space program through various | |
139 | sysfs attributes so that it can make appropriate changes to the | |
1b17b4e4 | 140 | metadata \- for example to mark a device as faulty. When necessary, |
e0fe762a N |
141 | .I md |
142 | will wait for the program to acknowledge the event by writing to a | |
143 | sysfs attribute. | |
144 | The manual page for | |
145 | .IR mdmon (8) | |
146 | contains more detail about this interaction. | |
147 | ||
148 | .SS CONTAINERS | |
149 | Many metadata formats use a single block of metadata to describe a | |
150 | number of different arrays which all use the same set of devices. | |
151 | In this case it is helpful for the kernel to know about the full set | |
152 | of devices as a whole. This set is known to md as a | |
153 | .IR container . | |
154 | A container is an | |
155 | .I md | |
156 | array with externally managed metadata and with device offset and size | |
157 | so that it just covers the metadata part of the devices. The | |
158 | remainder of each device is available to be incorporated into various | |
159 | arrays. | |
160 | ||
56eb10c0 | 161 | .SS LINEAR |
11a3e71d | 162 | |
956a13fb | 163 | A LINEAR array simply catenates the available space on each |
93e790af | 164 | drive to form one large virtual drive. |
11a3e71d NB |
165 | |
166 | One advantage of this arrangement over the more common RAID0 | |
167 | arrangement is that the array may be reconfigured at a later time with | |
93e790af SW |
168 | an extra drive, so the array is made bigger without disturbing the |
169 | data that is on the array. This can even be done on a live | |
11a3e71d NB |
170 | array. |
171 | ||
599e5a36 NB |
172 | If a chunksize is given with a LINEAR array, the usable space on each |
173 | device is rounded down to a multiple of this chunksize. | |
11a3e71d | 174 | |
56eb10c0 | 175 | .SS RAID0 |
11a3e71d NB |
176 | |
177 | A RAID0 array (which has zero redundancy) is also known as a | |
178 | striped array. | |
e0d19036 | 179 | A RAID0 array is configured at creation with a |
bcbb92d4 | 180 | .B "Chunk Size" |
e0fe762a N |
181 | which must be a power of two (prior to Linux 2.6.31), and at least 4 |
182 | kibibytes. | |
e0d19036 | 183 | |
2d465520 | 184 | The RAID0 driver assigns the first chunk of the array to the first |
e0d19036 | 185 | device, the second chunk to the second device, and so on until all |
e0fe762a | 186 | drives have been assigned one chunk. This collection of chunks forms a |
e0d19036 | 187 | .BR stripe . |
93e790af | 188 | Further chunks are gathered into stripes in the same way, and are |
e0d19036 NB |
189 | assigned to the remaining space in the drives. |
190 | ||
2d465520 NB |
191 | If devices in the array are not all the same size, then once the |
192 | smallest device has been exhausted, the RAID0 driver starts | |
e0d19036 NB |
193 | collecting chunks into smaller stripes that only span the drives which |
194 | still have remaining space. | |
195 | ||
196 | ||
56eb10c0 | 197 | .SS RAID1 |
e0d19036 NB |
198 | |
199 | A RAID1 array is also known as a mirrored set (though mirrors tend to | |
5787fa49 | 200 | provide reflected images, which RAID1 does not) or a plex. |
e0d19036 NB |
201 | |
202 | Once initialised, each device in a RAID1 array contains exactly the | |
203 | same data. Changes are written to all devices in parallel. Data is | |
204 | read from any one device. The driver attempts to distribute read | |
205 | requests across all devices to maximise performance. | |
206 | ||
207 | All devices in a RAID1 array should be the same size. If they are | |
208 | not, then only the amount of space available on the smallest device is | |
93e790af | 209 | used (any extra space on other devices is wasted). |
e0d19036 | 210 | |
3dacb890 IP |
211 | Note that the read balancing done by the driver does not make the RAID1 |
212 | performance profile be the same as for RAID0; a single stream of | |
213 | sequential input will not be accelerated (e.g. a single dd), but | |
214 | multiple sequential streams or a random workload will use more than one | |
215 | spindle. In theory, having an N-disk RAID1 will allow N sequential | |
216 | threads to read from all disks. | |
217 | ||
e0fe762a | 218 | Individual devices in a RAID1 can be marked as "write-mostly". |
1b17b4e4 | 219 | These drives are excluded from the normal read balancing and will only |
e0fe762a N |
220 | be read from when there is no other option. This can be useful for |
221 | devices connected over a slow link. | |
222 | ||
56eb10c0 | 223 | .SS RAID4 |
e0d19036 NB |
224 | |
225 | A RAID4 array is like a RAID0 array with an extra device for storing | |
aa88f531 NB |
226 | parity. This device is the last of the active devices in the |
227 | array. Unlike RAID0, RAID4 also requires that all stripes span all | |
e0d19036 NB |
228 | drives, so extra space on devices that are larger than the smallest is |
229 | wasted. | |
230 | ||
93e790af | 231 | When any block in a RAID4 array is modified, the parity block for that |
e0d19036 NB |
232 | stripe (i.e. the block in the parity device at the same device offset |
233 | as the stripe) is also modified so that the parity block always | |
93e790af | 234 | contains the "parity" for the whole stripe. I.e. its content is |
e0d19036 NB |
235 | equivalent to the result of performing an exclusive-or operation |
236 | between all the data blocks in the stripe. | |
237 | ||
238 | This allows the array to continue to function if one device fails. | |
239 | The data that was on that device can be calculated as needed from the | |
240 | parity block and the other data blocks. | |
241 | ||
56eb10c0 | 242 | .SS RAID5 |
e0d19036 NB |
243 | |
244 | RAID5 is very similar to RAID4. The difference is that the parity | |
245 | blocks for each stripe, instead of being on a single device, are | |
246 | distributed across all devices. This allows more parallelism when | |
93e790af | 247 | writing, as two different block updates will quite possibly affect |
e0d19036 NB |
248 | parity blocks on different devices so there is less contention. |
249 | ||
93e790af | 250 | This also allows more parallelism when reading, as read requests are |
e0d19036 NB |
251 | distributed over all the devices in the array instead of all but one. |
252 | ||
98c6faba NB |
253 | .SS RAID6 |
254 | ||
255 | RAID6 is similar to RAID5, but can handle the loss of any \fItwo\fP | |
256 | devices without data loss. Accordingly, it requires N+2 drives to | |
257 | store N drives worth of data. | |
258 | ||
259 | The performance for RAID6 is slightly lower but comparable to RAID5 in | |
260 | normal mode and single disk failure mode. It is very slow in dual | |
261 | disk failure mode, however. | |
262 | ||
599e5a36 NB |
263 | .SS RAID10 |
264 | ||
93e790af | 265 | RAID10 provides a combination of RAID1 and RAID0, and is sometimes known |
599e5a36 NB |
266 | as RAID1+0. Every datablock is duplicated some number of times, and |
267 | the resulting collection of datablocks are distributed over multiple | |
268 | drives. | |
269 | ||
93e790af | 270 | When configuring a RAID10 array, it is necessary to specify the number |
8dc92b41 CAM |
271 | of replicas of each data block that are required (this will usually |
272 | be\ 2) and whether their layout should be "near", "far" or "offset" | |
273 | (with "offset" being available since Linux\ 2.6.18). | |
274 | ||
275 | .B About the RAID10 Layout Examples: | |
276 | .br | |
277 | The examples below visualise the chunk distribution on the underlying | |
278 | devices for the respective layout. | |
279 | ||
280 | For simplicity it is assumed that the size of the chunks equals the | |
281 | size of the blocks of the underlying devices as well as those of the | |
282 | RAID10 device exported by the kernel (for example \fB/dev/md/\fPname). | |
283 | .br | |
284 | Therefore the chunks\ /\ chunk numbers map directly to the blocks\ /\ | |
285 | block addresses of the exported RAID10 device. | |
286 | ||
287 | Decimal numbers (0,\ 1, 2,\ ...) are the chunks of the RAID10 and due | |
288 | to the above assumption also the blocks and block addresses of the | |
289 | exported RAID10 device. | |
290 | .br | |
291 | Repeated numbers mean copies of a chunk\ /\ block (obviously on | |
292 | different underlying devices). | |
293 | .br | |
294 | Hexadecimal numbers (0x00,\ 0x01, 0x02,\ ...) are the block addresses | |
295 | of the underlying devices. | |
296 | ||
297 | .TP | |
298 | \fB "near" Layout\fP | |
299 | When "near" replicas are chosen, the multiple copies of a given chunk are laid | |
300 | out consecutively ("as close to each other as possible") across the stripes of | |
301 | the array. | |
302 | ||
303 | With an even number of devices, they will likely (unless some misalignment is | |
304 | present) lay at the very same offset on the different devices. | |
305 | .br | |
306 | This is as the "classic" RAID1+0; that is two groups of mirrored devices (in the | |
307 | example below the groups Device\ #1\ /\ #2 and Device\ #3\ /\ #4 are each a | |
308 | RAID1) both in turn forming a striped RAID0. | |
309 | ||
310 | .ne 10 | |
311 | .B Example with 2\ copies per chunk and an even number\ (4) of devices: | |
312 | .TS | |
313 | tab(;); | |
314 | C - - - - | |
315 | C | C | C | C | C | | |
316 | | - | - | - | - | - | | |
317 | | C | C | C | C | C | | |
318 | | C | C | C | C | C | | |
319 | | C | C | C | C | C | | |
320 | | C | C | C | C | C | | |
321 | | C | C | C | C | C | | |
322 | | C | C | C | C | C | | |
323 | | - | - | - | - | - | | |
324 | C C S C S | |
325 | C C S C S | |
326 | C C S S S | |
327 | C C S S S. | |
328 | ; | |
329 | ;Device #1;Device #2;Device #3;Device #4 | |
330 | 0x00;0;0;1;1 | |
331 | 0x01;2;2;3;3 | |
332 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\. | |
333 | :;:;:;:;: | |
334 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\. | |
335 | 0x80;254;254;255;255 | |
336 | ;\\---------v---------/;\\---------v---------/ | |
337 | ;RAID1;RAID1 | |
338 | ;\\---------------------v---------------------/ | |
339 | ;RAID0 | |
340 | .TE | |
341 | ||
342 | .ne 10 | |
343 | .B Example with 2\ copies per chunk and an odd number\ (5) of devices: | |
344 | .TS | |
345 | tab(;); | |
346 | C - - - - - | |
347 | C | C | C | C | C | C | | |
348 | | - | - | - | - | - | - | | |
349 | | C | C | C | C | C | C | | |
350 | | C | C | C | C | C | C | | |
351 | | C | C | C | C | C | C | | |
352 | | C | C | C | C | C | C | | |
353 | | C | C | C | C | C | C | | |
354 | | C | C | C | C | C | C | | |
355 | | - | - | - | - | - | - | | |
356 | C. | |
357 | ; | |
f99a9e15 | 358 | ;Dev #1;Dev #2;Dev #3;Dev #4;Dev #5 |
8dc92b41 CAM |
359 | 0x00;0;0;1;1;2 |
360 | 0x01;2;3;3;4;4 | |
361 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\. | |
362 | :;:;:;:;:;: | |
363 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\. | |
364 | 0x80;317;318;318;319;319 | |
365 | ; | |
366 | .TE | |
367 | ||
368 | .TP | |
369 | \fB "far" Layout\fP | |
370 | When "far" replicas are chosen, the multiple copies of a given chunk | |
371 | are laid out quite distant ("as far as reasonably possible") from each | |
372 | other. | |
373 | ||
374 | First a complete sequence of all data blocks (that is all the data one | |
375 | sees on the exported RAID10 block device) is striped over the | |
376 | devices. Then another (though "shifted") complete sequence of all data | |
377 | blocks; and so on (in the case of more than 2\ copies per chunk). | |
378 | ||
379 | The "shift" needed to prevent placing copies of the same chunks on the | |
380 | same devices is actually a cyclic permutation with offset\ 1 of each | |
381 | of the stripes within a complete sequence of chunks. | |
382 | .br | |
383 | The offset\ 1 is relative to the previous complete sequence of chunks, | |
384 | so in case of more than 2\ copies per chunk one gets the following | |
385 | offsets: | |
386 | .br | |
387 | 1.\ complete sequence of chunks: offset\ =\ \ 0 | |
388 | .br | |
389 | 2.\ complete sequence of chunks: offset\ =\ \ 1 | |
390 | .br | |
391 | 3.\ complete sequence of chunks: offset\ =\ \ 2 | |
392 | .br | |
393 | : | |
394 | .br | |
395 | n.\ complete sequence of chunks: offset\ =\ n-1 | |
396 | ||
397 | .ne 10 | |
398 | .B Example with 2\ copies per chunk and an even number\ (4) of devices: | |
399 | .TS | |
400 | tab(;); | |
401 | C - - - - | |
402 | C | C | C | C | C | | |
403 | | - | - | - | - | - | | |
404 | | C | C | C | C | C | L | |
405 | | C | C | C | C | C | L | |
406 | | C | C | C | C | C | L | |
407 | | C | C | C | C | C | L | |
408 | | C | C | C | C | C | L | |
409 | | C | C | C | C | C | L | |
410 | | C | C | C | C | C | L | |
411 | | C | C | C | C | C | L | |
412 | | C | C | C | C | C | L | |
413 | | C | C | C | C | C | L | |
414 | | C | C | C | C | C | L | |
415 | | C | C | C | C | C | L | |
416 | | - | - | - | - | - | | |
417 | C. | |
418 | ; | |
419 | ;Device #1;Device #2;Device #3;Device #4 | |
420 | ; | |
421 | 0x00;0;1;2;3;\\ | |
422 | 0x01;4;5;6;7;> [#] | |
423 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
424 | :;:;:;:;:;: | |
425 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
426 | 0x40;252;253;254;255;/ | |
427 | 0x41;3;0;1;2;\\ | |
428 | 0x42;7;4;5;6;> [#]~ | |
429 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
430 | :;:;:;:;:;: | |
431 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
432 | 0x80;255;252;253;254;/ | |
433 | ; | |
434 | .TE | |
435 | ||
436 | .ne 10 | |
437 | .B Example with 2\ copies per chunk and an odd number\ (5) of devices: | |
438 | .TS | |
439 | tab(;); | |
440 | C - - - - - | |
441 | C | C | C | C | C | C | | |
442 | | - | - | - | - | - | - | | |
443 | | C | C | C | C | C | C | L | |
444 | | C | C | C | C | C | C | L | |
445 | | C | C | C | C | C | C | L | |
446 | | C | C | C | C | C | C | L | |
447 | | C | C | C | C | C | C | L | |
448 | | C | C | C | C | C | C | L | |
449 | | C | C | C | C | C | C | L | |
450 | | C | C | C | C | C | C | L | |
451 | | C | C | C | C | C | C | L | |
452 | | C | C | C | C | C | C | L | |
453 | | C | C | C | C | C | C | L | |
454 | | C | C | C | C | C | C | L | |
455 | | - | - | - | - | - | - | | |
456 | C. | |
457 | ; | |
f99a9e15 | 458 | ;Dev #1;Dev #2;Dev #3;Dev #4;Dev #5 |
8dc92b41 CAM |
459 | ; |
460 | 0x00;0;1;2;3;4;\\ | |
461 | 0x01;5;6;7;8;9;> [#] | |
462 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
463 | :;:;:;:;:;:;: | |
464 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
465 | 0x40;315;316;317;318;319;/ | |
466 | 0x41;4;0;1;2;3;\\ | |
467 | 0x42;9;5;6;7;8;> [#]~ | |
468 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
469 | :;:;:;:;:;:;: | |
470 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;: | |
471 | 0x80;319;315;316;317;318;/ | |
472 | ; | |
473 | .TE | |
474 | ||
475 | With [#]\ being the complete sequence of chunks and [#]~\ the cyclic permutation | |
476 | with offset\ 1 thereof (in the case of more than 2 copies per chunk there would | |
477 | be ([#]~)~,\ (([#]~)~)~,\ ...). | |
478 | ||
479 | The advantage of this layout is that MD can easily spread sequential reads over | |
480 | the devices, making them similar to RAID0 in terms of speed. | |
481 | .br | |
482 | The cost is more seeking for writes, making them substantially slower. | |
483 | ||
484 | .TP | |
485 | \fB"offset" Layout\fP | |
486 | When "offset" replicas are chosen, all the copies of a given chunk are | |
487 | striped consecutively ("offset by the stripe length after each other") | |
488 | over the devices. | |
489 | ||
490 | Explained in detail, <number of devices> consecutive chunks are | |
491 | striped over the devices, immediately followed by a "shifted" copy of | |
492 | these chunks (and by further such "shifted" copies in the case of more | |
493 | than 2\ copies per chunk). | |
494 | .br | |
495 | This pattern repeats for all further consecutive chunks of the | |
496 | exported RAID10 device (in other words: all further data blocks). | |
497 | ||
498 | The "shift" needed to prevent placing copies of the same chunks on the | |
499 | same devices is actually a cyclic permutation with offset\ 1 of each | |
500 | of the striped copies of <number of devices> consecutive chunks. | |
501 | .br | |
502 | The offset\ 1 is relative to the previous striped copy of <number of | |
503 | devices> consecutive chunks, so in case of more than 2\ copies per | |
504 | chunk one gets the following offsets: | |
505 | .br | |
506 | 1.\ <number of devices> consecutive chunks: offset\ =\ \ 0 | |
507 | .br | |
508 | 2.\ <number of devices> consecutive chunks: offset\ =\ \ 1 | |
509 | .br | |
510 | 3.\ <number of devices> consecutive chunks: offset\ =\ \ 2 | |
511 | .br | |
512 | : | |
513 | .br | |
514 | n.\ <number of devices> consecutive chunks: offset\ =\ n-1 | |
515 | ||
516 | .ne 10 | |
517 | .B Example with 2\ copies per chunk and an even number\ (4) of devices: | |
518 | .TS | |
519 | tab(;); | |
520 | C - - - - | |
521 | C | C | C | C | C | | |
522 | | - | - | - | - | - | | |
523 | | C | C | C | C | C | L | |
524 | | C | C | C | C | C | L | |
525 | | C | C | C | C | C | L | |
526 | | C | C | C | C | C | L | |
527 | | C | C | C | C | C | L | |
528 | | C | C | C | C | C | L | |
529 | | C | C | C | C | C | L | |
530 | | C | C | C | C | C | L | |
531 | | C | C | C | C | C | L | |
532 | | - | - | - | - | - | | |
533 | C. | |
534 | ; | |
535 | ;Device #1;Device #2;Device #3;Device #4 | |
536 | ; | |
537 | 0x00;0;1;2;3;) AA | |
538 | 0x01;3;0;1;2;) AA~ | |
539 | 0x02;4;5;6;7;) AB | |
540 | 0x03;7;4;5;6;) AB~ | |
541 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;) \.\.\. | |
542 | :;:;:;:;:; : | |
543 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;) \.\.\. | |
544 | 0x79;251;252;253;254;) EX | |
545 | 0x80;254;251;252;253;) EX~ | |
546 | ; | |
547 | .TE | |
548 | ||
549 | .ne 10 | |
550 | .B Example with 2\ copies per chunk and an odd number\ (5) of devices: | |
551 | .TS | |
552 | tab(;); | |
553 | C - - - - - | |
554 | C | C | C | C | C | C | | |
555 | | - | - | - | - | - | - | | |
556 | | C | C | C | C | C | C | L | |
557 | | C | C | C | C | C | C | L | |
558 | | C | C | C | C | C | C | L | |
559 | | C | C | C | C | C | C | L | |
560 | | C | C | C | C | C | C | L | |
561 | | C | C | C | C | C | C | L | |
562 | | C | C | C | C | C | C | L | |
563 | | C | C | C | C | C | C | L | |
564 | | C | C | C | C | C | C | L | |
565 | | - | - | - | - | - | - | | |
566 | C. | |
567 | ; | |
f99a9e15 | 568 | ;Dev #1;Dev #2;Dev #3;Dev #4;Dev #5 |
8dc92b41 CAM |
569 | ; |
570 | 0x00;0;1;2;3;4;) AA | |
571 | 0x01;4;0;1;2;3;) AA~ | |
572 | 0x02;5;6;7;8;9;) AB | |
573 | 0x03;9;5;6;7;8;) AB~ | |
574 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;) \.\.\. | |
575 | :;:;:;:;:;:; : | |
576 | \.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;\.\.\.;) \.\.\. | |
577 | 0x79;314;315;316;317;318;) EX | |
578 | 0x80;318;314;315;316;317;) EX~ | |
579 | ; | |
580 | .TE | |
581 | ||
582 | With AA,\ AB,\ ..., AZ,\ BA,\ ... being the sets of <number of devices> consecutive | |
583 | chunks and AA~,\ AB~,\ ..., AZ~,\ BA~,\ ... the cyclic permutations with offset\ 1 | |
584 | thereof (in the case of more than 2 copies per chunk there would be (AA~)~,\ ... | |
585 | as well as ((AA~)~)~,\ ... and so on). | |
586 | ||
587 | This should give similar read characteristics to "far" if a suitably large chunk | |
588 | size is used, but without as much seeking for writes. | |
589 | .PP | |
590 | ||
b578481c | 591 | |
599e5a36 | 592 | It should be noted that the number of devices in a RAID10 array need |
93e790af | 593 | not be a multiple of the number of replica of each data block; however, |
599e5a36 NB |
594 | there must be at least as many devices as replicas. |
595 | ||
596 | If, for example, an array is created with 5 devices and 2 replicas, | |
597 | then space equivalent to 2.5 of the devices will be available, and | |
598 | every block will be stored on two different devices. | |
599 | ||
8dc92b41 | 600 | Finally, it is possible to have an array with both "near" and "far" |
93e790af | 601 | copies. If an array is configured with 2 near copies and 2 far |
599e5a36 NB |
602 | copies, then there will be a total of 4 copies of each block, each on |
603 | a different drive. This is an artifact of the implementation and is | |
604 | unlikely to be of real value. | |
605 | ||
bf40ab85 | 606 | .SS MULTIPATH |
e0d19036 NB |
607 | |
608 | MULTIPATH is not really a RAID at all as there is only one real device | |
609 | in a MULTIPATH md array. However there are multiple access points | |
610 | (paths) to this device, and one of these paths might fail, so there | |
611 | are some similarities. | |
612 | ||
a9d69660 | 613 | A MULTIPATH array is composed of a number of logically different |
2d465520 NB |
614 | devices, often fibre channel interfaces, that all refer the the same |
615 | real device. If one of these interfaces fails (e.g. due to cable | |
956a13fb | 616 | problems), the MULTIPATH driver will attempt to redirect requests to |
e0fe762a N |
617 | another interface. |
618 | ||
619 | The MULTIPATH drive is not receiving any ongoing development and | |
620 | should be considered a legacy driver. The device-mapper based | |
621 | multipath drivers should be preferred for new installations. | |
e0d19036 | 622 | |
b5e64645 | 623 | .SS FAULTY |
956a13fb | 624 | The FAULTY md module is provided for testing purposes. A FAULTY array |
b5e64645 NB |
625 | has exactly one component device and is normally assembled without a |
626 | superblock, so the md array created provides direct access to all of | |
627 | the data in the component device. | |
628 | ||
629 | The FAULTY module may be requested to simulate faults to allow testing | |
a9d69660 | 630 | of other md levels or of filesystems. Faults can be chosen to trigger |
b5e64645 | 631 | on read requests or write requests, and can be transient (a subsequent |
addc80c4 | 632 | read/write at the address will probably succeed) or persistent |
b5e64645 NB |
633 | (subsequent read/write of the same address will fail). Further, read |
634 | faults can be "fixable" meaning that they persist until a write | |
635 | request at the same address. | |
636 | ||
93e790af | 637 | Fault types can be requested with a period. In this case, the fault |
a9d69660 NB |
638 | will recur repeatedly after the given number of requests of the |
639 | relevant type. For example if persistent read faults have a period of | |
640 | 100, then every 100th read request would generate a fault, and the | |
b5e64645 NB |
641 | faulty sector would be recorded so that subsequent reads on that |
642 | sector would also fail. | |
643 | ||
644 | There is a limit to the number of faulty sectors that are remembered. | |
645 | Faults generated after this limit is exhausted are treated as | |
646 | transient. | |
647 | ||
a9d69660 | 648 | The list of faulty sectors can be flushed, and the active list of |
b5e64645 | 649 | failure modes can be cleared. |
e0d19036 NB |
650 | |
651 | .SS UNCLEAN SHUTDOWN | |
652 | ||
599e5a36 NB |
653 | When changes are made to a RAID1, RAID4, RAID5, RAID6, or RAID10 array |
654 | there is a possibility of inconsistency for short periods of time as | |
93e790af SW |
655 | each update requires at least two block to be written to different |
656 | devices, and these writes probably won't happen at exactly the same | |
599e5a36 NB |
657 | time. Thus if a system with one of these arrays is shutdown in the |
658 | middle of a write operation (e.g. due to power failure), the array may | |
659 | not be consistent. | |
e0d19036 | 660 | |
2d465520 | 661 | To handle this situation, the md driver marks an array as "dirty" |
e0d19036 | 662 | before writing any data to it, and marks it as "clean" when the array |
98c6faba NB |
663 | is being disabled, e.g. at shutdown. If the md driver finds an array |
664 | to be dirty at startup, it proceeds to correct any possibly | |
665 | inconsistency. For RAID1, this involves copying the contents of the | |
666 | first drive onto all other drives. For RAID4, RAID5 and RAID6 this | |
667 | involves recalculating the parity for each stripe and making sure that | |
599e5a36 NB |
668 | the parity block has the correct data. For RAID10 it involves copying |
669 | one of the replicas of each block onto all the others. This process, | |
670 | known as "resynchronising" or "resync" is performed in the background. | |
671 | The array can still be used, though possibly with reduced performance. | |
98c6faba NB |
672 | |
673 | If a RAID4, RAID5 or RAID6 array is degraded (missing at least one | |
93e790af | 674 | drive, two for RAID6) when it is restarted after an unclean shutdown, it cannot |
98c6faba NB |
675 | recalculate parity, and so it is possible that data might be |
676 | undetectably corrupted. The 2.4 md driver | |
e0d19036 | 677 | .B does not |
addc80c4 NB |
678 | alert the operator to this condition. The 2.6 md driver will fail to |
679 | start an array in this condition without manual intervention, though | |
35cc5be4 | 680 | this behaviour can be overridden by a kernel parameter. |
e0d19036 NB |
681 | |
682 | .SS RECOVERY | |
683 | ||
addc80c4 | 684 | If the md driver detects a write error on a device in a RAID1, RAID4, |
599e5a36 NB |
685 | RAID5, RAID6, or RAID10 array, it immediately disables that device |
686 | (marking it as faulty) and continues operation on the remaining | |
93e790af SW |
687 | devices. If there are spare drives, the driver will start recreating |
688 | on one of the spare drives the data which was on that failed drive, | |
599e5a36 NB |
689 | either by copying a working drive in a RAID1 configuration, or by |
690 | doing calculations with the parity block on RAID4, RAID5 or RAID6, or | |
93e790af | 691 | by finding and copying originals for RAID10. |
e0d19036 | 692 | |
addc80c4 NB |
693 | In kernels prior to about 2.6.15, a read error would cause the same |
694 | effect as a write error. In later kernels, a read-error will instead | |
695 | cause md to attempt a recovery by overwriting the bad block. i.e. it | |
696 | will find the correct data from elsewhere, write it over the block | |
697 | that failed, and then try to read it back again. If either the write | |
698 | or the re-read fail, md will treat the error the same way that a write | |
93e790af | 699 | error is treated, and will fail the whole device. |
addc80c4 | 700 | |
2d465520 | 701 | While this recovery process is happening, the md driver will monitor |
e0d19036 NB |
702 | accesses to the array and will slow down the rate of recovery if other |
703 | activity is happening, so that normal access to the array will not be | |
704 | unduly affected. When no other activity is happening, the recovery | |
705 | process proceeds at full speed. The actual speed targets for the two | |
706 | different situations can be controlled by the | |
707 | .B speed_limit_min | |
708 | and | |
709 | .B speed_limit_max | |
710 | control files mentioned below. | |
711 | ||
1cc44574 N |
712 | .SS SCRUBBING AND MISMATCHES |
713 | ||
714 | As storage devices can develop bad blocks at any time it is valuable | |
715 | to regularly read all blocks on all devices in an array so as to catch | |
716 | such bad blocks early. This process is called | |
717 | .IR scrubbing . | |
718 | ||
719 | md arrays can be scrubbed by writing either | |
720 | .I check | |
721 | or | |
722 | .I repair | |
723 | to the file | |
724 | .I md/sync_action | |
725 | in the | |
726 | .I sysfs | |
727 | directory for the device. | |
728 | ||
c93e9d68 | 729 | Requesting a scrub will cause |
1cc44574 N |
730 | .I md |
731 | to read every block on every device in the array, and check that the | |
c93e9d68 N |
732 | data is consistent. For RAID1 and RAID10, this means checking that the copies |
733 | are identical. For RAID4, RAID5, RAID6 this means checking that the | |
734 | parity block is (or blocks are) correct. | |
1cc44574 N |
735 | |
736 | If a read error is detected during this process, the normal read-error | |
737 | handling causes correct data to be found from other devices and to be | |
738 | written back to the faulty device. In many case this will | |
739 | effectively | |
740 | .I fix | |
741 | the bad block. | |
742 | ||
743 | If all blocks read successfully but are found to not be consistent, | |
744 | then this is regarded as a | |
745 | .IR mismatch . | |
746 | ||
747 | If | |
748 | .I check | |
749 | was used, then no action is taken to handle the mismatch, it is simply | |
750 | recorded. | |
751 | If | |
752 | .I repair | |
753 | was used, then a mismatch will be repaired in the same way that | |
754 | .I resync | |
c93e9d68 | 755 | repairs arrays. For RAID5/RAID6 new parity blocks are written. For RAID1/RAID10, |
1cc44574 N |
756 | all but one block are overwritten with the content of that one block. |
757 | ||
758 | A count of mismatches is recorded in the | |
759 | .I sysfs | |
760 | file | |
761 | .IR md/mismatch_cnt . | |
762 | This is set to zero when a | |
c93e9d68 | 763 | scrub starts and is incremented whenever a sector is |
1cc44574 N |
764 | found that is a mismatch. |
765 | .I md | |
766 | normally works in units much larger than a single sector and when it | |
1e49aaa0 | 767 | finds a mismatch, it does not determine exactly how many actual sectors were |
c93e9d68 N |
768 | affected but simply adds the number of sectors in the IO unit that was |
769 | used. So a value of 128 could simply mean that a single 64KB check | |
770 | found an error (128 x 512bytes = 64KB). | |
771 | ||
772 | If an array is created by | |
773 | .I mdadm | |
774 | with | |
1cc44574 N |
775 | .I \-\-assume\-clean |
776 | then a subsequent check could be expected to find some mismatches. | |
777 | ||
778 | On a truly clean RAID5 or RAID6 array, any mismatches should indicate | |
779 | a hardware problem at some level - software issues should never cause | |
780 | such a mismatch. | |
781 | ||
782 | However on RAID1 and RAID10 it is possible for software issues to | |
783 | cause a mismatch to be reported. This does not necessarily mean that | |
784 | the data on the array is corrupted. It could simply be that the | |
785 | system does not care what is stored on that part of the array - it is | |
786 | unused space. | |
787 | ||
788 | The most likely cause for an unexpected mismatch on RAID1 or RAID10 | |
789 | occurs if a swap partition or swap file is stored on the array. | |
790 | ||
791 | When the swap subsystem wants to write a page of memory out, it flags | |
792 | the page as 'clean' in the memory manager and requests the swap device | |
793 | to write it out. It is quite possible that the memory will be | |
794 | changed while the write-out is happening. In that case the 'clean' | |
795 | flag will be found to be clear when the write completes and so the | |
796 | swap subsystem will simply forget that the swapout had been attempted, | |
c93e9d68 | 797 | and will possibly choose a different page to write out. |
1cc44574 | 798 | |
c93e9d68 | 799 | If the swap device was on RAID1 (or RAID10), then the data is sent |
1cc44574 | 800 | from memory to a device twice (or more depending on the number of |
c93e9d68 N |
801 | devices in the array). Thus it is possible that the memory gets changed |
802 | between the times it is sent, so different data can be written to | |
803 | the different devices in the array. This will be detected by | |
1cc44574 N |
804 | .I check |
805 | as a mismatch. However it does not reflect any corruption as the | |
806 | block where this mismatch occurs is being treated by the swap system as | |
807 | being empty, and the data will never be read from that block. | |
808 | ||
809 | It is conceivable for a similar situation to occur on non-swap files, | |
810 | though it is less likely. | |
811 | ||
812 | Thus the | |
813 | .I mismatch_cnt | |
814 | value can not be interpreted very reliably on RAID1 or RAID10, | |
815 | especially when the device is used for swap. | |
816 | ||
817 | ||
599e5a36 NB |
818 | .SS BITMAP WRITE-INTENT LOGGING |
819 | ||
820 | From Linux 2.6.13, | |
821 | .I md | |
822 | supports a bitmap based write-intent log. If configured, the bitmap | |
823 | is used to record which blocks of the array may be out of sync. | |
824 | Before any write request is honoured, md will make sure that the | |
825 | corresponding bit in the log is set. After a period of time with no | |
826 | writes to an area of the array, the corresponding bit will be cleared. | |
827 | ||
828 | This bitmap is used for two optimisations. | |
829 | ||
1afe1167 | 830 | Firstly, after an unclean shutdown, the resync process will consult |
599e5a36 | 831 | the bitmap and only resync those blocks that correspond to bits in the |
1afe1167 | 832 | bitmap that are set. This can dramatically reduce resync time. |
599e5a36 NB |
833 | |
834 | Secondly, when a drive fails and is removed from the array, md stops | |
835 | clearing bits in the intent log. If that same drive is re-added to | |
836 | the array, md will notice and will only recover the sections of the | |
837 | drive that are covered by bits in the intent log that are set. This | |
838 | can allow a device to be temporarily removed and reinserted without | |
839 | causing an enormous recovery cost. | |
840 | ||
841 | The intent log can be stored in a file on a separate device, or it can | |
842 | be stored near the superblocks of an array which has superblocks. | |
843 | ||
93e790af | 844 | It is possible to add an intent log to an active array, or remove an |
addc80c4 | 845 | intent log if one is present. |
599e5a36 NB |
846 | |
847 | In 2.6.13, intent bitmaps are only supported with RAID1. Other levels | |
addc80c4 | 848 | with redundancy are supported from 2.6.15. |
599e5a36 | 849 | |
968d2a33 | 850 | .SS BAD BLOCK LIST |
bf95d0f3 N |
851 | |
852 | From Linux 3.5 each device in an | |
853 | .I md | |
854 | array can store a list of known-bad-blocks. This list is 4K in size | |
855 | and usually positioned at the end of the space between the superblock | |
856 | and the data. | |
857 | ||
858 | When a block cannot be read and cannot be repaired by writing data | |
859 | recovered from other devices, the address of the block is stored in | |
968d2a33 | 860 | the bad block list. Similarly if an attempt to write a block fails, |
bf95d0f3 N |
861 | the address will be recorded as a bad block. If attempting to record |
862 | the bad block fails, the whole device will be marked faulty. | |
863 | ||
864 | Attempting to read from a known bad block will cause a read error. | |
865 | Attempting to write to a known bad block will be ignored if any write | |
866 | errors have been reported by the device. If there have been no write | |
867 | errors then the data will be written to the known bad block and if | |
868 | that succeeds, the address will be removed from the list. | |
869 | ||
870 | This allows an array to fail more gracefully - a few blocks on | |
871 | different devices can be faulty without taking the whole array out of | |
872 | action. | |
873 | ||
968d2a33 | 874 | The list is particularly useful when recovering to a spare. If a few blocks |
bf95d0f3 | 875 | cannot be read from the other devices, the bulk of the recovery can |
968d2a33 | 876 | complete and those few bad blocks will be recorded in the bad block list. |
bf95d0f3 | 877 | |
28f83f6d SL |
878 | .SS RAID456 WRITE JOURNAL |
879 | ||
880 | Due to non-atomicity nature of RAID write operations, interruption of | |
881 | write operations (system crash, etc.) to RAID456 array can lead to | |
882 | inconsistent parity and data loss (so called RAID-5 write hole). | |
883 | ||
884 | To plug the write hole, from Linux 4.4 (to be confirmed), | |
885 | .I md | |
886 | supports write ahead journal for RAID456. When the array is created, | |
887 | an additional journal device can be added to the array through | |
888 | .IR write-journal | |
889 | option. The RAID write journal works similar to file system journals. | |
890 | Before writing to the data disks, md persists data AND parity of the | |
891 | stripe to the journal device. After crashes, md searches the journal | |
892 | device for incomplete write operations, and replay them to the data | |
893 | disks. | |
894 | ||
895 | When the journal device fails, the RAID array is forced to run in | |
896 | read-only mode. | |
897 | ||
599e5a36 NB |
898 | .SS WRITE-BEHIND |
899 | ||
900 | From Linux 2.6.14, | |
901 | .I md | |
addc80c4 | 902 | supports WRITE-BEHIND on RAID1 arrays. |
599e5a36 NB |
903 | |
904 | This allows certain devices in the array to be flagged as | |
905 | .IR write-mostly . | |
906 | MD will only read from such devices if there is no | |
907 | other option. | |
908 | ||
909 | If a write-intent bitmap is also provided, write requests to | |
910 | write-mostly devices will be treated as write-behind requests and md | |
911 | will not wait for writes to those requests to complete before | |
912 | reporting the write as complete to the filesystem. | |
913 | ||
914 | This allows for a RAID1 with WRITE-BEHIND to be used to mirror data | |
8f21823f | 915 | over a slow link to a remote computer (providing the link isn't too |
599e5a36 NB |
916 | slow). The extra latency of the remote link will not slow down normal |
917 | operations, but the remote system will still have a reasonably | |
918 | up-to-date copy of all data. | |
919 | ||
71574efb N |
920 | .SS FAILFAST |
921 | ||
922 | From Linux 4.10, | |
923 | .I | |
924 | md | |
925 | supports FAILFAST for RAID1 and RAID10 arrays. This is a flag that | |
926 | can be set on individual drives, though it is usually set on all | |
927 | drives, or no drives. | |
928 | ||
929 | When | |
930 | .I md | |
931 | sends an I/O request to a drive that is marked as FAILFAST, and when | |
932 | the array could survive the loss of that drive without losing data, | |
933 | .I md | |
934 | will request that the underlying device does not perform any retries. | |
935 | This means that a failure will be reported to | |
936 | .I md | |
937 | promptly, and it can mark the device as faulty and continue using the | |
938 | other device(s). | |
939 | .I md | |
940 | cannot control the timeout that the underlying devices use to | |
941 | determine failure. Any changes desired to that timeout must be set | |
942 | explictly on the underlying device, separately from using | |
943 | .IR mdadm . | |
944 | ||
945 | If a FAILFAST request does fail, and if it is still safe to mark the | |
946 | device as faulty without data loss, that will be done and the array | |
947 | will continue functioning on a reduced number of devices. If it is not | |
948 | possible to safely mark the device as faulty, | |
949 | .I md | |
950 | will retry the request without disabling retries in the underlying | |
951 | device. In any case, | |
952 | .I md | |
953 | will not attempt to repair read errors on a device marked as FAILFAST | |
954 | by writing out the correct. It will just mark the device as faulty. | |
955 | ||
956 | FAILFAST is appropriate for storage arrays that have a low probability | |
957 | of true failure, but will sometimes introduce unacceptable delays to | |
958 | I/O requests while performing internal maintenance. The value of | |
959 | setting FAILFAST involves a trade-off. The gain is that the chance of | |
960 | unacceptable delays is substantially reduced. The cost is that the | |
961 | unlikely event of data-loss on one device is slightly more likely to | |
962 | result in data-loss for the array. | |
963 | ||
964 | When a device in an array using FAILFAST is marked as faulty, it will | |
965 | usually become usable again in a short while. | |
966 | .I mdadm | |
967 | makes no attempt to detect that possibility. Some separate | |
968 | mechanism, tuned to the specific details of the expected failure modes, | |
969 | needs to be created to monitor devices to see when they return to full | |
970 | functionality, and to then re-add them to the array. In order of | |
971 | this "re-add" functionality to be effective, an array using FAILFAST | |
972 | should always have a write-intent bitmap. | |
973 | ||
addc80c4 NB |
974 | .SS RESTRIPING |
975 | ||
976 | .IR Restriping , | |
977 | also known as | |
978 | .IR Reshaping , | |
979 | is the processes of re-arranging the data stored in each stripe into a | |
980 | new layout. This might involve changing the number of devices in the | |
93e790af | 981 | array (so the stripes are wider), changing the chunk size (so stripes |
addc80c4 | 982 | are deeper or shallower), or changing the arrangement of data and |
956a13fb | 983 | parity (possibly changing the RAID level, e.g. 1 to 5 or 5 to 6). |
addc80c4 | 984 | |
c64881d7 N |
985 | As of Linux 2.6.35, md can reshape a RAID4, RAID5, or RAID6 array to |
986 | have a different number of devices (more or fewer) and to have a | |
987 | different layout or chunk size. It can also convert between these | |
988 | different RAID levels. It can also convert between RAID0 and RAID10, | |
989 | and between RAID0 and RAID4 or RAID5. | |
990 | Other possibilities may follow in future kernels. | |
addc80c4 NB |
991 | |
992 | During any stripe process there is a 'critical section' during which | |
35cc5be4 | 993 | live data is being overwritten on disk. For the operation of |
956a13fb | 994 | increasing the number of drives in a RAID5, this critical section |
addc80c4 NB |
995 | covers the first few stripes (the number being the product of the old |
996 | and new number of devices). After this critical section is passed, | |
997 | data is only written to areas of the array which no longer hold live | |
b3f1c093 | 998 | data \(em the live data has already been located away. |
addc80c4 | 999 | |
c64881d7 N |
1000 | For a reshape which reduces the number of devices, the 'critical |
1001 | section' is at the end of the reshape process. | |
1002 | ||
addc80c4 NB |
1003 | md is not able to ensure data preservation if there is a crash |
1004 | (e.g. power failure) during the critical section. If md is asked to | |
1005 | start an array which failed during a critical section of restriping, | |
1006 | it will fail to start the array. | |
1007 | ||
1008 | To deal with this possibility, a user-space program must | |
1009 | .IP \(bu 4 | |
1010 | Disable writes to that section of the array (using the | |
1011 | .B sysfs | |
1012 | interface), | |
1013 | .IP \(bu 4 | |
93e790af | 1014 | take a copy of the data somewhere (i.e. make a backup), |
addc80c4 | 1015 | .IP \(bu 4 |
93e790af | 1016 | allow the process to continue and invalidate the backup and restore |
addc80c4 NB |
1017 | write access once the critical section is passed, and |
1018 | .IP \(bu 4 | |
93e790af | 1019 | provide for restoring the critical data before restarting the array |
addc80c4 NB |
1020 | after a system crash. |
1021 | .PP | |
1022 | ||
1023 | .B mdadm | |
93e790af | 1024 | versions from 2.4 do this for growing a RAID5 array. |
addc80c4 NB |
1025 | |
1026 | For operations that do not change the size of the array, like simply | |
1027 | increasing chunk size, or converting RAID5 to RAID6 with one extra | |
93e790af SW |
1028 | device, the entire process is the critical section. In this case, the |
1029 | restripe will need to progress in stages, as a section is suspended, | |
c64881d7 | 1030 | backed up, restriped, and released. |
addc80c4 NB |
1031 | |
1032 | .SS SYSFS INTERFACE | |
93e790af | 1033 | Each block device appears as a directory in |
addc80c4 | 1034 | .I sysfs |
93e790af | 1035 | (which is usually mounted at |
addc80c4 NB |
1036 | .BR /sys ). |
1037 | For MD devices, this directory will contain a subdirectory called | |
1038 | .B md | |
1039 | which contains various files for providing access to information about | |
1040 | the array. | |
1041 | ||
1042 | This interface is documented more fully in the file | |
1043 | .B Documentation/md.txt | |
1044 | which is distributed with the kernel sources. That file should be | |
1045 | consulted for full documentation. The following are just a selection | |
1046 | of attribute files that are available. | |
1047 | ||
1048 | .TP | |
1049 | .B md/sync_speed_min | |
1050 | This value, if set, overrides the system-wide setting in | |
1051 | .B /proc/sys/dev/raid/speed_limit_min | |
1052 | for this array only. | |
1053 | Writing the value | |
93e790af SW |
1054 | .B "system" |
1055 | to this file will cause the system-wide setting to have effect. | |
addc80c4 NB |
1056 | |
1057 | .TP | |
1058 | .B md/sync_speed_max | |
1059 | This is the partner of | |
1060 | .B md/sync_speed_min | |
1061 | and overrides | |
1e49aaa0 | 1062 | .B /proc/sys/dev/raid/speed_limit_max |
addc80c4 NB |
1063 | described below. |
1064 | ||
1065 | .TP | |
1066 | .B md/sync_action | |
1067 | This can be used to monitor and control the resync/recovery process of | |
1068 | MD. | |
1069 | In particular, writing "check" here will cause the array to read all | |
1070 | data block and check that they are consistent (e.g. parity is correct, | |
1071 | or all mirror replicas are the same). Any discrepancies found are | |
1072 | .B NOT | |
1073 | corrected. | |
1074 | ||
1075 | A count of problems found will be stored in | |
1076 | .BR md/mismatch_count . | |
1077 | ||
1078 | Alternately, "repair" can be written which will cause the same check | |
1079 | to be performed, but any errors will be corrected. | |
1080 | ||
1081 | Finally, "idle" can be written to stop the check/repair process. | |
1082 | ||
1083 | .TP | |
1084 | .B md/stripe_cache_size | |
1085 | This is only available on RAID5 and RAID6. It records the size (in | |
1086 | pages per device) of the stripe cache which is used for synchronising | |
800053d6 DW |
1087 | all write operations to the array and all read operations if the array |
1088 | is degraded. The default is 256. Valid values are 17 to 32768. | |
addc80c4 | 1089 | Increasing this number can increase performance in some situations, at |
800053d6 DW |
1090 | some cost in system memory. Note, setting this value too high can |
1091 | result in an "out of memory" condition for the system. | |
1092 | ||
1093 | memory_consumed = system_page_size * nr_disks * stripe_cache_size | |
addc80c4 | 1094 | |
a5ee6dfb DW |
1095 | .TP |
1096 | .B md/preread_bypass_threshold | |
1097 | This is only available on RAID5 and RAID6. This variable sets the | |
1098 | number of times MD will service a full-stripe-write before servicing a | |
1099 | stripe that requires some "prereading". For fairness this defaults to | |
800053d6 DW |
1100 | 1. Valid values are 0 to stripe_cache_size. Setting this to 0 |
1101 | maximizes sequential-write throughput at the cost of fairness to threads | |
bcbb92d4 | 1102 | doing small or random writes. |
addc80c4 | 1103 | |
5787fa49 NB |
1104 | .SS KERNEL PARAMETERS |
1105 | ||
addc80c4 | 1106 | The md driver recognised several different kernel parameters. |
5787fa49 NB |
1107 | .TP |
1108 | .B raid=noautodetect | |
1109 | This will disable the normal detection of md arrays that happens at | |
1110 | boot time. If a drive is partitioned with MS-DOS style partitions, | |
1111 | then if any of the 4 main partitions has a partition type of 0xFD, | |
1112 | then that partition will normally be inspected to see if it is part of | |
1113 | an MD array, and if any full arrays are found, they are started. This | |
addc80c4 | 1114 | kernel parameter disables this behaviour. |
5787fa49 | 1115 | |
a9d69660 NB |
1116 | .TP |
1117 | .B raid=partitionable | |
1118 | .TP | |
1119 | .B raid=part | |
1120 | These are available in 2.6 and later kernels only. They indicate that | |
1121 | autodetected MD arrays should be created as partitionable arrays, with | |
1122 | a different major device number to the original non-partitionable md | |
1123 | arrays. The device number is listed as | |
1124 | .I mdp | |
1125 | in | |
1126 | .IR /proc/devices . | |
1127 | ||
addc80c4 NB |
1128 | .TP |
1129 | .B md_mod.start_ro=1 | |
e0fe762a N |
1130 | .TP |
1131 | .B /sys/module/md_mod/parameters/start_ro | |
addc80c4 NB |
1132 | This tells md to start all arrays in read-only mode. This is a soft |
1133 | read-only that will automatically switch to read-write on the first | |
1134 | write request. However until that write request, nothing is written | |
1135 | to any device by md, and in particular, no resync or recovery | |
1136 | operation is started. | |
1137 | ||
1138 | .TP | |
1139 | .B md_mod.start_dirty_degraded=1 | |
e0fe762a N |
1140 | .TP |
1141 | .B /sys/module/md_mod/parameters/start_dirty_degraded | |
addc80c4 NB |
1142 | As mentioned above, md will not normally start a RAID4, RAID5, or |
1143 | RAID6 that is both dirty and degraded as this situation can imply | |
1144 | hidden data loss. This can be awkward if the root filesystem is | |
93e790af | 1145 | affected. Using this module parameter allows such arrays to be started |
addc80c4 NB |
1146 | at boot time. It should be understood that there is a real (though |
1147 | small) risk of data corruption in this situation. | |
a9d69660 | 1148 | |
5787fa49 NB |
1149 | .TP |
1150 | .BI md= n , dev , dev ,... | |
a9d69660 NB |
1151 | .TP |
1152 | .BI md=d n , dev , dev ,... | |
5787fa49 NB |
1153 | This tells the md driver to assemble |
1154 | .B /dev/md n | |
1155 | from the listed devices. It is only necessary to start the device | |
1156 | holding the root filesystem this way. Other arrays are best started | |
1157 | once the system is booted. | |
1158 | ||
a9d69660 NB |
1159 | In 2.6 kernels, the |
1160 | .B d | |
1161 | immediately after the | |
1162 | .B = | |
1163 | indicates that a partitionable device (e.g. | |
1164 | .BR /dev/md/d0 ) | |
1165 | should be created rather than the original non-partitionable device. | |
1166 | ||
5787fa49 NB |
1167 | .TP |
1168 | .BI md= n , l , c , i , dev... | |
1169 | This tells the md driver to assemble a legacy RAID0 or LINEAR array | |
1170 | without a superblock. | |
1171 | .I n | |
1172 | gives the md device number, | |
1173 | .I l | |
dae45415 | 1174 | gives the level, 0 for RAID0 or \-1 for LINEAR, |
5787fa49 NB |
1175 | .I c |
1176 | gives the chunk size as a base-2 logarithm offset by twelve, so 0 | |
1177 | means 4K, 1 means 8K. | |
1178 | .I i | |
1179 | is ignored (legacy support). | |
e0d19036 | 1180 | |
56eb10c0 NB |
1181 | .SH FILES |
1182 | .TP | |
1183 | .B /proc/mdstat | |
1184 | Contains information about the status of currently running array. | |
1185 | .TP | |
1186 | .B /proc/sys/dev/raid/speed_limit_min | |
93e790af | 1187 | A readable and writable file that reflects the current "goal" rebuild |
56eb10c0 NB |
1188 | speed for times when non-rebuild activity is current on an array. |
1189 | The speed is in Kibibytes per second, and is a per-device rate, not a | |
93e790af | 1190 | per-array rate (which means that an array with more disks will shuffle |
e0fe762a | 1191 | more data for a given speed). The default is 1000. |
56eb10c0 NB |
1192 | |
1193 | .TP | |
1194 | .B /proc/sys/dev/raid/speed_limit_max | |
93e790af | 1195 | A readable and writable file that reflects the current "goal" rebuild |
56eb10c0 | 1196 | speed for times when no non-rebuild activity is current on an array. |
e0fe762a | 1197 | The default is 200,000. |
56eb10c0 NB |
1198 | |
1199 | .SH SEE ALSO | |
1200 | .BR mdadm (8), |