Documentation/fault-injection/fault-injection.rst

   1 ===========================================
   2 Fault injection capabilities infrastructure
   3 ===========================================
   4
   5 See also drivers/md/md-faulty.c and "every_nth" module option for scsi_debug.
   6
   7
   8 Available fault injection capabilities
   9 --------------------------------------
  10
  11 - failslab
  12
  13   injects slab allocation failures. (kmalloc(), kmem_cache_alloc(), ...)
  14
  15 - fail_page_alloc
  16
  17   injects page allocation failures. (alloc_pages(), get_free_pages(), ...)
  18
  19 - fail_usercopy
  20
  21   injects failures in user memory access functions. (copy_from_user(), get_user(), ...)
  22
  23 - fail_futex
  24
  25   injects futex deadlock and uaddr fault errors.
  26
  27 - fail_sunrpc
  28
  29   injects kernel RPC client and server failures.
  30
  31 - fail_make_request
  32
  33   injects disk IO errors on devices permitted by setting
  34   /sys/block/<device>/make-it-fail or
  35   /sys/block/<device>/<partition>/make-it-fail. (submit_bio_noacct())
  36
  37 - fail_mmc_request
  38
  39   injects MMC data errors on devices permitted by setting
  40   debugfs entries under /sys/kernel/debug/mmc0/fail_mmc_request
  41
  42 - fail_function
  43
  44   injects error return on specific functions, which are marked by
  45   ALLOW_ERROR_INJECTION() macro, by setting debugfs entries
  46   under /sys/kernel/debug/fail_function. No boot option supported.
  47
  48 - NVMe fault injection
  49
  50   inject NVMe status code and retry flag on devices permitted by setting
  51   debugfs entries under /sys/kernel/debug/nvme*/fault_inject. The default
  52   status code is NVME_SC_INVALID_OPCODE with no retry. The status code and
  53   retry flag can be set via the debugfs.
  54
  55
  56 Configure fault-injection capabilities behavior
  57 -----------------------------------------------
  58
  59 debugfs entries
  60 ^^^^^^^^^^^^^^^
  61
  62 fault-inject-debugfs kernel module provides some debugfs entries for runtime
  63 configuration of fault-injection capabilities.
  64
  65 - /sys/kernel/debug/fail*/probability:
  66
  67         likelihood of failure injection, in percent.
  68
  69         Format: <percent>
  70
  71         Note that one-failure-per-hundred is a very high error rate
  72         for some testcases.  Consider setting probability=100 and configure
  73         /sys/kernel/debug/fail*/interval for such testcases.
  74
  75 - /sys/kernel/debug/fail*/interval:
  76
  77         specifies the interval between failures, for calls to
  78         should_fail() that pass all the other tests.
  79
  80         Note that if you enable this, by setting interval>1, you will
  81         probably want to set probability=100.
  82
  83 - /sys/kernel/debug/fail*/times:
  84
  85         specifies how many times failures may happen at most. A value of -1
  86         means "no limit". Note, though, that this file only accepts unsigned
  87         values. So, if you want to specify -1, you better use 'printf' instead
  88         of 'echo', e.g.: $ printf %#x -1 > times
  89
  90 - /sys/kernel/debug/fail*/space:
  91
  92         specifies an initial resource "budget", decremented by "size"
  93         on each call to should_fail(,size).  Failure injection is
  94         suppressed until "space" reaches zero.
  95
  96 - /sys/kernel/debug/fail*/verbose
  97
  98         Format: { 0 | 1 | 2 }
  99
 100         specifies the verbosity of the messages when failure is
 101         injected.  '0' means no messages; '1' will print only a single
 102         log line per failure; '2' will print a call trace too -- useful
 103         to debug the problems revealed by fault injection.
 104
 105 - /sys/kernel/debug/fail*/task-filter:
 106
 107         Format: { 'Y' | 'N' }
 108
 109         A value of 'N' disables filtering by process (default).
 110         Any positive value limits failures to only processes indicated by
 111         /proc/<pid>/make-it-fail==1.
 112
 113 - /sys/kernel/debug/fail*/require-start,
 114   /sys/kernel/debug/fail*/require-end,
 115   /sys/kernel/debug/fail*/reject-start,
 116   /sys/kernel/debug/fail*/reject-end:
 117
 118         specifies the range of virtual addresses tested during
 119         stacktrace walking.  Failure is injected only if some caller
 120         in the walked stacktrace lies within the required range, and
 121         none lies within the rejected range.
 122         Default required range is [0,ULONG_MAX) (whole of virtual address space).
 123         Default rejected range is [0,0).
 124
 125 - /sys/kernel/debug/fail*/stacktrace-depth:
 126
 127         specifies the maximum stacktrace depth walked during search
 128         for a caller within [require-start,require-end) OR
 129         [reject-start,reject-end).
 130
 131 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-highmem:
 132
 133         Format: { 'Y' | 'N' }
 134
 135         default is 'Y', setting it to 'N' will also inject failures into
 136         highmem/user allocations (__GFP_HIGHMEM allocations).
 137
 138 - /sys/kernel/debug/failslab/ignore-gfp-wait:
 139 - /sys/kernel/debug/fail_page_alloc/ignore-gfp-wait:
 140
 141         Format: { 'Y' | 'N' }
 142
 143         default is 'Y', setting it to 'N' will also inject failures
 144         into allocations that can sleep (__GFP_DIRECT_RECLAIM allocations).
 145
 146 - /sys/kernel/debug/fail_page_alloc/min-order:
 147
 148         specifies the minimum page allocation order to be injected
 149         failures.
 150
 151 - /sys/kernel/debug/fail_futex/ignore-private:
 152
 153         Format: { 'Y' | 'N' }
 154
 155         default is 'N', setting it to 'Y' will disable failure injections
 156         when dealing with private (address space) futexes.
 157
 158 - /sys/kernel/debug/fail_sunrpc/ignore-client-disconnect:
 159
 160         Format: { 'Y' | 'N' }
 161
 162         default is 'N', setting it to 'Y' will disable disconnect
 163         injection on the RPC client.
 164
 165 - /sys/kernel/debug/fail_sunrpc/ignore-server-disconnect:
 166
 167         Format: { 'Y' | 'N' }
 168
 169         default is 'N', setting it to 'Y' will disable disconnect
 170         injection on the RPC server.
 171
 172 - /sys/kernel/debug/fail_sunrpc/ignore-cache-wait:
 173
 174         Format: { 'Y' | 'N' }
 175
 176         default is 'N', setting it to 'Y' will disable cache wait
 177         injection on the RPC server.
 178
 179 - /sys/kernel/debug/fail_function/inject:
 180
 181         Format: { 'function-name' | '!function-name' | '' }
 182
 183         specifies the target function of error injection by name.
 184         If the function name leads '!' prefix, given function is
 185         removed from injection list. If nothing specified ('')
 186         injection list is cleared.
 187
 188 - /sys/kernel/debug/fail_function/injectable:
 189
 190         (read only) shows error injectable functions and what type of
 191         error values can be specified. The error type will be one of
 192         below;
 193         - NULL: retval must be 0.
 194         - ERRNO: retval must be -1 to -MAX_ERRNO (-4096).
 195         - ERR_NULL: retval must be 0 or -1 to -MAX_ERRNO (-4096).
 196
 197 - /sys/kernel/debug/fail_function/<function-name>/retval:
 198
 199         specifies the "error" return value to inject to the given function.
 200         This will be created when the user specifies a new injection entry.
 201         Note that this file only accepts unsigned values. So, if you want to
 202         use a negative errno, you better use 'printf' instead of 'echo', e.g.:
 203         $ printf %#x -12 > retval
 204
 205 Boot option
 206 ^^^^^^^^^^^
 207
 208 In order to inject faults while debugfs is not available (early boot time),
 209 use the boot option::
 210
 211         failslab=
 212         fail_page_alloc=
 213         fail_usercopy=
 214         fail_make_request=
 215         fail_futex=
 216         mmc_core.fail_request=<interval>,<probability>,<space>,<times>
 217
 218 proc entries
 219 ^^^^^^^^^^^^
 220
 221 - /proc/<pid>/fail-nth,
 222   /proc/self/task/<tid>/fail-nth:
 223
 224         Write to this file of integer N makes N-th call in the task fail.
 225         Read from this file returns a integer value. A value of '0' indicates
 226         that the fault setup with a previous write to this file was injected.
 227         A positive integer N indicates that the fault wasn't yet injected.
 228         Note that this file enables all types of faults (slab, futex, etc).
 229         This setting takes precedence over all other generic debugfs settings
 230         like probability, interval, times, etc. But per-capability settings
 231         (e.g. fail_futex/ignore-private) take precedence over it.
 232
 233         This feature is intended for systematic testing of faults in a single
 234         system call. See an example below.
 235
 236 How to add new fault injection capability
 237 -----------------------------------------
 238
 239 - #include <linux/fault-inject.h>
 240
 241 - define the fault attributes
 242
 243   DECLARE_FAULT_ATTR(name);
 244
 245   Please see the definition of struct fault_attr in fault-inject.h
 246   for details.
 247
 248 - provide a way to configure fault attributes
 249
 250 - boot option
 251
 252   If you need to enable the fault injection capability from boot time, you can
 253   provide boot option to configure it. There is a helper function for it:
 254
 255         setup_fault_attr(attr, str);
 256
 257 - debugfs entries
 258
 259   failslab, fail_page_alloc, fail_usercopy, and fail_make_request use this way.
 260   Helper functions:
 261
 262         fault_create_debugfs_attr(name, parent, attr);
 263
 264 - module parameters
 265
 266   If the scope of the fault injection capability is limited to a
 267   single kernel module, it is better to provide module parameters to
 268   configure the fault attributes.
 269
 270 - add a hook to insert failures
 271
 272   Upon should_fail() returning true, client code should inject a failure:
 273
 274         should_fail(attr, size);
 275
 276 Application Examples
 277 --------------------
 278
 279 - Inject slab allocation failures into module init/exit code::
 280
 281     #!/bin/bash
 282
 283     FAILTYPE=failslab
 284     echo Y > /sys/kernel/debug/$FAILTYPE/task-filter
 285     echo 10 > /sys/kernel/debug/$FAILTYPE/probability
 286     echo 100 > /sys/kernel/debug/$FAILTYPE/interval
 287     printf %#x -1 > /sys/kernel/debug/$FAILTYPE/times
 288     echo 0 > /sys/kernel/debug/$FAILTYPE/space
 289     echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
 290     echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
 291
 292     faulty_system()
 293     {
 294         bash -c "echo 1 > /proc/self/make-it-fail && exec $*"
 295     }
 296
 297     if [ $# -eq 0 ]
 298     then
 299         echo "Usage: $0 modulename [ modulename ... ]"
 300         exit 1
 301     fi
 302
 303     for m in $*
 304     do
 305         echo inserting $m...
 306         faulty_system modprobe $m
 307
 308         echo removing $m...
 309         faulty_system modprobe -r $m
 310     done
 311
 312 ------------------------------------------------------------------------------
 313
 314 - Inject page allocation failures only for a specific module::
 315
 316     #!/bin/bash
 317
 318     FAILTYPE=fail_page_alloc
 319     module=$1
 320
 321     if [ -z $module ]
 322     then
 323         echo "Usage: $0 <modulename>"
 324         exit 1
 325     fi
 326
 327     modprobe $module
 328
 329     if [ ! -d /sys/module/$module/sections ]
 330     then
 331         echo Module $module is not loaded
 332         exit 1
 333     fi
 334
 335     cat /sys/module/$module/sections/.text > /sys/kernel/debug/$FAILTYPE/require-start
 336     cat /sys/module/$module/sections/.data > /sys/kernel/debug/$FAILTYPE/require-end
 337
 338     echo N > /sys/kernel/debug/$FAILTYPE/task-filter
 339     echo 10 > /sys/kernel/debug/$FAILTYPE/probability
 340     echo 100 > /sys/kernel/debug/$FAILTYPE/interval
 341     printf %#x -1 > /sys/kernel/debug/$FAILTYPE/times
 342     echo 0 > /sys/kernel/debug/$FAILTYPE/space
 343     echo 2 > /sys/kernel/debug/$FAILTYPE/verbose
 344     echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-wait
 345     echo Y > /sys/kernel/debug/$FAILTYPE/ignore-gfp-highmem
 346     echo 10 > /sys/kernel/debug/$FAILTYPE/stacktrace-depth
 347
 348     trap "echo 0 > /sys/kernel/debug/$FAILTYPE/probability" SIGINT SIGTERM EXIT
 349
 350     echo "Injecting errors into the module $module... (interrupt to stop)"
 351     sleep 1000000
 352
 353 ------------------------------------------------------------------------------
 354
 355 - Inject open_ctree error while btrfs mount::
 356
 357     #!/bin/bash
 358
 359     rm -f testfile.img
 360     dd if=/dev/zero of=testfile.img bs=1M seek=1000 count=1
 361     DEVICE=$(losetup --show -f testfile.img)
 362     mkfs.btrfs -f $DEVICE
 363     mkdir -p tmpmnt
 364
 365     FAILTYPE=fail_function
 366     FAILFUNC=open_ctree
 367     echo $FAILFUNC > /sys/kernel/debug/$FAILTYPE/inject
 368     printf %#x -12 > /sys/kernel/debug/$FAILTYPE/$FAILFUNC/retval
 369     echo N > /sys/kernel/debug/$FAILTYPE/task-filter
 370     echo 100 > /sys/kernel/debug/$FAILTYPE/probability
 371     echo 0 > /sys/kernel/debug/$FAILTYPE/interval
 372     printf %#x -1 > /sys/kernel/debug/$FAILTYPE/times
 373     echo 0 > /sys/kernel/debug/$FAILTYPE/space
 374     echo 1 > /sys/kernel/debug/$FAILTYPE/verbose
 375
 376     mount -t btrfs $DEVICE tmpmnt
 377     if [ $? -ne 0 ]
 378     then
 379         echo "SUCCESS!"
 380     else
 381         echo "FAILED!"
 382         umount tmpmnt
 383     fi
 384
 385     echo > /sys/kernel/debug/$FAILTYPE/inject
 386
 387     rmdir tmpmnt
 388     losetup -d $DEVICE
 389     rm testfile.img
 390
 391
 392 Tool to run command with failslab or fail_page_alloc
 393 ----------------------------------------------------
 394 In order to make it easier to accomplish the tasks mentioned above, we can use
 395 tools/testing/fault-injection/failcmd.sh.  Please run a command
 396 "./tools/testing/fault-injection/failcmd.sh --help" for more information and
 397 see the following examples.
 398
 399 Examples:
 400
 401 Run a command "make -C tools/testing/selftests/ run_tests" with injecting slab
 402 allocation failure::
 403
 404         # ./tools/testing/fault-injection/failcmd.sh \
 405                 -- make -C tools/testing/selftests/ run_tests
 406
 407 Same as above except to specify 100 times failures at most instead of one time
 408 at most by default::
 409
 410         # ./tools/testing/fault-injection/failcmd.sh --times=100 \
 411                 -- make -C tools/testing/selftests/ run_tests
 412
 413 Same as above except to inject page allocation failure instead of slab
 414 allocation failure::
 415
 416         # env FAILCMD_TYPE=fail_page_alloc \
 417                 ./tools/testing/fault-injection/failcmd.sh --times=100 \
 418                 -- make -C tools/testing/selftests/ run_tests
 419
 420 Systematic faults using fail-nth
 421 ---------------------------------
 422
 423 The following code systematically faults 0-th, 1-st, 2-nd and so on
 424 capabilities in the socketpair() system call::
 425
 426   #include <sys/types.h>
 427   #include <sys/stat.h>
 428   #include <sys/socket.h>
 429   #include <sys/syscall.h>
 430   #include <fcntl.h>
 431   #include <unistd.h>
 432   #include <string.h>
 433   #include <stdlib.h>
 434   #include <stdio.h>
 435   #include <errno.h>
 436
 437   int main()
 438   {
 439         int i, err, res, fail_nth, fds[2];
 440         char buf[128];
 441
 442         system("echo N > /sys/kernel/debug/failslab/ignore-gfp-wait");
 443         sprintf(buf, "/proc/self/task/%ld/fail-nth", syscall(SYS_gettid));
 444         fail_nth = open(buf, O_RDWR);
 445         for (i = 1;; i++) {
 446                 sprintf(buf, "%d", i);
 447                 write(fail_nth, buf, strlen(buf));
 448                 res = socketpair(AF_LOCAL, SOCK_STREAM, 0, fds);
 449                 err = errno;
 450                 pread(fail_nth, buf, sizeof(buf), 0);
 451                 if (res == 0) {
 452                         close(fds[0]);
 453                         close(fds[1]);
 454                 }
 455                 printf("%d-th fault %c: res=%d/%d\n", i, atoi(buf) ? 'N' : 'Y',
 456                         res, err);
 457                 if (atoi(buf))
 458                         break;
 459         }
 460         return 0;
 461   }
 462
 463 An example output::
 464
 465         1-th fault Y: res=-1/23
 466         2-th fault Y: res=-1/23
 467         3-th fault Y: res=-1/12
 468         4-th fault Y: res=-1/12
 469         5-th fault Y: res=-1/23
 470         6-th fault Y: res=-1/23
 471         7-th fault Y: res=-1/23
 472         8-th fault Y: res=-1/12
 473         9-th fault Y: res=-1/12
 474         10-th fault Y: res=-1/12
 475         11-th fault Y: res=-1/12
 476         12-th fault Y: res=-1/12
 477         13-th fault Y: res=-1/12
 478         14-th fault Y: res=-1/12
 479         15-th fault Y: res=-1/12
 480         16-th fault N: res=0/12