docs/COREDUMP.md

   1 ---
   2 title: systemd Coredump Handling
   3 category: Concepts
   4 layout: default
   5 SPDX-License-Identifier: LGPL-2.1-or-later
   6 ---
   7
   8 # systemd Coredump Handling
   9
  10 ## Support in the Service Manager (PID 1)
  11
  12 The systemd service manager natively provides coredump handling functionality,
  13 as implemented by the Linux kernel. Specifically, PID 1 provides the following
  14 functionality:
  15
  16 1. During very early boot it will raise the
  17    [`LIMIT_CORE`](https://man7.org/linux/man-pages/man2/getrlimit.2.html)
  18    resource limit for itself to infinity (and thus implicitly also all its
  19    children). This removes any limits on the size of generated coredumps, for
  20    all invoked processes, from earliest boot on. (The Linux kernel sets the
  21    limit to 0 by default.)
  22
  23 2. At the same time it will turn off coredump handling in the kernel by writing
  24    `|/bin/false` into `/proc/sys/kernel/core_pattern` (also known as the
  25    "`kernel.core_pattern` sysctl"; see
  26    [core(5)](https://man7.org/linux/man-pages/man5/core.5.html) for
  27    details). This means that coredumps are not actually processed. (The Linux
  28    kernel sets the pattern to `core` by default, so that coredumps are written
  29    to the current working directory of the crashing process.)
  30
  31 Net effect: after PID1 has started and performed this setup coredumps are
  32 disabled, but by means of the the `kernel.core_pattern` sysctl rather than by
  33 size limit. This is generally preferable, since the pattern can be updated
  34 trivially at the right time to enable coredumping once the system is ready,
  35 taking comprehensive effect on all userspace. (Or to say this differently:
  36 disabling coredumps via the size limit is problematic, since it cannot easily
  37 be undone without iterating through all already running processes once the
  38 system is ready for coredump handling.)
  39
  40 Processing of core dumps may be enabled at the appropriate time by updating the
  41 `kernel.core_pattern` sysctl. Only coredumps that happen later will be
  42 processed.
  43
  44 During the final shutdown phase the `kernel.core_pattern` sysctl is updated
  45 again to `|/bin/false`, disabling coredump support again, should it have been
  46 enabled in the meantime.
  47
  48 This means coredump handling is generally not available during earliest boot
  49 and latest shutdown, reflecting the fact that storage is typically not
  50 available in these environments, and many other facilities are missing too that
  51 are required to collect and process a coredump successfully.
  52
  53 ## `systemd-coredump` Handler
  54
  55 The systemd suite provides a coredump handler
  56 [`systemd-coredump`](https://www.freedesktop.org/software/systemd/man/systemd-coredump.html)
  57 which can be enabled at build-time. It is activated during boot via the
  58 `/usr/lib/sysctl.d/50-coredump.conf` drop-in file for
  59 `systemd-sysctl.service`. It registers the `systemd-coredump` tool as
  60 `kernel.core_pattern` sysctl.
  61
  62 `systemd-coredump` is implemented as socket activated service: when the kernel
  63 invokes the userspace coredump handler, the received coredump file descriptor
  64 is immediately handed off to the socket activated service
  65 `systemd-coredump@.service` via the `systemd-coredump.socket` socket unit. This
  66 means the coredump handler runs for a very short time only, and the potentially
  67 *heavy* and security sensitive coredump processing work is done as part of the
  68 specified service unit, and thus can take benefit of regular service resource
  69 management and sandboxing.
  70
  71 The `systemd-coredump` handler will extract a backtrace and [ELF packaging
  72 metadata](https://systemd.io/ELF_PACKAGE_METADATA) from any coredumps it
  73 receives and log both. The information about coredumps stored in the journal
  74 can be enumerated and queried with the
  75 [`coredumpctl`](https://www.freedesktop.org/software/systemd/man/coredumpctl.html)
  76 tool, for example for directly invoking a debugger such as `gdb` on a collected
  77 coredump.
  78
  79 The handler writes coredump files to `/var/lib/systemd/coredump/`.  Old files
  80 are cleaned up periodically by
  81 [`systemd-tmpfiles(8)`](https://www.freedesktop.org/software/systemd/man/systemd-tmpfiles.html).
  82
  83 ## User Experience
  84
  85 With the above, any coredumps generated on the system are by default collected
  86 and turned into logged events — except during very early boot and late
  87 shutdown. Individual services, processes or users can opt-out of coredump
  88 collection, by setting `LIMIT_CORE` to 0 (or alternatively invoke
  89 [`PR_SET_DUMPABLE`](https://man7.org/linux/man-pages/man2/prctl.2.html)). The
  90 resource limit can be set freely by daemons/processes/users to arbitrary
  91 values, which the coredump handler will respect. The `coredumpctl` tool may be
  92 used to further analyze/debug coredumps.
  93
  94 ## Alternative Coredump Handlers
  95
  96 While we recommend usage of the `systemd-coredump` handler, it's fully
  97 supported to use alternative coredump handlers instead. A similar
  98 implementation pattern is recommended. Specifically:
  99
 100 1. Use a `sysctl.d/` drop-in to register your handler with the kernel. Make
 101    sure to include the `%c` specifier in the pattern (which reflects the
 102    crashing process' `RLIMIT_CORE`) and act on it: limit the stored coredump
 103    file to the specified limit.
 104
 105 2. Do not do heavy processing directly in the coredump handler. Instead,
 106    quickly pass off the kernel's coredump file descriptor to an
 107    auxiliary service running as service under the service manager, so that it
 108    can be done under supervision, sandboxing and resource management.
 109
 110 Note that at any given time only a single handler can be enabled, i.e. the
 111 `kernel.core_pattern` sysctl cannot reference multiple executables.
 112
 113 ## Packaging
 114
 115 It might make sense to split `systemd-coredump` into a separate distribution
 116 package. If doing so, make sure that `/usr/lib/sysctl.d/50-coredump.conf` and
 117 the associated service and socket units are also added to the split off package.
 118
 119 Note that in a scenario where `systemd-coredump` is split out and not
 120 installed, coredumping is turned off during the entire runtime of the system —
 121 unless an alternative handler is installed, or behaviour is manually reverted
 122 to legacy style handling (see below).
 123
 124 ## Restoring Legacy Coredump Handling
 125
 126 The default policy of the kernel to write coredumps into the current working
 127 directory of the crashing process is considered highly problematic by many,
 128 including by the systemd maintainers. Nonetheless, if users locally want to
 129 return to this behaviour, two changes must be made (followed by a reboot):
 130
 131 ```console
 132 $ mkdir -p /etc/sysctl.d
 133 $ cat >/etc/sysctl.d/50-coredump.conf <<EOF
 134 # Party like it's 1995!
 135 kernel.core_pattern=core
 136 EOF
 137 ```
 138
 139 and
 140
 141 ```console
 142 $ mkdir -p /etc/systemd/system.conf.d
 143 $ cat >/etc/systemd/system.conf.d/50-coredump.conf <<EOF
 144 [Manager]
 145 DefaultLimitCORE=0:infinity
 146 EOF
 147 ```