]>
Commit | Line | Data |
---|---|---|
b6bd98eb LP |
1 | --- |
2 | title: systemd Coredump Handling | |
3 | category: Concepts | |
4 | layout: default | |
5 | SPDX-License-Identifier: LGPL-2.1-or-later | |
6 | --- | |
7 | ||
8 | # systemd Coredump Handling | |
9 | ||
10 | ## Support in the Service Manager (PID 1) | |
11 | ||
12 | The systemd service manager natively provides coredump handling functionality, | |
77d47e6d | 13 | as implemented by the Linux kernel. |
14 | Specifically, PID 1 provides the following functionality: | |
b6bd98eb LP |
15 | |
16 | 1. During very early boot it will raise the | |
17 | [`LIMIT_CORE`](https://man7.org/linux/man-pages/man2/getrlimit.2.html) | |
77d47e6d | 18 | resource limit for itself to infinity (and thus implicitly also all its children). |
19 | This removes any limits on the size of generated coredumps, | |
20 | for all invoked processes, from earliest boot on. | |
21 | (The Linux kernel sets the limit to 0 by default.) | |
b6bd98eb LP |
22 | |
23 | 2. At the same time it will turn off coredump handling in the kernel by writing | |
24 | `|/bin/false` into `/proc/sys/kernel/core_pattern` (also known as the | |
25 | "`kernel.core_pattern` sysctl"; see | |
26 | [core(5)](https://man7.org/linux/man-pages/man5/core.5.html) for | |
77d47e6d | 27 | details). |
28 | This means that coredumps are not actually processed. | |
29 | (The Linux kernel sets the pattern to `core` by default, so that coredumps are written | |
b6bd98eb LP |
30 | to the current working directory of the crashing process.) |
31 | ||
32 | Net effect: after PID1 has started and performed this setup coredumps are | |
c309b9e9 | 33 | disabled, but by means of the `kernel.core_pattern` sysctl rather than by |
77d47e6d | 34 | size limit. |
35 | This is generally preferable, since the pattern can be updated trivially at the right time to enable coredumping once the system is ready, taking comprehensive effect on all userspace. | |
36 | (Or to say this differently: disabling coredumps via the size limit is problematic, since it cannot easily | |
37 | be undone without iterating through all already running processes once the system is ready for coredump handling.) | |
b6bd98eb LP |
38 | |
39 | Processing of core dumps may be enabled at the appropriate time by updating the | |
77d47e6d | 40 | `kernel.core_pattern` sysctl. |
41 | Only coredumps that happen later will be processed. | |
b6bd98eb LP |
42 | |
43 | During the final shutdown phase the `kernel.core_pattern` sysctl is updated | |
44 | again to `|/bin/false`, disabling coredump support again, should it have been | |
45 | enabled in the meantime. | |
46 | ||
47 | This means coredump handling is generally not available during earliest boot | |
48 | and latest shutdown, reflecting the fact that storage is typically not | |
49 | available in these environments, and many other facilities are missing too that | |
50 | are required to collect and process a coredump successfully. | |
51 | ||
52 | ## `systemd-coredump` Handler | |
53 | ||
54 | The systemd suite provides a coredump handler | |
55 | [`systemd-coredump`](https://www.freedesktop.org/software/systemd/man/systemd-coredump.html) | |
56 | which can be enabled at build-time. It is activated during boot via the | |
57 | `/usr/lib/sysctl.d/50-coredump.conf` drop-in file for | |
58 | `systemd-sysctl.service`. It registers the `systemd-coredump` tool as | |
59 | `kernel.core_pattern` sysctl. | |
60 | ||
61 | `systemd-coredump` is implemented as socket activated service: when the kernel | |
62 | invokes the userspace coredump handler, the received coredump file descriptor | |
63 | is immediately handed off to the socket activated service | |
64 | `systemd-coredump@.service` via the `systemd-coredump.socket` socket unit. This | |
65 | means the coredump handler runs for a very short time only, and the potentially | |
66 | *heavy* and security sensitive coredump processing work is done as part of the | |
67 | specified service unit, and thus can take benefit of regular service resource | |
68 | management and sandboxing. | |
69 | ||
77d47e6d | 70 | The `systemd-coredump` handler will extract a backtrace and |
0d592a5e | 71 | [ELF packaging metadata](/ELF_PACKAGE_METADATA) from any coredumps it |
77d47e6d | 72 | receives and log both. |
73 | The information about coredumps stored in the journal can be enumerated and queried with the | |
b6bd98eb LP |
74 | [`coredumpctl`](https://www.freedesktop.org/software/systemd/man/coredumpctl.html) |
75 | tool, for example for directly invoking a debugger such as `gdb` on a collected | |
76 | coredump. | |
77 | ||
77d47e6d | 78 | The handler writes coredump files to `/var/lib/systemd/coredump/`. |
79 | Old files are cleaned up periodically by | |
b6bd98eb LP |
80 | [`systemd-tmpfiles(8)`](https://www.freedesktop.org/software/systemd/man/systemd-tmpfiles.html). |
81 | ||
82 | ## User Experience | |
83 | ||
84 | With the above, any coredumps generated on the system are by default collected | |
85 | and turned into logged events — except during very early boot and late | |
77d47e6d | 86 | shutdown. |
87 | Individual services, processes or users can opt-out of coredump collection, | |
88 | by setting `LIMIT_CORE` to 0 (or alternatively invoke | |
89 | [`PR_SET_DUMPABLE`](https://man7.org/linux/man-pages/man2/prctl.2.html)). | |
90 | The resource limit can be set freely by daemons/processes/users to arbitrary | |
91 | values, which the coredump handler will respect. | |
92 | The `coredumpctl` tool may be used to further analyze/debug coredumps. | |
b6bd98eb LP |
93 | |
94 | ## Alternative Coredump Handlers | |
95 | ||
96 | While we recommend usage of the `systemd-coredump` handler, it's fully | |
77d47e6d | 97 | supported to use alternative coredump handlers instead. |
98 | A similar implementation pattern is recommended. | |
99 | Specifically: | |
b6bd98eb | 100 | |
77d47e6d | 101 | 1. Use a `sysctl.d/` drop-in to register your handler with the kernel. |
102 | Make sure to include the `%c` specifier in the pattern (which reflects the | |
103 | crashing process' `RLIMIT_CORE`) and act on it: | |
104 | limit the stored coredump file to the specified limit. | |
b6bd98eb | 105 | |
77d47e6d | 106 | 2. Do not do heavy processing directly in the coredump handler. |
107 | Instead, quickly pass off the kernel's coredump file descriptor to an | |
108 | auxiliary service running as service under the service manager, | |
109 | so that it can be done under supervision, sandboxing and resource management. | |
b6bd98eb LP |
110 | |
111 | Note that at any given time only a single handler can be enabled, i.e. the | |
112 | `kernel.core_pattern` sysctl cannot reference multiple executables. | |
113 | ||
114 | ## Packaging | |
115 | ||
116 | It might make sense to split `systemd-coredump` into a separate distribution | |
77d47e6d | 117 | package. |
118 | If doing so, make sure that `/usr/lib/sysctl.d/50-coredump.conf` and | |
b6bd98eb LP |
119 | the associated service and socket units are also added to the split off package. |
120 | ||
121 | Note that in a scenario where `systemd-coredump` is split out and not | |
122 | installed, coredumping is turned off during the entire runtime of the system — | |
123 | unless an alternative handler is installed, or behaviour is manually reverted | |
124 | to legacy style handling (see below). | |
125 | ||
126 | ## Restoring Legacy Coredump Handling | |
127 | ||
128 | The default policy of the kernel to write coredumps into the current working | |
129 | directory of the crashing process is considered highly problematic by many, | |
77d47e6d | 130 | including by the systemd maintainers. |
131 | Nonetheless, if users locally want to return to this behaviour, two changes must be made (followed by a reboot): | |
b6bd98eb LP |
132 | |
133 | ```console | |
134 | $ mkdir -p /etc/sysctl.d | |
135 | $ cat >/etc/sysctl.d/50-coredump.conf <<EOF | |
136 | # Party like it's 1995! | |
137 | kernel.core_pattern=core | |
138 | EOF | |
139 | ``` | |
140 | ||
141 | and | |
142 | ||
143 | ```console | |
144 | $ mkdir -p /etc/systemd/system.conf.d | |
145 | $ cat >/etc/systemd/system.conf.d/50-coredump.conf <<EOF | |
146 | [Manager] | |
6fb595bc | 147 | DefaultLimitCORE=0:infinity |
b6bd98eb LP |
148 | EOF |
149 | ``` |