]>
Commit | Line | Data |
---|---|---|
b6bd98eb LP |
1 | --- |
2 | title: systemd Coredump Handling | |
3 | category: Concepts | |
4 | layout: default | |
5 | SPDX-License-Identifier: LGPL-2.1-or-later | |
6 | --- | |
7 | ||
8 | # systemd Coredump Handling | |
9 | ||
10 | ## Support in the Service Manager (PID 1) | |
11 | ||
12 | The systemd service manager natively provides coredump handling functionality, | |
13 | as implemented by the Linux kernel. Specifically, PID 1 provides the following | |
14 | functionality: | |
15 | ||
16 | 1. During very early boot it will raise the | |
17 | [`LIMIT_CORE`](https://man7.org/linux/man-pages/man2/getrlimit.2.html) | |
18 | resource limit for itself to infinity (and thus implicitly also all its | |
19 | children). This removes any limits on the size of generated coredumps, for | |
20 | all invoked processes, from earliest boot on. (The Linux kernel sets the | |
21 | limit to 0 by default.) | |
22 | ||
23 | 2. At the same time it will turn off coredump handling in the kernel by writing | |
24 | `|/bin/false` into `/proc/sys/kernel/core_pattern` (also known as the | |
25 | "`kernel.core_pattern` sysctl"; see | |
26 | [core(5)](https://man7.org/linux/man-pages/man5/core.5.html) for | |
27 | details). This means that coredumps are not actually processed. (The Linux | |
28 | kernel sets the pattern to `core` by default, so that coredumps are written | |
29 | to the current working directory of the crashing process.) | |
30 | ||
31 | Net effect: after PID1 has started and performed this setup coredumps are | |
32 | disabled, but by means of the the `kernel.core_pattern` sysctl rather than by | |
33 | size limit. This is generally preferable, since the pattern can be updated | |
34 | trivially at the right time to enable coredumping once the system is ready, | |
35 | taking comprehensive effect on all userspace. (Or to say this differently: | |
36 | disabling coredumps via the size limit is problematic, since it cannot easily | |
37 | be undone without iterating through all already running processes once the | |
38 | system is ready for coredump handling.) | |
39 | ||
40 | Processing of core dumps may be enabled at the appropriate time by updating the | |
41 | `kernel.core_pattern` sysctl. Only coredumps that happen later will be | |
42 | processed. | |
43 | ||
44 | During the final shutdown phase the `kernel.core_pattern` sysctl is updated | |
45 | again to `|/bin/false`, disabling coredump support again, should it have been | |
46 | enabled in the meantime. | |
47 | ||
48 | This means coredump handling is generally not available during earliest boot | |
49 | and latest shutdown, reflecting the fact that storage is typically not | |
50 | available in these environments, and many other facilities are missing too that | |
51 | are required to collect and process a coredump successfully. | |
52 | ||
53 | ## `systemd-coredump` Handler | |
54 | ||
55 | The systemd suite provides a coredump handler | |
56 | [`systemd-coredump`](https://www.freedesktop.org/software/systemd/man/systemd-coredump.html) | |
57 | which can be enabled at build-time. It is activated during boot via the | |
58 | `/usr/lib/sysctl.d/50-coredump.conf` drop-in file for | |
59 | `systemd-sysctl.service`. It registers the `systemd-coredump` tool as | |
60 | `kernel.core_pattern` sysctl. | |
61 | ||
62 | `systemd-coredump` is implemented as socket activated service: when the kernel | |
63 | invokes the userspace coredump handler, the received coredump file descriptor | |
64 | is immediately handed off to the socket activated service | |
65 | `systemd-coredump@.service` via the `systemd-coredump.socket` socket unit. This | |
66 | means the coredump handler runs for a very short time only, and the potentially | |
67 | *heavy* and security sensitive coredump processing work is done as part of the | |
68 | specified service unit, and thus can take benefit of regular service resource | |
69 | management and sandboxing. | |
70 | ||
71 | The `systemd-coredump` handler will extract a backtrace and [ELF packaging | |
72 | metadata](https://systemd.io/ELF_PACKAGE_METADATA) from any coredumps it | |
94d82b59 | 73 | receives and log both. The information about coredumps stored in the journal |
b6bd98eb LP |
74 | can be enumerated and queried with the |
75 | [`coredumpctl`](https://www.freedesktop.org/software/systemd/man/coredumpctl.html) | |
76 | tool, for example for directly invoking a debugger such as `gdb` on a collected | |
77 | coredump. | |
78 | ||
79 | The handler writes coredump files to `/var/lib/systemd/coredump/`. Old files | |
80 | are cleaned up periodically by | |
81 | [`systemd-tmpfiles(8)`](https://www.freedesktop.org/software/systemd/man/systemd-tmpfiles.html). | |
82 | ||
83 | ## User Experience | |
84 | ||
85 | With the above, any coredumps generated on the system are by default collected | |
86 | and turned into logged events — except during very early boot and late | |
87 | shutdown. Individual services, processes or users can opt-out of coredump | |
88 | collection, by setting `LIMIT_CORE` to 0 (or alternatively invoke | |
89 | [`PR_SET_DUMPABLE`](https://man7.org/linux/man-pages/man2/prctl.2.html)). The | |
90 | resource limit can be set freely by daemons/processes/users to arbitrary | |
91 | values, which the coredump handler will respect. The `coredumpctl` tool may be | |
92 | used to further analyze/debug coredumps. | |
93 | ||
94 | ## Alternative Coredump Handlers | |
95 | ||
96 | While we recommend usage of the `systemd-coredump` handler, it's fully | |
97 | supported to use alternative coredump handlers instead. A similar | |
98 | implementation pattern is recommended. Specifically: | |
99 | ||
100 | 1. Use a `sysctl.d/` drop-in to register your handler with the kernel. Make | |
101 | sure to include the `%c` specifier in the pattern (which reflects the | |
102 | crashing process' `RLIMIT_CORE`) and act on it: limit the stored coredump | |
103 | file to the specified limit. | |
104 | ||
105 | 2. Do not do heavy processing directly in the coredump handler. Instead, | |
106 | quickly pass off the kernel's coredump file descriptor to an | |
107 | auxiliary service running as service under the service manager, so that it | |
108 | can be done under supervision, sandboxing and resource management. | |
109 | ||
110 | Note that at any given time only a single handler can be enabled, i.e. the | |
111 | `kernel.core_pattern` sysctl cannot reference multiple executables. | |
112 | ||
113 | ## Packaging | |
114 | ||
115 | It might make sense to split `systemd-coredump` into a separate distribution | |
116 | package. If doing so, make sure that `/usr/lib/sysctl.d/50-coredump.conf` and | |
117 | the associated service and socket units are also added to the split off package. | |
118 | ||
119 | Note that in a scenario where `systemd-coredump` is split out and not | |
120 | installed, coredumping is turned off during the entire runtime of the system — | |
121 | unless an alternative handler is installed, or behaviour is manually reverted | |
122 | to legacy style handling (see below). | |
123 | ||
124 | ## Restoring Legacy Coredump Handling | |
125 | ||
126 | The default policy of the kernel to write coredumps into the current working | |
127 | directory of the crashing process is considered highly problematic by many, | |
128 | including by the systemd maintainers. Nonetheless, if users locally want to | |
129 | return to this behaviour, two changes must be made (followed by a reboot): | |
130 | ||
131 | ```console | |
132 | $ mkdir -p /etc/sysctl.d | |
133 | $ cat >/etc/sysctl.d/50-coredump.conf <<EOF | |
134 | # Party like it's 1995! | |
135 | kernel.core_pattern=core | |
136 | EOF | |
137 | ``` | |
138 | ||
139 | and | |
140 | ||
141 | ```console | |
142 | $ mkdir -p /etc/systemd/system.conf.d | |
143 | $ cat >/etc/systemd/system.conf.d/50-coredump.conf <<EOF | |
144 | [Manager] | |
6fb595bc | 145 | DefaultLimitCORE=0:infinity |
b6bd98eb LP |
146 | EOF |
147 | ``` |