]>
Commit | Line | Data |
---|---|---|
6b2a2776 | 1 | --- |
2 | title: Diagnosing Boot Problems | |
3 | category: Manuals and Documentation for Users and Administrators | |
4 | layout: default | |
5 | SPDX-License-Identifier: LGPL-2.1-or-later | |
6 | --- | |
7 | ||
8 | # Diagnosing Boot Problems | |
9 | ||
10 | If your machine gets stuck during boot, first check if the hang happens before or after control passes to systemd. | |
11 | ||
837cc38a | 12 | Try to boot without `rhgb` and `quiet` on the kernel command line. |
13 | If you see some messages like these: | |
6b2a2776 | 14 | |
15 | * Welcome to Fedora _VERSION_ (_codename_)!" | |
16 | * Starting _name_... | |
17 | * \[ OK \] Started _name_. | |
18 | ||
837cc38a | 19 | then systemd is running. |
20 | (See an actual [screenshot](../assets/f17boot.png).) | |
6b2a2776 | 21 | |
837cc38a | 22 | Debugging always gets easier if you can get a shell. |
23 | If you do not get a login prompt, try switching to a different virtual terminal using CTRL+ALT+F\_\_. | |
24 | Problems with a display server startup may manifest themselves as a missing login on tty1, but other VTs working. | |
6b2a2776 | 25 | |
837cc38a | 26 | If the boot stops without presenting you with a login on any virtual console, |
27 | let it retry for _up to 5 minutes_ before declaring it definitely stuck. | |
28 | There is a chance that a service that has trouble starting will be killed after this timeout and the boot will continue normally. | |
29 | Another possibility is that a device for an important mountpoint will fail to appear and you will be presented with _emergency mode_. | |
6b2a2776 | 30 | |
31 | ## If You Get No Shell | |
32 | ||
837cc38a | 33 | If you get neither a normal login nor the emergency mode shell, |
34 | you will need to do additional steps to get debugging information out of the machine. | |
6b2a2776 | 35 | |
36 | * Try CTRL+ALT+DEL to reboot. | |
837cc38a | 37 | * If it does not reboot, mention it in your bugreport. Meanwhile force the reboot with |
38 | [SysRq](http://fedoraproject.org/wiki/QA/Sysrq) | |
39 | or hard reset. | |
6b2a2776 | 40 | * When booting the next time, you will have to add some kernel command line arguments depending on which of the debugging strategies you choose from the following options. |
41 | ||
42 | ### Debug Logging to a Serial Console | |
43 | ||
837cc38a | 44 | If you have a hardware serial console available or if you are debugging in a virtual machine |
45 | (e.g. using virt-manager you can switch your view to a serial console in the menu View -> Text Consoles or connect from the terminal using `virsh console MACHINE`), | |
46 | you can ask systemd to log lots of useful debugging information to it by booting with: | |
6b2a2776 | 47 | |
48 | ```sh | |
49 | systemd.log_level=debug systemd.log_target=console console=ttyS0,38400 console=tty1 | |
50 | ``` | |
51 | ||
837cc38a | 52 | The above is useful if pid 1 is failing, but if a later but critical boot service is broken (such as networking), |
53 | you can configure journald to forward to the console by using: | |
6b2a2776 | 54 | |
55 | ```sh | |
56 | systemd.journald.forward_to_console=1 console=ttyS0,38400 console=tty1 | |
57 | ``` | |
58 | ||
59 | console= can be specified multiple times, systemd will output to all of them. | |
60 | ||
61 | ### Booting into Rescue or Emergency Targets | |
62 | ||
837cc38a | 63 | To boot directly into rescue target add `systemd.unit=rescue.target` or just `1` to the kernel command line. |
64 | This target is useful if the problem occurs somewhere after the basic system is brought up, during the starting of "normal" services. | |
65 | If this is the case, you should be able to disable the bad service from here. | |
66 | If the rescue target will not boot either, the more minimal emergency target might. | |
6b2a2776 | 67 | |
837cc38a | 68 | To boot directly into emergency shell add `systemd.unit=emergency.target` or `emergency` to the kernel command line. |
69 | Note that in the emergency shell you will have to remount the root filesystem read-write by yourself before editing any files: | |
6b2a2776 | 70 | |
71 | ```sh | |
72 | mount -o remount,rw / | |
73 | ``` | |
74 | ||
837cc38a | 75 | Common issues that can be resolved in the emergency shell are bad lines in `/etc/fstab`. |
76 | After fixing **/etc/fstab**, run `systemctl daemon-reload` to let systemd refresh its view of it. | |
6b2a2776 | 77 | |
837cc38a | 78 | If not even the emergency target works, you can boot directly into a shell with `init=/bin/sh`. |
79 | This may be necessary in case systemd itself or some libraries it depends on are damaged by filesystem corruption. | |
80 | You may need to reinstall working versions of the affected packages. | |
6b2a2776 | 81 | |
82 | If `init=/bin/sh` does not work, you must boot from another medium. | |
83 | ||
84 | ### Early Debug Shell | |
85 | ||
837cc38a | 86 | You can enable shell access to be available very early in the startup process to fall back on and diagnose systemd related boot up issues with various systemctl commands. |
87 | Enable it using: | |
6b2a2776 | 88 | |
89 | ```sh | |
90 | systemctl enable debug-shell.service | |
91 | ``` | |
92 | ||
93 | or by specifying | |
94 | ||
95 | ```sh | |
92ecff60 | 96 | systemd.debug_shell=1 |
6b2a2776 | 97 | ``` |
98 | ||
99 | on the kernel command line. | |
100 | ||
837cc38a | 101 | **Tip**: If you find yourself in a situation where you cannot use systemctl to communicate with a running systemd |
102 | (e.g. when setting this up from a different booted system), | |
103 | you can avoid communication with the manager by specifying `--root=`: | |
6b2a2776 | 104 | |
105 | ```sh | |
106 | systemctl --root=/ enable debug-shell.service | |
107 | ``` | |
108 | ||
837cc38a | 109 | Once enabled, the next time you boot you will be able to switch to tty9 using CTRL+ALT+F9 and have a root shell there available from an early point in the booting process. |
110 | You can use the shell for checking the status of services, reading logs, looking for stuck jobs with `systemctl list-jobs`, etc. | |
6b2a2776 | 111 | |
837cc38a | 112 | **Warning:** Use this shell only for debugging! |
113 | Do not forget to disable systemd-debug-shell.service after you've finished debugging your boot problems. | |
114 | Leaving the root shell always available would be a security risk. | |
6b2a2776 | 115 | |
837cc38a | 116 | It is also possible to alias `kbrequest.target` to `debug-shell.service` to start the debug shell on demand. |
117 | This has the same security implications, but avoids running the shell always. | |
6b2a2776 | 118 | |
119 | ### verify prerequisites | |
120 | ||
837cc38a | 121 | A (at least partly) populated `/dev` is required. |
122 | Depending on your setup (e.g. on embedded systems), | |
123 | check that the Linux kernel config options `CONFIG_DEVTMPFS` and `CONFIG_DEVTMPFS_MOUNT` are set. | |
124 | Also support for cgroups and fanotify is recommended for a flawless operation, so check that the Linux kernel config options `CONFIG_CGROUPS` and `CONFIG_FANOTIFY` are set. | |
125 | The message "Failed to get D-Bus connection: No connection to service manager." | |
126 | during various `systemctl` operations is an indicator that these are missing. | |
6b2a2776 | 127 | |
128 | ## If You Can Get a Shell | |
129 | ||
837cc38a | 130 | When you have systemd running to the extent that it can provide you with a shell, |
131 | please use it to extract useful information for debugging. | |
132 | Boot with these parameters on the kernel command line: | |
6b2a2776 | 133 | |
134 | ```sh | |
135 | systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on | |
136 | ``` | |
137 | ||
837cc38a | 138 | in order to increase the verbosity of systemd, to let systemd write its logs to the kernel log buffer, |
139 | to increase the size of the kernel log buffer, and to prevent the kernel from discarding messages. | |
140 | After reaching the shell, look at the log: | |
6b2a2776 | 141 | |
142 | ```sh | |
143 | journalctl -b | |
144 | ``` | |
145 | ||
146 | When reporting a bug, pipe that to a file and attach it to the bug report. | |
147 | ||
148 | To check for possibly stuck jobs use: | |
149 | ||
150 | ```sh | |
151 | systemctl list-jobs | |
152 | ``` | |
153 | ||
154 | The jobs that are listed as "running" are the ones that must complete before the "waiting" ones will be allowed to start executing. | |
155 | ||
6b2a2776 | 156 | # Diagnosing Shutdown Problems |
157 | ||
837cc38a | 158 | Just like with boot problems, when you encounter a hang during shutting down, |
159 | make sure you wait _at least 5 minutes_ to distinguish a permanent hang from a broken service that's just timing out. | |
160 | Then it's worth testing whether the system reacts to CTRL+ALT+DEL in any way. | |
6b2a2776 | 161 | |
837cc38a | 162 | If shutdown (whether it be to reboot or power-off) of your system gets stuck, |
163 | first test if the kernel itself is able to reboot or power-off the machine forcedly using one of these commands: | |
6b2a2776 | 164 | |
165 | ```sh | |
166 | reboot -f | |
167 | poweroff -f | |
168 | ``` | |
169 | ||
170 | If either one of the commands does not work, it's more likely to be a kernel, not systemd bug. | |
171 | ||
172 | ## Shutdown Completes Eventually | |
173 | ||
174 | If normal reboot or poweroff work, but take a suspiciously long time, then | |
175 | ||
176 | * boot with the debug options: | |
177 | ||
178 | ```sh | |
179 | systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on enforcing=0 | |
180 | ``` | |
181 | ||
837cc38a | 182 | * save the following script as `/usr/lib/systemd/system-shutdown/debug.sh` and make it executable: |
6b2a2776 | 183 | |
184 | ```sh | |
185 | #!/bin/sh | |
186 | mount -o remount,rw / | |
187 | dmesg > /shutdown-log.txt | |
188 | mount -o remount,ro / | |
189 | ``` | |
190 | ||
191 | * reboot | |
192 | ||
837cc38a | 193 | Look for timeouts logged in the resulting file `shutdown-log.txt` and/or attach it to a bugreport. |
6b2a2776 | 194 | |
195 | ## Shutdown Never Finishes | |
196 | ||
837cc38a | 197 | If normal reboot or poweroff never finish even after waiting a few minutes, |
198 | the above method to create the shutdown log will not help and the log must be obtained using other methods. | |
199 | Two options that are useful for debugging boot problems can be used also for shutdown problems: | |
6b2a2776 | 200 | |
201 | * use a serial console | |
202 | * use a debug shell - not only is it available from early boot, it also stays active until late shutdown. | |
203 | ||
6b2a2776 | 204 | # Status and Logs of Services |
205 | ||
206 | When the start of a service fails, systemctl will give you a generic error message: | |
207 | ||
208 | ```sh | |
209 | # systemctl start foo.service | |
210 | Job failed. See system journal and 'systemctl status' for details. | |
211 | ``` | |
212 | ||
837cc38a | 213 | The service may have printed its own error message, but you do not see it, |
214 | because services run by systemd are not related to your login session and their outputs are not connected to your terminal. | |
215 | That does not mean the output is lost though. | |
216 | By default the stdout, | |
217 | stderr of services are directed to the systemd _journal_ and the logs that services produce via `syslog(3)` go there too. | |
218 | systemd also stores the exit code of failed services. | |
219 | Let's check: | |
6b2a2776 | 220 | |
221 | ```sh | |
222 | # systemctl status foo.service | |
223 | foo.service - mmm service | |
837cc38a | 224 | Loaded: loaded (/etc/systemd/system/foo.service; static) |
225 | Active: failed (Result: exit-code) since Fri, 11 May 2012 20:26:23 +0200; 4s ago | |
226 | Process: 1329 ExecStart=/usr/local/bin/foo (code=exited, status=1/FAILURE) | |
227 | CGroup: name=systemd:/system/foo.service | |
6b2a2776 | 228 | |
229 | May 11 20:26:23 scratch foo[1329]: Failed to parse config | |
230 | ``` | |
231 | ||
837cc38a | 232 | In this example the service ran as a process with PID 1329 and exited with error code 1. |
233 | If you run systemctl status as root or as a user from the `adm` group, | |
234 | you will get a few lines from the journal that the service wrote. | |
235 | In the example the service produced just one error message. | |
6b2a2776 | 236 | |
237 | To list the journal, use the `journalctl` command. | |
238 | ||
837cc38a | 239 | If you have a syslog service (such as rsyslog) running, the journal will also forward the messages to it, |
240 | so you'll find them in `/var/log/messages` (depending on rsyslog's configuration). | |
6b2a2776 | 241 | |
242 | # Reporting systemd Bugs | |
243 | ||
837cc38a | 244 | Be prepared to include some information (logs) about your system as well. |
245 | These should be complete (no snippets please), not in an archive, uncompressed. | |
6b2a2776 | 246 | |
837cc38a | 247 | Please report bugs to your distribution's bug tracker first. |
248 | If you are sure that you are encountering an upstream bug, then first check | |
249 | [for existing bug reports](https://github.com/systemd/systemd/issues/), | |
250 | and if your issue is not listed | |
251 | [file a new bug](https://github.com/systemd/systemd/issues/new). | |
6b2a2776 | 252 | |
253 | ## Information to Attach to a Bug Report | |
254 | ||
255 | Whenever possible, the following should be mentioned and attached to your bug report: | |
256 | ||
837cc38a | 257 | * The exact kernel command-line used. |
258 | Typically from the bootloader configuration file (e.g. `/boot/grub2/grub.cfg`) or from `/proc/cmdline` | |
6b2a2776 | 259 | * The journal (the output of `journalctl -b > journal.txt`) |
260 | * ideally after booting with `systemd.log_level=debug systemd.log_target=kmsg log_buf_len=1M printk.devkmsg=on` | |
261 | * The output of a systemd dump: `systemd-analyze dump > systemd-dump.txt` | |
262 | * The output of `/usr/lib/systemd/systemd --test --system --log-level=debug > systemd-test.txt 2>&1` |