]>
Commit | Line | Data |
---|---|---|
635dea27 | 1 | --- |
a0fadf66 | 2 | title: Container Interface |
635dea27 LP |
3 | category: Interfaces |
4 | layout: default | |
5 | --- | |
6 | ||
7 | # The Container Interface | |
8 | ||
9 | Also consult [Writing Virtual Machine or Container | |
10 | Managers](http://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers). | |
11 | ||
12 | systemd has a number of interfaces for interacting with container managers, | |
13 | when systemd is used inside of an OS container. If you work on a container | |
14 | manager, please consider supporting the following interfaces. | |
15 | ||
16 | ## Execution Environment | |
17 | ||
18 | 1. If the container manager wants to control the hostname for a container | |
d2e825b4 ZJS |
19 | running systemd it may just set it before invoking systemd, and systemd will |
20 | leave it unmodified when there is no hostname configured in `/etc/hostname` | |
21 | (that file overrides whatever is pre-initialized by the container manager). | |
22 | ||
23 | 2. Make sure to pre-mount `/proc/`, `/sys/`, and `/sys/fs/selinux/` before | |
24 | invoking systemd, and mount `/proc/sys/`, `/sys/`, and `/sys/fs/selinux/` | |
25 | read-only in order to prevent the container from altering the host kernel's | |
26 | configuration settings. (As a special exception, if your container has | |
27 | network namespaces enabled, feel free to make `/proc/sys/net/` writable). | |
28 | systemd and various other subsystems (such as the SELinux userspace) have | |
29 | been modified to behave accordingly when these file systems are read-only. | |
30 | (It's OK to mount `/sys/` as `tmpfs` btw, and only mount a subset of its | |
31 | sub-trees from the real `sysfs` to hide `/sys/firmware/`, `/sys/kernel/` and | |
32 | so on. If you do that, make sure to mark `/sys/` read-only, as that | |
33 | condition is what systemd looks for, and is what is considered to be the API | |
34 | in this context.) | |
635dea27 LP |
35 | |
36 | 3. Pre-mount `/dev/` as (container private) `tmpfs` for the container and bind | |
37 | mount some suitable TTY to `/dev/console`. Also, make sure to create device | |
38 | nodes for `/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, | |
39 | `/dev/urandom`, `/dev/tty`, `/dev/ptmx` in `/dev/`. It is not necessary to | |
40 | create `/dev/fd` or `/dev/stdout`, as systemd will do that on its own. Make | |
41 | sure to set up a `BPF_PROG_TYPE_CGROUP_DEVICE` BPF program — on cgroupv2 — | |
42 | or the `devices` cgroup controller — on cgroupv1 — so that no other devices | |
43 | but these may be created in the container. Note that many systemd services | |
d2e825b4 ZJS |
44 | use `PrivateDevices=`, which means that systemd will set up a private |
45 | `/dev/` for them for which it needs to be able to create these device nodes. | |
46 | Dropping `CAP_MKNOD` for containers is hence generally not advisable, but | |
47 | see below. | |
48 | ||
49 | 4. `systemd-udevd` is not available in containers (and refuses to start), and | |
50 | hence device dependencies are unavailable. The `systemd-udevd` unit files | |
51 | will check for `/sys/` being read-only, as an indication whether device | |
52 | management can work. Therefore make sure to mount `/sys/` read-only in the | |
53 | container (see above). Various clients of `systemd-udevd` also check the | |
54 | read-only state of `/sys/`, including PID 1 itself and `systemd-networkd`. | |
635dea27 LP |
55 | |
56 | 5. If systemd detects it is run in a container it will spawn a single shell on | |
57 | `/dev/console`, and not care about VTs or multiple gettys on VTs. (But see | |
58 | `$container_ttys` below.) | |
59 | ||
60 | 6. Either pre-mount all cgroup hierarchies in full into the container, or leave | |
61 | that to systemd which will do so if they are missing. Note that it is | |
62 | explicitly *not* OK to just mount a sub-hierarchy into the container as that | |
63 | is incompatible with `/proc/$PID/cgroup` (which lists full paths). Also the | |
64 | root-level cgroup directories tend to be quite different from inner | |
65 | directories, and that distinction matters. It is OK however, to mount the | |
66 | "upper" parts read-only of the hierarchies, and only allow write-access to | |
67 | the cgroup sub-tree the container runs in. It's also a good idea to mount | |
68 | all controller hierarchies with exception of `name=systemd` fully read-only | |
69 | (this only applies to cgroupv1, of course), to protect the controllers from | |
70 | alteration from inside the containers. Or to turn this around: only the | |
71 | cgroup sub-tree of the container itself (on cgroupv2 in the unified | |
d2e825b4 | 72 | hierarchy, and on cgroupv1 in the `name=systemd` hierarchy) may be writable |
635dea27 LP |
73 | to the container. |
74 | ||
75 | 7. Create the control group root of your container by either running your | |
76 | container as a service (in case you have one container manager instance per | |
77 | container instance) or creating one scope unit for each container instance | |
78 | via systemd's transient unit API (in case you have one container manager | |
79 | that manages all instances. Either way, make sure to set `Delegate=yes` in | |
80 | it. This ensures that that the unit you created will be part of all cgroup | |
81 | controllers (or at least the ones systemd understands). The latter may also | |
82 | be done via `systemd-machined`'s `CreateMachine()` API. Make sure to use the | |
d2e825b4 ZJS |
83 | cgroup path systemd put your process in for all operations of the container. |
84 | Do not add new cgroup directories to the top of the tree. This will not only | |
85 | confuse systemd and the admin, but also prevent your implementation from | |
86 | being "stackable". | |
635dea27 LP |
87 | |
88 | ## Environment Variables | |
89 | ||
d2e825b4 ZJS |
90 | 1. To allow systemd (and other programs) to identify that it is executed within |
91 | a container, please set the `$container` environment variable for PID 1 in | |
635dea27 LP |
92 | the container to a short lowercase string identifying your |
93 | implementation. With this in place the `ConditionVirtualization=` setting in | |
94 | unit files will work properly. Example: `container=lxc-libvirt` | |
95 | ||
96 | 2. systemd has special support for allowing container managers to initialize | |
97 | the UUID for `/etc/machine-id` to some manager supplied value. This is only | |
98 | enabled if `/etc/machine-id` is empty (i.e. not yet set) at boot time of the | |
d2e825b4 ZJS |
99 | container. The container manager should set `$container_uuid` as environment |
100 | variable for the container's PID 1 to the container UUID. (This is similar | |
101 | to the effect of `qemu`'s `-uuid` switch). Note that you should pass only a | |
102 | UUID here that is actually unique (i.e. only one running container should | |
103 | have a specific UUID), and gets changed when a container gets duplicated. | |
104 | Also note that systemd will try to persistently store the UUID in | |
105 | `/etc/machine-id` (if writable) when this option is used, hence you should | |
106 | always pass the same UUID here. Keeping the externally used UUID for a | |
107 | container and the internal one in sync is hopefully useful to minimize | |
108 | surprise for the administrator. | |
635dea27 LP |
109 | |
110 | 3. systemd can automatically spawn login gettys on additional ptys. A container | |
d2e825b4 | 111 | manager can set the `$container_ttys` environment variable for the |
635dea27 LP |
112 | container's PID 1 to tell it on which ptys to spawn gettys. The variable |
113 | should take a space separated list of pty names, without the leading `/dev/` | |
114 | prefix, but with the `pts/` prefix included. Note that despite the | |
115 | variable's name you may only specify ptys, and not other types of ttys. Also | |
116 | you need to specify the pty itself, a symlink will not suffice. This is | |
117 | implemented in | |
d2e825b4 ZJS |
118 | [systemd-getty-generator(8)](https://www.freedesktop.org/software/systemd/man/systemd-getty-generator.html). |
119 | Note that this variable should not include the pty that `/dev/console` maps | |
120 | to if it maps to one (see below). Example: if the container receives | |
121 | `container_ttys=pts/7 pts/8 pts/14` it will spawn three additional login | |
122 | gettys on ptys 7, 8, and 14. | |
635dea27 LP |
123 | |
124 | ## Advanced Integration | |
125 | ||
126 | 1. Consider syncing `/etc/localtime` from the host file system into the | |
127 | container. Make it a relative symlink to the containers's zoneinfo dir, as | |
128 | usual. Tools rely on being able to determine the timezone setting from the | |
d2e825b4 ZJS |
129 | symlink value, and making it relative looks nice even if people list the |
130 | container's `/etc/` from the host. | |
635dea27 LP |
131 | |
132 | 2. Make the container journal available in the host, by automatically | |
d2e825b4 ZJS |
133 | symlinking the container journal directory into the host journal directory. |
134 | More precisely, link `/var/log/journal/<container-machine-id>` of the | |
135 | container into the same dir of the host. Administrators can then | |
635dea27 | 136 | automatically browse all container journals (correctly interleaved) by |
d2e825b4 | 137 | issuing `journalctl -m`. The container machine ID can be determined from |
635dea27 LP |
138 | `/etc/machine-id` in the container. |
139 | ||
140 | 3. If the container manager wants to cleanly shutdown the container, it might | |
d2e825b4 ZJS |
141 | be a good idea to send `SIGRTMIN+3` to its init process. systemd will then |
142 | do a clean shutdown. Note however, that since only systemd understands | |
143 | `SIGRTMIN+3` like this, this might confuse other init systems. | |
635dea27 LP |
144 | |
145 | 4. To support [Socket Activated | |
146 | Containers](http://0pointer.de/blog/projects/socket-activated-containers.html) | |
147 | the container manager should be capable of being run as a systemd | |
148 | service. It will then receive the sockets starting with FD 3, the number of | |
149 | passed FDs in `$LISTEN_FDS` and its PID as `$LISTEN_PID`. It should take | |
150 | these and pass them on to the container's init process, also setting | |
151 | $LISTEN_FDS and `$LISTEN_PID` (basically, it can just leave the FDs and | |
d2e825b4 | 152 | `$LISTEN_FDS` untouched, but it needs to adjust `$LISTEN_PID` to the |
635dea27 LP |
153 | container init process). That's all that's necessary to make socket |
154 | activation work. The protocol to hand sockets from systemd to services is | |
d2e825b4 | 155 | hence the same as from the container manager to the container systemd. For |
635dea27 LP |
156 | further details see the explanations of |
157 | [sd_listen_fds(1)](http://0pointer.de/public/systemd-man/sd_listen_fds.html) | |
158 | and the [blog story for service | |
159 | developers](http://0pointer.de/blog/projects/socket-activation.html). | |
160 | ||
d2e825b4 ZJS |
161 | 5. Container managers should stay away from the cgroup hierarchy outside of the |
162 | unit they created for their container. That's private property of systemd, | |
163 | and no other code should modify it. | |
635dea27 LP |
164 | |
165 | ## Networking | |
166 | ||
167 | 1. Inside of a container, if a `veth` link is named `host0`, `systemd-networkd` | |
d2e825b4 ZJS |
168 | running inside of the container will by default run DHCPv4, DHCPv6, and |
169 | IPv4LL clients on it. It is thus recommended that container managers that | |
170 | add a `veth` link to a container name it `host0`, to get an automatically | |
171 | configured network, with no manual setup. | |
635dea27 | 172 | |
d2e825b4 ZJS |
173 | 2. Outside of a container, if a `veth` link is prefixed "ve-", `systemd-networkd` |
174 | will by default run DHCPv4 and DHCPv6 servers on it, as well as IPv4LL. It | |
175 | is thus recommended that container managers that add a `veth` link to a | |
176 | container name the external side `ve-` + the container name. | |
635dea27 | 177 | |
d2e825b4 | 178 | 3. It is recommended to configure stable MAC addresses for container `veth` |
635dea27 LP |
179 | devices, for example hashed out of the container names. That way it is more |
180 | likely that DHCP and IPv4LL will acquire stable addresses. | |
181 | ||
182 | ## What You Shouldn't Do | |
183 | ||
184 | 1. Do not drop `CAP_MKNOD` from the container. `PrivateDevices=` is a commonly | |
185 | used service setting that provides a service with its own, private, minimal | |
186 | version of `/dev/`. To set this up systemd in the container needs this | |
187 | capability. If you take away the capability than all services that set this | |
d2e825b4 ZJS |
188 | flag will cease to work. Use `BPF_PROG_TYPE_CGROUP_DEVICE` BPF programs — on |
189 | cgroupv2 — or the `devices` controller — on cgroupv1 — to restrict what | |
190 | device nodes the container can create instead of taking away the capability | |
191 | wholesale. (Also see the section about fully unprivileged containers below.) | |
192 | ||
193 | 2. Do not drop `CAP_SYS_ADMIN` from the container. A number of the most | |
194 | commonly used file system namespacing related settings, such as | |
195 | `PrivateDevices=`, `ProtectHome=`, `ProtectSystem=`, `MountFlags=`, | |
196 | `PrivateTmp=`, `ReadWriteDirectories=`, `ReadOnlyDirectories=`, | |
197 | `InaccessibleDirectories=`, and `MountFlags=` need to be able to open new | |
198 | mount namespaces and the mount certain file systems into them. You break all | |
199 | services that make use of these options if you drop the capability. Also | |
635dea27 | 200 | note that logind mounts `XDG_RUNTIME_DIR` as `tmpfs` for all logged in users |
d2e825b4 ZJS |
201 | and that won't work either if you take away the capability. (Also see |
202 | section about fully unprivileged containers below.) | |
635dea27 LP |
203 | |
204 | 3. Do not cross-link `/dev/kmsg` with `/dev/console`. They are different things, | |
205 | you cannot link them to each other. | |
206 | ||
d2e825b4 ZJS |
207 | 4. Do not pretend that the real VTs are available in the container. The VT |
208 | subsystem consists of all the devices `/dev/tty*`, `/dev/vcs*`, `/dev/vcsa*` | |
635dea27 | 209 | plus their `sysfs` counterparts. They speak specific `ioctl()`s and |
d2e825b4 ZJS |
210 | understand specific escape sequences, that other ptys don't understand. |
211 | Hence, it is explicitly not OK to mount a pty to `/dev/tty1`, `/dev/tty2`, | |
212 | `/dev/tty3`. This is explicitly not supported. | |
635dea27 LP |
213 | |
214 | 5. Don't pretend that passing arbitrary devices to containers could really work | |
d2e825b4 | 215 | well. For example, do not pass device nodes for block devices to the |
635dea27 LP |
216 | container. Device access (with the exception of network devices) is not |
217 | virtualized on Linux. Enumeration and probing of meta information from | |
218 | `/sys/` and elsewhere is not possible to do correctly in a container. Simply | |
219 | adding a specific device node to a container's `/dev/` is *not* *enough* to | |
220 | do the job, as `systemd-udevd` and suchlike are not available at all, and no | |
221 | devices will appear available or enumerable, inside the container. | |
222 | ||
223 | 6. Don't mount only a sub-tree of the `cgroupfs` into the container. This will not | |
224 | work as `/proc/$PID/cgroup` lists full paths and cannot be matched up with | |
225 | the actual `cgroupfs` tree visible, then. (You may "prune" some branches | |
226 | though, see above.) | |
227 | ||
228 | 7. Do not make `/sys/` writable in the container. If you do, | |
229 | `systemd-udevd.service` is started to manage your devices — inside the | |
d2e825b4 | 230 | container, but that will cause conflicts and errors given that the Linux |
635dea27 LP |
231 | device model is not virtualized for containers on Linux and thus the |
232 | containers and the host would try to manage the same devices, fighting for | |
233 | ownership. Multiple other subsystems of systemd similarly test for `/sys/` | |
234 | being writable to decide whether to use `systemd-udevd` or assume that | |
235 | device management is properly available on the instance. Among them | |
236 | `systemd-networkd` and `systemd-logind`. The conditionalization on the | |
237 | read-only state of `/sys/` enables a nice automatism: as soon as `/sys/` and | |
238 | the Linux device model are changed to be virtualized properly the container | |
239 | payload can make use of that, simply by marking `/sys/` writable. (Note that | |
240 | as special exception, the devices in `/sys/class/net/` are virtualized | |
241 | already, if network namespacing is used. Thus it is OK to mount the relevant | |
242 | sub-directories of `/sys/` writable, but make sure to leave the root of | |
243 | `/sys/` read-only.) | |
244 | ||
245 | ## Fully Unprivileged Container Payload | |
246 | ||
247 | First things first, to make this clear: Linux containers are not a security | |
d2e825b4 | 248 | technology right now. There are more holes in the model than in swiss cheese. |
635dea27 | 249 | |
d2e825b4 | 250 | For example: if you do not use user namespacing, and share root and other users |
635dea27 | 251 | between container and host, the `struct user` structures will be shared between |
d2e825b4 ZJS |
252 | host and container, and hence `RLIMIT_NPROC` and so of the container users |
253 | affect the host and other containers, and vice versa. This is a major security | |
254 | hole, and actually is a real-life problem: since Avahi sets `RLIMIT_NPROC` of | |
255 | its user to 2 (to effectively disallow `fork()`ing) you cannot run more than | |
256 | one Avahi instance on the entire system... | |
635dea27 LP |
257 | |
258 | People have been asking to be able to run systemd without `CAP_SYS_ADMIN` and | |
259 | `CAP_SYS_MKNOD` in the container. This is now supported to some level in | |
260 | systemd, but we recommend against it (see above). If `CAP_SYS_ADMIN` and | |
261 | `CAP_SYS_MKNOD` are missing from the container systemd will now gracefully turn | |
262 | off `PrivateTmp=`, `PrivateNetwork=`, `ProtectHome=`, `ProtectSystem=` and | |
263 | others, because those capabilities are required to implement these options. The | |
264 | services using these settings (which include many of systemd's own) will hence | |
265 | run in a different, less secure environment when the capabilities are missing | |
266 | than with them around. | |
267 | ||
268 | With user namespacing in place things get much better. With user namespaces the | |
269 | `struct user` issue described above goes away, and containers can keep | |
270 | `CAP_SYS_ADMIN` safely for the user namespace, as capabilities are virtualized | |
271 | and having capabilities inside a container doesn't mean one also has them | |
272 | outside. | |
273 | ||
274 | ## Final Words | |
275 | ||
276 | If you write software that wants to detect whether it is run in a container, | |
277 | please check `/proc/1/environ` and look for the `container=` environment | |
278 | variable. Do not assume the environment variable is inherited down the process | |
279 | tree. It generally is not. Hence check the environment block of PID 1, not your | |
280 | own. Note though that that file is only accessible to root. systemd hence early | |
281 | on also copies the value into `/run/systemd/container`, which is readable for | |
282 | everybody. However, that's a systemd-specific interface and other init systems | |
283 | are unlikely to do the same. | |
284 | ||
285 | Note that it is our intention to make systemd systems work flawlessly and | |
286 | out-of-the-box in containers. In fact we are interested to ensure that the same | |
287 | OS image can be booted on a bare system, in a VM and in a container, and behave | |
288 | correctly each time. If you notice that some component in systemd does not work | |
289 | in a container as it should, even though the container manager implements | |
290 | everything documented above, please contact us. |