2 title: Container Interface
5 SPDX-License-Identifier: LGPL-2.1-or-later
8 # The Container Interface
10 Also consult [Writing Virtual Machine or Container
11 Managers](https://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers).
13 systemd has a number of interfaces for interacting with container managers,
14 when systemd is used inside of an OS container. If you work on a container
15 manager, please consider supporting the following interfaces.
17 ## Execution Environment
19 1. If the container manager wants to control the hostname for a container
20 running systemd it may just set it before invoking systemd, and systemd will
21 leave it unmodified when there is no hostname configured in `/etc/hostname`
22 (that file overrides whatever is pre-initialized by the container manager).
24 2. Make sure to pre-mount `/proc/`, `/sys/`, and `/sys/fs/selinux/` before
25 invoking systemd, and mount `/sys/`, `/sys/fs/selinux/` and `/proc/sys/`
26 read-only (the latter via e.g. a read-only bind mount on itself) in order
27 to prevent the container from altering the host kernel's configuration
28 settings. (As a special exception, if your container has network namespaces
29 enabled, feel free to make `/proc/sys/net/` writable. If it also has user, ipc,
30 uts and pid namespaces enabled, the entire `/proc/sys` can be left writable).
31 systemd and various other subsystems (such as the SELinux userspace) have
32 been modified to behave accordingly when these file systems are read-only.
33 (It's OK to mount `/sys/` as `tmpfs` btw, and only mount a subset of its
34 sub-trees from the real `sysfs` to hide `/sys/firmware/`, `/sys/kernel/` and
35 so on. If you do that, make sure to mark `/sys/` read-only, as that
36 condition is what systemd looks for, and is what is considered to be the API
39 3. Pre-mount `/dev/` as (container private) `tmpfs` for the container and bind
40 mount some suitable TTY to `/dev/console`. If this is a pty, make sure to
41 not close the controlling pty during systemd's lifetime. PID 1 will close
42 ttys, to avoid being killed by SAK. It only opens ttys for the time it
43 actually needs to print something. Also, make sure to create device nodes
44 for `/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom`,
45 `/dev/tty`, `/dev/ptmx` in `/dev/`. It is not necessary to create `/dev/fd`
46 or `/dev/stdout`, as systemd will do that on its own. Make sure to set up a
47 `BPF_PROG_TYPE_CGROUP_DEVICE` BPF program — on cgroupv2 — or the `devices`
48 cgroup controller — on cgroupv1 — so that no other devices but these may be
49 created in the container. Note that many systemd services use
50 `PrivateDevices=`, which means that systemd will set up a private `/dev/`
51 for them for which it needs to be able to create these device nodes.
52 Dropping `CAP_MKNOD` for containers is hence generally not advisable, but
55 4. `systemd-udevd` is not available in containers (and refuses to start), and
56 hence device dependencies are unavailable. The `systemd-udevd` unit files
57 will check for `/sys/` being read-only, as an indication whether device
58 management can work. Therefore make sure to mount `/sys/` read-only in the
59 container (see above). Various clients of `systemd-udevd` also check the
60 read-only state of `/sys/`, including PID 1 itself and `systemd-networkd`.
62 5. If systemd detects it is run in a container it will spawn a single shell on
63 `/dev/console`, and not care about VTs or multiple gettys on VTs. (But see
64 `$container_ttys` below.)
66 6. Either pre-mount all cgroup hierarchies in full into the container, or leave
67 that to systemd which will do so if they are missing. Note that it is
68 explicitly *not* OK to just mount a sub-hierarchy into the container as that
69 is incompatible with `/proc/$PID/cgroup` (which lists full paths). Also the
70 root-level cgroup directories tend to be quite different from inner
71 directories, and that distinction matters. It is OK however, to mount the
72 "upper" parts read-only of the hierarchies, and only allow write-access to
73 the cgroup sub-tree the container runs in. It's also a good idea to mount
74 all controller hierarchies with exception of `name=systemd` fully read-only
75 (this only applies to cgroupv1, of course), to protect the controllers from
76 alteration from inside the containers. Or to turn this around: only the
77 cgroup sub-tree of the container itself (on cgroupv2 in the unified
78 hierarchy, and on cgroupv1 in the `name=systemd` hierarchy) may be writable
81 7. Create the control group root of your container by either running your
82 container as a service (in case you have one container manager instance per
83 container instance) or creating one scope unit for each container instance
84 via systemd's transient unit API (in case you have one container manager
85 that manages all instances. Either way, make sure to set `Delegate=yes` in
86 it. This ensures that the unit you created will be part of all cgroup
87 controllers (or at least the ones systemd understands). The latter may also
88 be done via `systemd-machined`'s `CreateMachine()` API. Make sure to use the
89 cgroup path systemd put your process in for all operations of the container.
90 Do not add new cgroup directories to the top of the tree. This will not only
91 confuse systemd and the admin, but also prevent your implementation from
94 ## Environment Variables
96 1. To allow systemd (and other programs) to identify that it is executed within
97 a container, please set the `$container` environment variable for PID 1 in
98 the container to a short lowercase string identifying your
99 implementation. With this in place the `ConditionVirtualization=` setting in
100 unit files will work properly. Example: `container=lxc-libvirt`
102 2. systemd has special support for allowing container managers to initialize
103 the UUID for `/etc/machine-id` to some manager supplied value. This is only
104 enabled if `/etc/machine-id` is empty (i.e. not yet set) at boot time of the
105 container. The container manager should set `$container_uuid` as environment
106 variable for the container's PID 1 to the container UUID. (This is similar
107 to the effect of `qemu`'s `-uuid` switch). Note that you should pass only a
108 UUID here that is actually unique (i.e. only one running container should
109 have a specific UUID), and gets changed when a container gets duplicated.
110 Also note that systemd will try to persistently store the UUID in
111 `/etc/machine-id` (if writable) when this option is used, hence you should
112 always pass the same UUID here. Keeping the externally used UUID for a
113 container and the internal one in sync is hopefully useful to minimize
114 surprise for the administrator.
116 3. systemd can automatically spawn login gettys on additional ptys. A container
117 manager can set the `$container_ttys` environment variable for the
118 container's PID 1 to tell it on which ptys to spawn gettys. The variable
119 should take a space separated list of pty names, without the leading `/dev/`
120 prefix, but with the `pts/` prefix included. Note that despite the
121 variable's name you may only specify ptys, and not other types of ttys. Also
122 you need to specify the pty itself, a symlink will not suffice. This is
124 [systemd-getty-generator(8)](https://www.freedesktop.org/software/systemd/man/systemd-getty-generator.html).
125 Note that this variable should not include the pty that `/dev/console` maps
126 to if it maps to one (see below). Example: if the container receives
127 `container_ttys=pts/7 pts/8 pts/14` it will spawn three additional login
128 gettys on ptys 7, 8, and 14.
130 4. To allow applications to detect the OS version and other metadata of the host
131 running the container manager, if this is considered desirable, please parse
132 the host's `/etc/os-release` and set a `$container_host_<key>=<VALUE>`
133 environment variable for the ID fields described by the [os-release
134 interface](https://www.freedesktop.org/software/systemd/man/os-release.html), eg:
135 `$container_host_id=debian`
136 `$container_host_build_id=2020-06-15`
137 `$container_host_variant_id=server`
138 `$container_host_version_id=10`
140 5. systemd supports passing immutable binary data blobs with limited size and
141 restricted access to services via the `ImportCredential=`, `LoadCredential=`
142 and `SetCredential=` settings. The same protocol may be used to pass credentials
143 from the container manager to systemd itself. The credential data should be
144 placed in some location (ideally a read-only and non-swappable file system,
145 like 'ramfs'), and the absolute path to this directory exported in the
146 `$CREDENTIALS_DIRECTORY` environment variable. If the container managers
147 does this, the credentials passed to the service manager can be propagated
148 to services via `LoadCredential=` or `ImportCredential=` (see ...). The
149 container manager can choose any path, but `/run/host/credentials` is
152 ## Advanced Integration
154 1. Consider syncing `/etc/localtime` from the host file system into the
155 container. Make it a relative symlink to the containers's zoneinfo dir, as
156 usual. Tools rely on being able to determine the timezone setting from the
157 symlink value, and making it relative looks nice even if people list the
158 container's `/etc/` from the host.
160 2. Make the container journal available in the host, by automatically
161 symlinking the container journal directory into the host journal directory.
162 More precisely, link `/var/log/journal/<container-machine-id>` of the
163 container into the same dir of the host. Administrators can then
164 automatically browse all container journals (correctly interleaved) by
165 issuing `journalctl -m`. The container machine ID can be determined from
166 `/etc/machine-id` in the container.
168 3. If the container manager wants to cleanly shutdown the container, it might
169 be a good idea to send `SIGRTMIN+3` to its init process. systemd will then
170 do a clean shutdown. Note however, that since only systemd understands
171 `SIGRTMIN+3` like this, this might confuse other init systems.
173 4. To support [Socket Activated
174 Containers](http://0pointer.de/blog/projects/socket-activated-containers.html)
175 the container manager should be capable of being run as a systemd
176 service. It will then receive the sockets starting with FD 3, the number of
177 passed FDs in `$LISTEN_FDS` and its PID as `$LISTEN_PID`. It should take
178 these and pass them on to the container's init process, also setting
179 $LISTEN_FDS and `$LISTEN_PID` (basically, it can just leave the FDs and
180 `$LISTEN_FDS` untouched, but it needs to adjust `$LISTEN_PID` to the
181 container init process). That's all that's necessary to make socket
182 activation work. The protocol to hand sockets from systemd to services is
183 hence the same as from the container manager to the container systemd. For
184 further details see the explanations of
185 [sd_listen_fds(1)](http://0pointer.de/public/systemd-man/sd_listen_fds.html)
186 and the [blog story for service
187 developers](http://0pointer.de/blog/projects/socket-activation.html).
189 5. Container managers should stay away from the cgroup hierarchy outside of the
190 unit they created for their container. That's private property of systemd,
191 and no other code should modify it.
193 6. systemd running inside the container can report when boot-up is complete
194 using the usual `sd_notify()` protocol that is also used when a service
195 wants to tell the service manager about readiness. A container manager can
196 set the `$NOTIFY_SOCKET` environment variable to a suitable socket path to
197 make use of this functionality. (Also see information about
198 `/run/host/notify` below.)
202 1. Inside of a container, if a `veth` link is named `host0`, `systemd-networkd`
203 running inside of the container will by default run DHCPv4, DHCPv6, and
204 IPv4LL clients on it. It is thus recommended that container managers that
205 add a `veth` link to a container name it `host0`, to get an automatically
206 configured network, with no manual setup.
208 2. Outside of a container, if a `veth` link is prefixed "ve-", `systemd-networkd`
209 will by default run DHCPv4 and DHCPv6 servers on it, as well as IPv4LL. It
210 is thus recommended that container managers that add a `veth` link to a
211 container name the external side `ve-` + the container name.
213 3. It is recommended to configure stable MAC addresses for container `veth`
214 devices, for example, hashed out of the container names. That way it is more
215 likely that DHCP and IPv4LL will acquire stable addresses.
217 ## The `/run/host/` Hierarchy
219 Container managers may place certain resources the manager wants to provide to
220 the container payload below the `/run/host/` hierarchy. This hierarchy should
221 be mostly immutable (possibly some subdirs might be writable, but the top-level
222 hierarchy — and probably most subdirs should be read-only to the
223 container). Note that this hierarchy is used by various container managers, and
224 care should be taken to avoid naming conflicts. `systemd` (and in particular
225 `systemd-nspawn`) use the hierarchy for the following resources:
227 1. The `/run/host/incoming/` directory mount point is configured for `MS_SLAVE`
228 mount propagation with the host, and is used as intermediary location for
229 mounts to establish in the container, for the implementation of `machinectl
230 bind`. Container payload should usually not directly interact with this
231 directory: it's used by code outside the container to insert mounts inside
232 it only, and is mostly an internal vehicle to achieve this. Other container
233 managers that want to implement similar functionality might consider using
236 2. The `/run/host/inaccessible/` directory may be set up by the container
237 manager to include six file nodes: `reg`, `dir`, `fifo`, `sock`, `chr`,
238 `blk`. These nodes correspond with the six types of file nodes Linux knows
239 (with the exceptions of symlinks). Each node should be of the specific type
240 and have an all zero access mode, i.e. be inaccessible. The two device node
241 types should have major and minor of zero (which are unallocated devices on
242 Linux). These nodes are used as mount source for implementing the
243 `InaccessiblePath=` setting of unit files, i.e. file nodes to mask this way
244 are overmounted with these "inaccessible" inodes, guaranteeing that the file
245 node type does not change this way but the nodes still become
246 inaccessible. Note that systemd when run as PID 1 in the container payload
247 will create these nodes on its own if not passed in by the container
248 manager. However, in that case it likely lacks the privileges to create the
249 character and block devices nodes (there are fallbacks for this case).
251 3. The `/run/host/notify` path is a good choice to place the `sd_notify()`
252 socket in, that may be used for the container's PID 1 to report to the
253 container manager when boot-up is complete. The path used for this doesn't
254 matter much as it is communicated via the `$NOTIFY_SOCKET` environment
255 variable, following the usual protocol for this, however it's suitable, and
256 recommended place for this socket in case ready notification is desired.
258 4. The `/run/host/os-release` file contains the `/etc/os-release` file of the
259 host, i.e. may be used by the container payload to gather limited
260 information about the host environment, on top of what `uname -a` reports.
262 5. The `/run/host/container-manager` file may be used to pass the same
263 information as the `$container` environment variable (see above), i.e. a
264 short string identifying the container manager implementation. This file
265 should be newline terminated. Passing this information via this file has the
266 benefit that payload code can easily access it, even when running
267 unprivileged without access to the container PID 1's environment block.
269 6. The `/run/host/container-uuid` file may be used to pass the same information
270 as the `$container_uuid` environment variable (see above). This file should
271 be newline terminated.
273 7. The `/run/host/credentials/` directory is a good place to pass credentials
274 into the container, using the `$CREDENTIALS_DIRECTORY` protocol, see above.
276 ## What You Shouldn't Do
278 1. Do not drop `CAP_MKNOD` from the container. `PrivateDevices=` is a commonly
279 used service setting that provides a service with its own, private, minimal
280 version of `/dev/`. To set this up systemd in the container needs this
281 capability. If you take away the capability, then all services that set this
282 flag will cease to work. Use `BPF_PROG_TYPE_CGROUP_DEVICE` BPF programs — on
283 cgroupv2 — or the `devices` controller — on cgroupv1 — to restrict what
284 device nodes the container can create instead of taking away the capability
285 wholesale. (Also see the section about fully unprivileged containers below.)
287 2. Do not drop `CAP_SYS_ADMIN` from the container. A number of the most
288 commonly used file system namespacing related settings, such as
289 `PrivateDevices=`, `ProtectHome=`, `ProtectSystem=`, `MountFlags=`,
290 `PrivateTmp=`, `ReadWriteDirectories=`, `ReadOnlyDirectories=`,
291 `InaccessibleDirectories=`, and `MountFlags=` need to be able to open new
292 mount namespaces and the mount certain file systems into them. You break all
293 services that make use of these options if you drop the capability. Also
294 note that logind mounts `XDG_RUNTIME_DIR` as `tmpfs` for all logged in users
295 and that won't work either if you take away the capability. (Also see
296 section about fully unprivileged containers below.)
298 3. Do not cross-link `/dev/kmsg` with `/dev/console`. They are different things,
299 you cannot link them to each other.
301 4. Do not pretend that the real VTs are available in the container. The VT
302 subsystem consists of all the devices `/dev/tty[0-9]*`, `/dev/vcs*`,
303 `/dev/vcsa*` plus their `sysfs` counterparts. They speak specific `ioctl()`s
304 and understand specific escape sequences, that other ptys don't understand.
305 Hence, it is explicitly not OK to mount a pty to `/dev/tty1`, `/dev/tty2`,
306 `/dev/tty3`. This is explicitly not supported.
308 5. Don't pretend that passing arbitrary devices to containers could really work
309 well. For example, do not pass device nodes for block devices to the
310 container. Device access (with the exception of network devices) is not
311 virtualized on Linux. Enumeration and probing of meta information from
312 `/sys/` and elsewhere is not possible to do correctly in a container. Simply
313 adding a specific device node to a container's `/dev/` is *not* *enough* to
314 do the job, as `systemd-udevd` and suchlike are not available at all, and no
315 devices will appear available or enumerable, inside the container.
317 6. Don't mount only a sub-tree of the `cgroupfs` into the container. This will not
318 work as `/proc/$PID/cgroup` lists full paths and cannot be matched up with
319 the actual `cgroupfs` tree visible, then. (You may "prune" some branches
322 7. Do not make `/sys/` writable in the container. If you do,
323 `systemd-udevd.service` is started to manage your devices — inside the
324 container, but that will cause conflicts and errors given that the Linux
325 device model is not virtualized for containers on Linux and thus the
326 containers and the host would try to manage the same devices, fighting for
327 ownership. Multiple other subsystems of systemd similarly test for `/sys/`
328 being writable to decide whether to use `systemd-udevd` or assume that
329 device management is properly available on the instance. Among them
330 `systemd-networkd` and `systemd-logind`. The conditionalization on the
331 read-only state of `/sys/` enables a nice automatism: as soon as `/sys/` and
332 the Linux device model are changed to be virtualized properly the container
333 payload can make use of that, simply by marking `/sys/` writable. (Note that
334 as special exception, the devices in `/sys/class/net/` are virtualized
335 already, if network namespacing is used. Thus it is OK to mount the relevant
336 sub-directories of `/sys/` writable, but make sure to leave the root of
339 8. Do not pass the `CAP_AUDIT_CONTROL`, `CAP_AUDIT_READ`, `CAP_AUDIT_WRITE`
340 capabilities to the container, in particular not to those making use of user
341 namespaces. The kernel's audit subsystem is still not virtualized for
342 containers, and passing these credentials is pointless hence, given the
343 actual attempt to make use of the audit subsystem will fail. Note that
344 systemd's audit support is partially conditioned on these capabilities, thus
345 by dropping them you ensure that you get an entirely clean boot, as systemd
346 will make no attempt to use it. If you pass the capabilities to the payload
347 systemd will assume that audit is available and works, and some components
348 will subsequently fail in various ways. Note that once the kernel learnt
349 native support for container-virtualized audit, adding the capability to the
350 container description will automatically make the container payload use it.
352 ## Fully Unprivileged Container Payload
354 First things first, to make this clear: Linux containers are not a security
355 technology right now. There are more holes in the model than in swiss cheese.
357 For example: if you do not use user namespacing, and share root and other users
358 between container and host, the `struct user` structures will be shared between
359 host and container, and hence `RLIMIT_NPROC` and so of the container users
360 affect the host and other containers, and vice versa. This is a major security
361 hole, and actually is a real-life problem: since Avahi sets `RLIMIT_NPROC` of
362 its user to 2 (to effectively disallow `fork()`ing) you cannot run more than
363 one Avahi instance on the entire system...
365 People have been asking to be able to run systemd without `CAP_SYS_ADMIN` and
366 `CAP_SYS_MKNOD` in the container. This is now supported to some level in
367 systemd, but we recommend against it (see above). If `CAP_SYS_ADMIN` and
368 `CAP_SYS_MKNOD` are missing from the container systemd will now gracefully turn
369 off `PrivateTmp=`, `PrivateNetwork=`, `ProtectHome=`, `ProtectSystem=` and
370 others, because those capabilities are required to implement these options. The
371 services using these settings (which include many of systemd's own) will hence
372 run in a different, less secure environment when the capabilities are missing
373 than with them around.
375 With user namespacing in place things get much better. With user namespaces the
376 `struct user` issue described above goes away, and containers can keep
377 `CAP_SYS_ADMIN` safely for the user namespace, as capabilities are virtualized
378 and having capabilities inside a container doesn't mean one also has them
383 If you write software that wants to detect whether it is run in a container,
384 please check `/proc/1/environ` and look for the `container=` environment
385 variable. Do not assume the environment variable is inherited down the process
386 tree. It generally is not. Hence check the environment block of PID 1, not your
387 own. Note though that this file is only accessible to root. systemd hence early
388 on also copies the value into `/run/systemd/container`, which is readable for
389 everybody. However, that's a systemd-specific interface and other init systems
390 are unlikely to do the same.
392 Note that it is our intention to make systemd systems work flawlessly and
393 out-of-the-box in containers. In fact, we are interested to ensure that the same
394 OS image can be booted on a bare system, in a VM and in a container, and behave
395 correctly each time. If you notice that some component in systemd does not work
396 in a container as it should, even though the container manager implements
397 everything documented above, please contact us.
projects / thirdparty / systemd.git / blob