docs/CONTAINER_INTERFACE.md

   1 ---
   2 title: Container Interface
   3 category: Interfaces
   4 layout: default
   5 ---
   6
   7 # The Container Interface
   8
   9 Also consult [Writing Virtual Machine or Container
  10 Managers](http://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers).
  11
  12 systemd has a number of interfaces for interacting with container managers,
  13 when systemd is used inside of an OS container. If you work on a container
  14 manager, please consider supporting the following interfaces.
  15
  16 ## Execution Environment
  17
  18 1. If the container manager wants to control the hostname for a container
  19    running systemd it may just set it before invoking systemd, and systemd will
  20    leave it unmodified when there is no hostname configured in `/etc/hostname`
  21    (that file overrides whatever is pre-initialized by the container manager).
  22
  23 2. Make sure to pre-mount `/proc/`, `/sys/`, and `/sys/fs/selinux/` before
  24    invoking systemd, and mount `/proc/sys/`, `/sys/`, and `/sys/fs/selinux/`
  25    read-only in order to prevent the container from altering the host kernel's
  26    configuration settings. (As a special exception, if your container has
  27    network namespaces enabled, feel free to make `/proc/sys/net/` writable).
  28    systemd and various other subsystems (such as the SELinux userspace) have
  29    been modified to behave accordingly when these file systems are read-only.
  30    (It's OK to mount `/sys/` as `tmpfs` btw, and only mount a subset of its
  31    sub-trees from the real `sysfs` to hide `/sys/firmware/`, `/sys/kernel/` and
  32    so on. If you do that, make sure to mark `/sys/` read-only, as that
  33    condition is what systemd looks for, and is what is considered to be the API
  34    in this context.)
  35
  36 3. Pre-mount `/dev/` as (container private) `tmpfs` for the container and bind
  37    mount some suitable TTY to `/dev/console`. Also, make sure to create device
  38    nodes for `/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`,
  39    `/dev/urandom`, `/dev/tty`, `/dev/ptmx` in `/dev/`. It is not necessary to
  40    create `/dev/fd` or `/dev/stdout`, as systemd will do that on its own. Make
  41    sure to set up a `BPF_PROG_TYPE_CGROUP_DEVICE` BPF program — on cgroupv2 —
  42    or the `devices` cgroup controller — on cgroupv1 — so that no other devices
  43    but these may be created in the container. Note that many systemd services
  44    use `PrivateDevices=`, which means that systemd will set up a private
  45    `/dev/` for them for which it needs to be able to create these device nodes.
  46    Dropping `CAP_MKNOD` for containers is hence generally not advisable, but
  47    see below.
  48
  49 4. `systemd-udevd` is not available in containers (and refuses to start), and
  50    hence device dependencies are unavailable. The `systemd-udevd` unit files
  51    will check for `/sys/` being read-only, as an indication whether device
  52    management can work. Therefore make sure to mount `/sys/` read-only in the
  53    container (see above). Various clients of `systemd-udevd` also check the
  54    read-only state of `/sys/`, including PID 1 itself and `systemd-networkd`.
  55
  56 5. If systemd detects it is run in a container it will spawn a single shell on
  57    `/dev/console`, and not care about VTs or multiple gettys on VTs. (But see
  58    `$container_ttys` below.)
  59
  60 6. Either pre-mount all cgroup hierarchies in full into the container, or leave
  61    that to systemd which will do so if they are missing. Note that it is
  62    explicitly *not* OK to just mount a sub-hierarchy into the container as that
  63    is incompatible with `/proc/$PID/cgroup` (which lists full paths). Also the
  64    root-level cgroup directories tend to be quite different from inner
  65    directories, and that distinction matters. It is OK however, to mount the
  66    "upper" parts read-only of the hierarchies, and only allow write-access to
  67    the cgroup sub-tree the container runs in. It's also a good idea to mount
  68    all controller hierarchies with exception of `name=systemd` fully read-only
  69    (this only applies to cgroupv1, of course), to protect the controllers from
  70    alteration from inside the containers. Or to turn this around: only the
  71    cgroup sub-tree of the container itself (on cgroupv2 in the unified
  72    hierarchy, and on cgroupv1 in the `name=systemd` hierarchy) may be writable
  73    to the container.
  74
  75 7. Create the control group root of your container by either running your
  76    container as a service (in case you have one container manager instance per
  77    container instance) or creating one scope unit for each container instance
  78    via systemd's transient unit API (in case you have one container manager
  79    that manages all instances. Either way, make sure to set `Delegate=yes` in
  80    it. This ensures that that the unit you created will be part of all cgroup
  81    controllers (or at least the ones systemd understands). The latter may also
  82    be done via `systemd-machined`'s `CreateMachine()` API. Make sure to use the
  83    cgroup path systemd put your process in for all operations of the container.
  84    Do not add new cgroup directories to the top of the tree. This will not only
  85    confuse systemd and the admin, but also prevent your implementation from
  86    being "stackable".
  87
  88 ## Environment Variables
  89
  90 1. To allow systemd (and other programs) to identify that it is executed within
  91    a container, please set the `$container` environment variable for PID 1 in
  92    the container to a short lowercase string identifying your
  93    implementation. With this in place the `ConditionVirtualization=` setting in
  94    unit files will work properly. Example: `container=lxc-libvirt`
  95
  96 2. systemd has special support for allowing container managers to initialize
  97    the UUID for `/etc/machine-id` to some manager supplied value. This is only
  98    enabled if `/etc/machine-id` is empty (i.e. not yet set) at boot time of the
  99    container. The container manager should set `$container_uuid` as environment
 100    variable for the container's PID 1 to the container UUID. (This is similar
 101    to the effect of `qemu`'s `-uuid` switch). Note that you should pass only a
 102    UUID here that is actually unique (i.e. only one running container should
 103    have a specific UUID), and gets changed when a container gets duplicated.
 104    Also note that systemd will try to persistently store the UUID in
 105    `/etc/machine-id` (if writable) when this option is used, hence you should
 106    always pass the same UUID here. Keeping the externally used UUID for a
 107    container and the internal one in sync is hopefully useful to minimize
 108    surprise for the administrator.
 109
 110 3. systemd can automatically spawn login gettys on additional ptys. A container
 111    manager can set the `$container_ttys` environment variable for the
 112    container's PID 1 to tell it on which ptys to spawn gettys. The variable
 113    should take a space separated list of pty names, without the leading `/dev/`
 114    prefix, but with the `pts/` prefix included. Note that despite the
 115    variable's name you may only specify ptys, and not other types of ttys. Also
 116    you need to specify the pty itself, a symlink will not suffice. This is
 117    implemented in
 118    [systemd-getty-generator(8)](https://www.freedesktop.org/software/systemd/man/systemd-getty-generator.html).
 119    Note that this variable should not include the pty that `/dev/console` maps
 120    to if it maps to one (see below). Example: if the container receives
 121    `container_ttys=pts/7 pts/8 pts/14` it will spawn three additional login
 122    gettys on ptys 7, 8, and 14.
 123
 124 4. To allow applications to detect the OS version and other metadata of the host
 125    running the container manager, if this is considered desirable, please parse
 126    the host's `/etc/os-release` and set a `$container_host_<key>=<VALUE>`
 127    environment variable for the ID fields described by the [os-release
 128    interface](https://www.freedesktop.org/software/systemd/man/os-release.html), eg:
 129    `$container_host_id=debian`
 130    `$container_host_build_id=2020-06-15`
 131    `$container_host_variant_id=server`
 132    `$container_host_version_id=10`
 133
 134 ## Advanced Integration
 135
 136 1. Consider syncing `/etc/localtime` from the host file system into the
 137    container. Make it a relative symlink to the containers's zoneinfo dir, as
 138    usual. Tools rely on being able to determine the timezone setting from the
 139    symlink value, and making it relative looks nice even if people list the
 140    container's `/etc/` from the host.
 141
 142 2. Make the container journal available in the host, by automatically
 143    symlinking the container journal directory into the host journal directory.
 144    More precisely, link `/var/log/journal/<container-machine-id>` of the
 145    container into the same dir of the host. Administrators can then
 146    automatically browse all container journals (correctly interleaved) by
 147    issuing `journalctl -m`. The container machine ID can be determined from
 148    `/etc/machine-id` in the container.
 149
 150 3. If the container manager wants to cleanly shutdown the container, it might
 151    be a good idea to send `SIGRTMIN+3` to its init process. systemd will then
 152    do a clean shutdown. Note however, that since only systemd understands
 153    `SIGRTMIN+3` like this, this might confuse other init systems.
 154
 155 4. To support [Socket Activated
 156    Containers](http://0pointer.de/blog/projects/socket-activated-containers.html)
 157    the container manager should be capable of being run as a systemd
 158    service. It will then receive the sockets starting with FD 3, the number of
 159    passed FDs in `$LISTEN_FDS` and its PID as `$LISTEN_PID`. It should take
 160    these and pass them on to the container's init process, also setting
 161    $LISTEN_FDS and `$LISTEN_PID` (basically, it can just leave the FDs and
 162    `$LISTEN_FDS` untouched, but it needs to adjust `$LISTEN_PID` to the
 163    container init process). That's all that's necessary to make socket
 164    activation work. The protocol to hand sockets from systemd to services is
 165    hence the same as from the container manager to the container systemd. For
 166    further details see the explanations of
 167    [sd_listen_fds(1)](http://0pointer.de/public/systemd-man/sd_listen_fds.html)
 168    and the [blog story for service
 169    developers](http://0pointer.de/blog/projects/socket-activation.html).
 170
 171 5. Container managers should stay away from the cgroup hierarchy outside of the
 172    unit they created for their container. That's private property of systemd,
 173    and no other code should modify it.
 174
 175 6. systemd running inside the container can report when boot-up is complete
 176    using the usual `sd_notify()` protocol that is also used when a service
 177    wants to tell the service manager about readiness. A container manager can
 178    set the `$NOTIFY_SOCKET` environment variable to a suitable socket path to
 179    make use of this functionality. (Also see information about
 180    `/run/host/notify` below.)
 181
 182 ## Networking
 183
 184 1. Inside of a container, if a `veth` link is named `host0`, `systemd-networkd`
 185    running inside of the container will by default run DHCPv4, DHCPv6, and
 186    IPv4LL clients on it. It is thus recommended that container managers that
 187    add a `veth` link to a container name it `host0`, to get an automatically
 188    configured network, with no manual setup.
 189
 190 2. Outside of a container, if a `veth` link is prefixed "ve-", `systemd-networkd`
 191    will by default run DHCPv4 and DHCPv6 servers on it, as well as IPv4LL. It
 192    is thus recommended that container managers that add a `veth` link to a
 193    container name the external side `ve-` + the container name.
 194
 195 3. It is recommended to configure stable MAC addresses for container `veth`
 196    devices, for example hashed out of the container names. That way it is more
 197    likely that DHCP and IPv4LL will acquire stable addresses.
 198
 199 ## The `/run/host/` Hierarchy
 200
 201 Container managers may place certain resources the manager wants to provide to
 202 the container payload below the `/run/host/` hierarchy. This hierarchy should
 203 be mostly immutable (possibly some subdirs might be writable, but the top-level
 204 hierarchy — and probably most subdirs should be read-only to the
 205 container). Note that this hierarchy is used by various container managers, and
 206 care should be taken to avoid naming conflicts. `systemd` (and in particular
 207 `systemd-nspawn`) use the hierarchy for the following resources:
 208
 209 1. The `/run/host/incoming/` directory mount point is configured for `MS_SLAVE`
 210    mount propagation with the host, and is used as intermediary location for
 211    mounts to establish in the container, for the implementation of `machinectl
 212    bind`. Container payload should usually not directly interact with this
 213    directory: it's used by code outside the container to insert mounts inside
 214    it only, and is mostly an internal vehicle to achieve this. Other container
 215    managers that want to implement similar functionality might consider using
 216    the same directory.
 217
 218 2. The `/run/host/inaccessible/` directory may be set up by the container
 219    manager to include six file nodes: `reg`, `dir`, `fifo`, `sock`, `chr`,
 220    `blk`. These nodes correspond with the six types of file nodes Linux knows
 221    (with the exceptions of symlinks). Each node should be of the specific type
 222    and have an all zero access mode, i.e. be inaccessible. The two device node
 223    types should have major and minor of zero (which are unallocated devices on
 224    Linux). These nodes are used as mount source for implementing the
 225    `InaccessiblePath=` setting of unit files, i.e. file nodes to mask this way
 226    are overmounted with these "inaccessible" inodes, guaranteeing that the file
 227    node type does not change this way but the nodes still become
 228    inaccessible. Note that systemd when run as PID 1 in the container payload
 229    will create these nodes on its own if not passed in by the container
 230    manager. However, in that case it likely lacks the privileges to create the
 231    character and block devices nodes (there all fallbacks for this case).
 232
 233 3. The `/run/host/notify` path is a good choice to place the `sd_notify()`
 234    socket in, that may be used for the container's PID 1 to report to the
 235    container manager when boot-up is complete. The path used for this doesn't
 236    matter much as it is communicated via the `$NOTIFY_SOCKET` environment
 237    variable, following the usual protocol for this, however it's suitable, and
 238    recommended place for this socket in case ready notification is desired.
 239
 240 4. The `/run/host/os-release` file contains the `/etc/os-release` file of the
 241    host, i.e. may be used by the container payload to gather limited
 242    information about the host environment, on top of what `uname -a` reports.
 243
 244 5. The `/run/host/container-manager` file may be used to pass the same
 245    information as the `$container` environment variable (see above), i.e. a
 246    short string identifying the container manager implementation. This file
 247    should be newline terminated. Passing this information via this file has the
 248    benefit that payload code can easily access it, even when running
 249    unprivileged without access to the container PID1's environment block.
 250
 251 6. The `/run/host/container-uuid` file may be used to pass the same information
 252    as the `$container_uuid` environment variable (see above). This file should
 253    be newline terminated.
 254
 255 ## What You Shouldn't Do
 256
 257 1. Do not drop `CAP_MKNOD` from the container. `PrivateDevices=` is a commonly
 258    used service setting that provides a service with its own, private, minimal
 259    version of `/dev/`. To set this up systemd in the container needs this
 260    capability. If you take away the capability than all services that set this
 261    flag will cease to work. Use `BPF_PROG_TYPE_CGROUP_DEVICE` BPF programs — on
 262    cgroupv2 — or the `devices` controller — on cgroupv1 — to restrict what
 263    device nodes the container can create instead of taking away the capability
 264    wholesale. (Also see the section about fully unprivileged containers below.)
 265
 266 2. Do not drop `CAP_SYS_ADMIN` from the container. A number of the most
 267    commonly used file system namespacing related settings, such as
 268    `PrivateDevices=`, `ProtectHome=`, `ProtectSystem=`, `MountFlags=`,
 269    `PrivateTmp=`, `ReadWriteDirectories=`, `ReadOnlyDirectories=`,
 270    `InaccessibleDirectories=`, and `MountFlags=` need to be able to open new
 271    mount namespaces and the mount certain file systems into them. You break all
 272    services that make use of these options if you drop the capability. Also
 273    note that logind mounts `XDG_RUNTIME_DIR` as `tmpfs` for all logged in users
 274    and that won't work either if you take away the capability. (Also see
 275    section about fully unprivileged containers below.)
 276
 277 3. Do not cross-link `/dev/kmsg` with `/dev/console`. They are different things,
 278    you cannot link them to each other.
 279
 280 4. Do not pretend that the real VTs are available in the container. The VT
 281    subsystem consists of all the devices `/dev/tty*`, `/dev/vcs*`, `/dev/vcsa*`
 282    plus their `sysfs` counterparts. They speak specific `ioctl()`s and
 283    understand specific escape sequences, that other ptys don't understand.
 284    Hence, it is explicitly not OK to mount a pty to `/dev/tty1`, `/dev/tty2`,
 285    `/dev/tty3`. This is explicitly not supported.
 286
 287 5. Don't pretend that passing arbitrary devices to containers could really work
 288    well. For example, do not pass device nodes for block devices to the
 289    container. Device access (with the exception of network devices) is not
 290    virtualized on Linux. Enumeration and probing of meta information from
 291    `/sys/` and elsewhere is not possible to do correctly in a container. Simply
 292    adding a specific device node to a container's `/dev/` is *not* *enough* to
 293    do the job, as `systemd-udevd` and suchlike are not available at all, and no
 294    devices will appear available or enumerable, inside the container.
 295
 296 6. Don't mount only a sub-tree of the `cgroupfs` into the container. This will not
 297    work as `/proc/$PID/cgroup` lists full paths and cannot be matched up with
 298    the actual `cgroupfs` tree visible, then. (You may "prune" some branches
 299    though, see above.)
 300
 301 7. Do not make `/sys/` writable in the container. If you do,
 302    `systemd-udevd.service` is started to manage your devices — inside the
 303    container, but that will cause conflicts and errors given that the Linux
 304    device model is not virtualized for containers on Linux and thus the
 305    containers and the host would try to manage the same devices, fighting for
 306    ownership. Multiple other subsystems of systemd similarly test for `/sys/`
 307    being writable to decide whether to use `systemd-udevd` or assume that
 308    device management is properly available on the instance. Among them
 309    `systemd-networkd` and `systemd-logind`. The conditionalization on the
 310    read-only state of `/sys/` enables a nice automatism: as soon as `/sys/` and
 311    the Linux device model are changed to be virtualized properly the container
 312    payload can make use of that, simply by marking `/sys/` writable. (Note that
 313    as special exception, the devices in `/sys/class/net/` are virtualized
 314    already, if network namespacing is used. Thus it is OK to mount the relevant
 315    sub-directories of `/sys/` writable, but make sure to leave the root of
 316    `/sys/` read-only.)
 317
 318 ## Fully Unprivileged Container Payload
 319
 320 First things first, to make this clear: Linux containers are not a security
 321 technology right now. There are more holes in the model than in swiss cheese.
 322
 323 For example: if you do not use user namespacing, and share root and other users
 324 between container and host, the `struct user` structures will be shared between
 325 host and container, and hence `RLIMIT_NPROC` and so of the container users
 326 affect the host and other containers, and vice versa. This is a major security
 327 hole, and actually is a real-life problem: since Avahi sets `RLIMIT_NPROC` of
 328 its user to 2 (to effectively disallow `fork()`ing) you cannot run more than
 329 one Avahi instance on the entire system...
 330
 331 People have been asking to be able to run systemd without `CAP_SYS_ADMIN` and
 332 `CAP_SYS_MKNOD` in the container. This is now supported to some level in
 333 systemd, but we recommend against it (see above). If `CAP_SYS_ADMIN` and
 334 `CAP_SYS_MKNOD` are missing from the container systemd will now gracefully turn
 335 off `PrivateTmp=`, `PrivateNetwork=`, `ProtectHome=`, `ProtectSystem=` and
 336 others, because those capabilities are required to implement these options. The
 337 services using these settings (which include many of systemd's own) will hence
 338 run in a different, less secure environment when the capabilities are missing
 339 than with them around.
 340
 341 With user namespacing in place things get much better. With user namespaces the
 342 `struct user` issue described above goes away, and containers can keep
 343 `CAP_SYS_ADMIN` safely for the user namespace, as capabilities are virtualized
 344 and having capabilities inside a container doesn't mean one also has them
 345 outside.
 346
 347 ## Final Words
 348
 349 If you write software that wants to detect whether it is run in a container,
 350 please check `/proc/1/environ` and look for the `container=` environment
 351 variable. Do not assume the environment variable is inherited down the process
 352 tree. It generally is not. Hence check the environment block of PID 1, not your
 353 own. Note though that that file is only accessible to root. systemd hence early
 354 on also copies the value into `/run/systemd/container`, which is readable for
 355 everybody. However, that's a systemd-specific interface and other init systems
 356 are unlikely to do the same.
 357
 358 Note that it is our intention to make systemd systems work flawlessly and
 359 out-of-the-box in containers. In fact we are interested to ensure that the same
 360 OS image can be booted on a bare system, in a VM and in a container, and behave
 361 correctly each time. If you notice that some component in systemd does not work
 362 in a container as it should, even though the container manager implements
 363 everything documented above, please contact us.