docs/CONTAINER_INTERFACE.md

   1 ---
   2 title: Container Interface
   3 category: Interfaces
   4 layout: default
   5 SPDX-License-Identifier: LGPL-2.1-or-later
   6 ---
   7
   8 # The Container Interface
   9
  10 Also consult [Writing Virtual Machine or Container
  11 Managers](https://systemd.io/WRITING_VM_AND_CONTAINER_MANAGERS).
  12
  13 systemd has a number of interfaces for interacting with container managers,
  14 when systemd is used inside of an OS container. If you work on a container
  15 manager, please consider supporting the following interfaces.
  16
  17 ## Execution Environment
  18
  19 1. If the container manager wants to control the hostname for a container
  20    running systemd it may just set it before invoking systemd, and systemd will
  21    leave it unmodified when there is no hostname configured in `/etc/hostname`
  22    (that file overrides whatever is pre-initialized by the container manager).
  23
  24 2. Make sure to pre-mount `/proc/`, `/sys/`, and `/sys/fs/selinux/` before
  25    invoking systemd, and mount `/sys/`, `/sys/fs/selinux/` and `/proc/sys/`
  26    read-only (the latter via e.g. a read-only bind mount on itself) in order
  27    to prevent the container from altering the host kernel's configuration
  28    settings. (As a special exception, if your container has network namespaces
  29    enabled, feel free to make `/proc/sys/net/` writable. If it also has user, ipc,
  30    uts and pid namespaces enabled, the entire `/proc/sys` can be left writable).
  31    systemd and various other subsystems (such as the SELinux userspace) have
  32    been modified to behave accordingly when these file systems are read-only.
  33    (It's OK to mount `/sys/` as `tmpfs` btw, and only mount a subset of its
  34    sub-trees from the real `sysfs` to hide `/sys/firmware/`, `/sys/kernel/` and
  35    so on. If you do that, make sure to mark `/sys/` read-only, as that
  36    condition is what systemd looks for, and is what is considered to be the API
  37    in this context.)
  38
  39 3. Pre-mount `/dev/` as (container private) `tmpfs` for the container and bind
  40    mount some suitable TTY to `/dev/console`. If this is a pty, make sure to
  41    not close the controlling pty during systemd's lifetime. PID 1 will close
  42    ttys, to avoid being killed by SAK. It only opens ttys for the time it
  43    actually needs to print something. Also, make sure to create device nodes
  44    for `/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom`,
  45    `/dev/tty`, `/dev/ptmx` in `/dev/`. It is not necessary to create `/dev/fd`
  46    or `/dev/stdout`, as systemd will do that on its own. Make sure to set up a
  47    `BPF_PROG_TYPE_CGROUP_DEVICE` BPF program — on cgroupv2 — or the `devices`
  48    cgroup controller — on cgroupv1 — so that no other devices but these may be
  49    created in the container. Note that many systemd services use
  50    `PrivateDevices=`, which means that systemd will set up a private `/dev/`
  51    for them for which it needs to be able to create these device nodes.
  52    Dropping `CAP_MKNOD` for containers is hence generally not advisable, but
  53    see below.
  54
  55 4. `systemd-udevd` is not available in containers (and refuses to start), and
  56    hence device dependencies are unavailable. The `systemd-udevd` unit files
  57    will check for `/sys/` being read-only, as an indication whether device
  58    management can work. Therefore make sure to mount `/sys/` read-only in the
  59    container (see above). Various clients of `systemd-udevd` also check the
  60    read-only state of `/sys/`, including PID 1 itself and `systemd-networkd`.
  61
  62 5. If systemd detects it is run in a container it will spawn a single shell on
  63    `/dev/console`, and not care about VTs or multiple gettys on VTs. (But see
  64    `$container_ttys` below.)
  65
  66 6. Either pre-mount all cgroup hierarchies in full into the container, or leave
  67    that to systemd which will do so if they are missing. Note that it is
  68    explicitly *not* OK to just mount a sub-hierarchy into the container as that
  69    is incompatible with `/proc/$PID/cgroup` (which lists full paths). Also the
  70    root-level cgroup directories tend to be quite different from inner
  71    directories, and that distinction matters. It is OK however, to mount the
  72    "upper" parts read-only of the hierarchies, and only allow write-access to
  73    the cgroup sub-tree the container runs in. It's also a good idea to mount
  74    all controller hierarchies with exception of `name=systemd` fully read-only
  75    (this only applies to cgroupv1, of course), to protect the controllers from
  76    alteration from inside the containers. Or to turn this around: only the
  77    cgroup sub-tree of the container itself (on cgroupv2 in the unified
  78    hierarchy, and on cgroupv1 in the `name=systemd` hierarchy) may be writable
  79    to the container.
  80
  81 7. Create the control group root of your container by either running your
  82    container as a service (in case you have one container manager instance per
  83    container instance) or creating one scope unit for each container instance
  84    via systemd's transient unit API (in case you have one container manager
  85    that manages all instances. Either way, make sure to set `Delegate=yes` in
  86    it. This ensures that the unit you created will be part of all cgroup
  87    controllers (or at least the ones systemd understands). The latter may also
  88    be done via `systemd-machined`'s `CreateMachine()` API. Make sure to use the
  89    cgroup path systemd put your process in for all operations of the container.
  90    Do not add new cgroup directories to the top of the tree. This will not only
  91    confuse systemd and the admin, but also prevent your implementation from
  92    being "stackable".
  93
  94 ## Environment Variables
  95
  96 1. To allow systemd (and other programs) to identify that it is executed within
  97    a container, please set the `$container` environment variable for PID 1 in
  98    the container to a short lowercase string identifying your
  99    implementation. With this in place the `ConditionVirtualization=` setting in
 100    unit files will work properly. Example: `container=lxc-libvirt`
 101
 102 2. systemd has special support for allowing container managers to initialize
 103    the UUID for `/etc/machine-id` to some manager supplied value. This is only
 104    enabled if `/etc/machine-id` is empty (i.e. not yet set) at boot time of the
 105    container. The container manager should set `$container_uuid` as environment
 106    variable for the container's PID 1 to the container UUID. (This is similar
 107    to the effect of `qemu`'s `-uuid` switch). Note that you should pass only a
 108    UUID here that is actually unique (i.e. only one running container should
 109    have a specific UUID), and gets changed when a container gets duplicated.
 110    Also note that systemd will try to persistently store the UUID in
 111    `/etc/machine-id` (if writable) when this option is used, hence you should
 112    always pass the same UUID here. Keeping the externally used UUID for a
 113    container and the internal one in sync is hopefully useful to minimize
 114    surprise for the administrator.
 115
 116 3. systemd can automatically spawn login gettys on additional ptys. A container
 117    manager can set the `$container_ttys` environment variable for the
 118    container's PID 1 to tell it on which ptys to spawn gettys. The variable
 119    should take a space separated list of pty names, without the leading `/dev/`
 120    prefix, but with the `pts/` prefix included. Note that despite the
 121    variable's name you may only specify ptys, and not other types of ttys. Also
 122    you need to specify the pty itself, a symlink will not suffice. This is
 123    implemented in
 124    [systemd-getty-generator(8)](https://www.freedesktop.org/software/systemd/man/latest/systemd-getty-generator.html).
 125    Note that this variable should not include the pty that `/dev/console` maps
 126    to if it maps to one (see below). Example: if the container receives
 127    `container_ttys=pts/7 pts/8 pts/14` it will spawn three additional login
 128    gettys on ptys 7, 8, and 14.
 129
 130 4. To allow applications to detect the OS version and other metadata of the host
 131    running the container manager, if this is considered desirable, please parse
 132    the host's `/etc/os-release` and set a `$container_host_<key>=<VALUE>`
 133    environment variable for the ID fields described by the [os-release
 134    interface](https://www.freedesktop.org/software/systemd/man/latest/os-release.html), eg:
 135    `$container_host_id=debian`
 136    `$container_host_build_id=2020-06-15`
 137    `$container_host_variant_id=server`
 138    `$container_host_version_id=10`
 139
 140 5. systemd supports passing immutable binary data blobs with limited size and
 141    restricted access to services via the `ImportCredential=`, `LoadCredential=`
 142    and `SetCredential=` settings. The same protocol may be used to pass credentials
 143    from the container manager to systemd itself. The credential data should be
 144    placed in some location (ideally a read-only and non-swappable file system,
 145    like 'ramfs'), and the absolute path to this directory exported in the
 146    `$CREDENTIALS_DIRECTORY` environment variable. If the container managers
 147    does this, the credentials passed to the service manager can be propagated
 148    to services via `LoadCredential=` or `ImportCredential=` (see ...). The
 149    container manager can choose any path, but `/run/host/credentials` is
 150    recommended.
 151
 152 ## Advanced Integration
 153
 154 1. Consider syncing `/etc/localtime` from the host file system into the
 155    container. Make it a relative symlink to the containers's zoneinfo dir, as
 156    usual. Tools rely on being able to determine the timezone setting from the
 157    symlink value, and making it relative looks nice even if people list the
 158    container's `/etc/` from the host.
 159
 160 2. Make the container journal available in the host, by automatically
 161    symlinking the container journal directory into the host journal directory.
 162    More precisely, link `/var/log/journal/<container-machine-id>` of the
 163    container into the same dir of the host. Administrators can then
 164    automatically browse all container journals (correctly interleaved) by
 165    issuing `journalctl -m`. The container machine ID can be determined from
 166    `/etc/machine-id` in the container.
 167
 168 3. If the container manager wants to cleanly shut down the container, it might
 169    be a good idea to send `SIGRTMIN+3` to its init process. systemd will then
 170    do a clean shutdown. Note however, that since only systemd understands
 171    `SIGRTMIN+3` like this, this might confuse other init systems. A container
 172    manager may implement the `$NOTIFY_SOCKET` protocol mentioned below in which
 173    case it will receive a notification message `X_SYSTEMD_SIGNALS_LEVEL=2` that
 174    indicates if and when these additional signal handlers are installed. If
 175    these signals are sent to the container's PID 1 before this notification
 176    message is sent they might not be handled correctly yet.
 177
 178 4. To support [Socket Activated
 179    Containers](https://0pointer.de/blog/projects/socket-activated-containers.html)
 180    the container manager should be capable of being run as a systemd
 181    service. It will then receive the sockets starting with FD 3, the number of
 182    passed FDs in `$LISTEN_FDS` and its PID as `$LISTEN_PID`. It should take
 183    these and pass them on to the container's init process, also setting
 184    $LISTEN_FDS and `$LISTEN_PID` (basically, it can just leave the FDs and
 185    `$LISTEN_FDS` untouched, but it needs to adjust `$LISTEN_PID` to the
 186    container init process). That's all that's necessary to make socket
 187    activation work. The protocol to hand sockets from systemd to services is
 188    hence the same as from the container manager to the container systemd. For
 189    further details see the explanations of
 190    [sd_listen_fds(1)](https://0pointer.de/public/systemd-man/sd_listen_fds.html)
 191    and the [blog story for service
 192    developers](https://0pointer.de/blog/projects/socket-activation.html).
 193
 194 5. Container managers should stay away from the cgroup hierarchy outside of the
 195    unit they created for their container. That's private property of systemd,
 196    and no other code should modify it.
 197
 198 6. systemd running inside the container can report when boot-up is complete,
 199    boot progress and functionality as well as various other bits of system
 200    information using the `sd_notify()` protocol that is also used when a
 201    service wants to tell the service manager about readiness. A container
 202    manager can set the `$NOTIFY_SOCKET` environment variable to a suitable
 203    socket path to make use of this functionality. (Also see information about
 204    `/run/host/notify` below, as well as the Readiness Protocol section on
 205    [systemd(1)](https://www.freedesktop.org/software/systemd/man/latest/systemd.html)
 206
 207 ## Networking
 208
 209 1. Inside of a container, if a `veth` link is named `host0`, `systemd-networkd`
 210    running inside of the container will by default run DHCPv4, DHCPv6, and
 211    IPv4LL clients on it. It is thus recommended that container managers that
 212    add a `veth` link to a container name it `host0`, to get an automatically
 213    configured network, with no manual setup.
 214
 215 2. Outside of a container, if a `veth` link is prefixed "ve-", `systemd-networkd`
 216    will by default run DHCPv4 and DHCPv6 servers on it, as well as IPv4LL. It
 217    is thus recommended that container managers that add a `veth` link to a
 218    container name the external side `ve-` + the container name.
 219
 220 3. It is recommended to configure stable MAC addresses for container `veth`
 221    devices, for example, hashed out of the container names. That way it is more
 222    likely that DHCP and IPv4LL will acquire stable addresses.
 223
 224 ## The `/run/host/` Hierarchy
 225
 226 Container managers may place certain resources the manager wants to provide to
 227 the container payload below the `/run/host/` hierarchy. This hierarchy should
 228 be mostly immutable (possibly some subdirs might be writable, but the top-level
 229 hierarchy — and probably most subdirs should be read-only to the
 230 container). Note that this hierarchy is used by various container managers, and
 231 care should be taken to avoid naming conflicts. `systemd` (and in particular
 232 `systemd-nspawn`) use the hierarchy for the following resources:
 233
 234 1. The `/run/host/incoming/` directory mount point is configured for `MS_SLAVE`
 235    mount propagation with the host, and is used as intermediary location for
 236    mounts to establish in the container, for the implementation of `machinectl
 237    bind`. Container payload should usually not directly interact with this
 238    directory: it's used by code outside the container to insert mounts inside
 239    it only, and is mostly an internal vehicle to achieve this. Other container
 240    managers that want to implement similar functionality might consider using
 241    the same directory.
 242
 243 2. The `/run/host/inaccessible/` directory may be set up by the container
 244    manager to include six file nodes: `reg`, `dir`, `fifo`, `sock`, `chr`,
 245    `blk`. These nodes correspond with the six types of file nodes Linux knows
 246    (with the exceptions of symlinks). Each node should be of the specific type
 247    and have an all zero access mode, i.e. be inaccessible. The two device node
 248    types should have major and minor of zero (which are unallocated devices on
 249    Linux). These nodes are used as mount source for implementing the
 250    `InaccessiblePath=` setting of unit files, i.e. file nodes to mask this way
 251    are overmounted with these "inaccessible" inodes, guaranteeing that the file
 252    node type does not change this way but the nodes still become
 253    inaccessible. Note that systemd when run as PID 1 in the container payload
 254    will create these nodes on its own if not passed in by the container
 255    manager. However, in that case it likely lacks the privileges to create the
 256    character and block devices nodes (there are fallbacks for this case).
 257
 258 3. The `/run/host/notify` path is a good choice to place the `sd_notify()`
 259    socket in, that may be used for the container's PID 1 to report to the
 260    container manager when boot-up is complete. The path used for this doesn't
 261    matter much as it is communicated via the `$NOTIFY_SOCKET` environment
 262    variable, following the usual protocol for this, however it's suitable, and
 263    recommended place for this socket in case ready notification is desired.
 264
 265 4. The `/run/host/os-release` file contains the `/etc/os-release` file of the
 266    host, i.e. may be used by the container payload to gather limited
 267    information about the host environment, on top of what `uname -a` reports.
 268
 269 5. The `/run/host/container-manager` file may be used to pass the same
 270    information as the `$container` environment variable (see above), i.e. a
 271    short string identifying the container manager implementation. This file
 272    should be newline terminated. Passing this information via this file has the
 273    benefit that payload code can easily access it, even when running
 274    unprivileged without access to the container PID 1's environment block.
 275
 276 6. The `/run/host/container-uuid` file may be used to pass the same information
 277    as the `$container_uuid` environment variable (see above). This file should
 278    be newline terminated.
 279
 280 7. The `/run/host/credentials/` directory is a good place to pass credentials
 281    into the container, using the `$CREDENTIALS_DIRECTORY` protocol, see above.
 282
 283 8. The `/run/host/unix-export/` directory shall be writable from the container
 284    payload, and is where container payload can bind `AF_UNIX` sockets in that
 285    shall be *exported* to the host, so that the host can connect to them. The
 286    container manager should bind mount this directory on the host side
 287    (read-only ideally), so that the host can connect to contained sockets. This
 288    is most prominently used by `systemd-ssh-generator` when run in such a
 289    container to automatically bind an SSH socket into that directory, which
 290    then can be used to connect to the container.
 291
 292 9. The `/run/host/unix-export/ssh` `AF_UNIX` socket will be automatically bound
 293    by `systemd-ssh-generator` in the container if possible, and can be used to
 294    connect to the container.
 295
 296 10. The `/run/host/userdb/` directory may be used to drop-in additional JSON
 297     user records that `nss-systemd` inside the container shall include in the
 298     system's user database. This is useful to make host users and their home
 299     directories automatically accessible to containers in transitive
 300     fashion. See `nss-systemd(8)` for details.
 301
 302 11. The `/run/host/home/` directory may be used to bind mount host home
 303     directories of users that shall be made available in the container to. This
 304     may be used in combination with `/run/host/userdb/` above: one defines the
 305     user record, the other contains the user's home directory.
 306
 307 ## What You Shouldn't Do
 308
 309 1. Do not drop `CAP_MKNOD` from the container. `PrivateDevices=` is a commonly
 310    used service setting that provides a service with its own, private, minimal
 311    version of `/dev/`. To set this up systemd in the container needs this
 312    capability. If you take away the capability, then all services that set this
 313    flag will cease to work. Use `BPF_PROG_TYPE_CGROUP_DEVICE` BPF programs — on
 314    cgroupv2 — or the `devices` controller — on cgroupv1 — to restrict what
 315    device nodes the container can create instead of taking away the capability
 316    wholesale. (Also see the section about fully unprivileged containers below.)
 317
 318 2. Do not drop `CAP_SYS_ADMIN` from the container. A number of the most
 319    commonly used file system namespacing related settings, such as
 320    `PrivateDevices=`, `ProtectHome=`, `ProtectSystem=`, `MountFlags=`,
 321    `PrivateTmp=`, `ReadWriteDirectories=`, `ReadOnlyDirectories=`,
 322    `InaccessibleDirectories=`, and `MountFlags=` need to be able to open new
 323    mount namespaces and the mount certain file systems into them. You break all
 324    services that make use of these options if you drop the capability. Also
 325    note that logind mounts `XDG_RUNTIME_DIR` as `tmpfs` for all logged in users
 326    and that won't work either if you take away the capability. (Also see
 327    section about fully unprivileged containers below.)
 328
 329 3. Do not cross-link `/dev/kmsg` with `/dev/console`. They are different things,
 330    you cannot link them to each other.
 331
 332 4. Do not pretend that the real VTs are available in the container. The VT
 333    subsystem consists of all the devices `/dev/tty[0-9]*`, `/dev/vcs*`,
 334    `/dev/vcsa*` plus their `sysfs` counterparts. They speak specific `ioctl()`s
 335    and understand specific escape sequences, that other ptys don't understand.
 336    Hence, it is explicitly not OK to mount a pty to `/dev/tty1`, `/dev/tty2`,
 337    `/dev/tty3`. This is explicitly not supported.
 338
 339 5. Don't pretend that passing arbitrary devices to containers could really work
 340    well. For example, do not pass device nodes for block devices to the
 341    container. Device access (with the exception of network devices) is not
 342    virtualized on Linux. Enumeration and probing of meta information from
 343    `/sys/` and elsewhere is not possible to do correctly in a container. Simply
 344    adding a specific device node to a container's `/dev/` is *not* *enough* to
 345    do the job, as `systemd-udevd` and suchlike are not available at all, and no
 346    devices will appear available or enumerable, inside the container.
 347
 348 6. Don't mount only a sub-tree of the `cgroupfs` into the container. This will not
 349    work as `/proc/$PID/cgroup` lists full paths and cannot be matched up with
 350    the actual `cgroupfs` tree visible, then. (You may "prune" some branches
 351    though, see above.)
 352
 353 7. Do not make `/sys/` writable in the container. If you do,
 354    `systemd-udevd.service` is started to manage your devices — inside the
 355    container, but that will cause conflicts and errors given that the Linux
 356    device model is not virtualized for containers on Linux and thus the
 357    containers and the host would try to manage the same devices, fighting for
 358    ownership. Multiple other subsystems of systemd similarly test for `/sys/`
 359    being writable to decide whether to use `systemd-udevd` or assume that
 360    device management is properly available on the instance. Among them
 361    `systemd-networkd` and `systemd-logind`. The conditionalization on the
 362    read-only state of `/sys/` enables a nice automatism: as soon as `/sys/` and
 363    the Linux device model are changed to be virtualized properly the container
 364    payload can make use of that, simply by marking `/sys/` writable. (Note that
 365    as special exception, the devices in `/sys/class/net/` are virtualized
 366    already, if network namespacing is used. Thus it is OK to mount the relevant
 367    sub-directories of `/sys/` writable, but make sure to leave the root of
 368    `/sys/` read-only.)
 369
 370 8. Do not pass the `CAP_AUDIT_CONTROL`, `CAP_AUDIT_READ`, `CAP_AUDIT_WRITE`
 371    capabilities to the container, in particular not to those making use of user
 372    namespaces. The kernel's audit subsystem is still not virtualized for
 373    containers, and passing these credentials is pointless hence, given the
 374    actual attempt to make use of the audit subsystem will fail. Note that
 375    systemd's audit support is partially conditioned on these capabilities, thus
 376    by dropping them you ensure that you get an entirely clean boot, as systemd
 377    will make no attempt to use it. If you pass the capabilities to the payload
 378    systemd will assume that audit is available and works, and some components
 379    will subsequently fail in various ways. Note that once the kernel learnt
 380    native support for container-virtualized audit, adding the capability to the
 381    container description will automatically make the container payload use it.
 382
 383 ## Fully Unprivileged Container Payload
 384
 385 First things first, to make this clear: Linux containers are not a security
 386 technology right now. There are more holes in the model than in swiss cheese.
 387
 388 For example: if you do not use user namespacing, and share root and other users
 389 between container and host, the `struct user` structures will be shared between
 390 host and container, and hence `RLIMIT_NPROC` and so of the container users
 391 affect the host and other containers, and vice versa. This is a major security
 392 hole, and actually is a real-life problem: since Avahi sets `RLIMIT_NPROC` of
 393 its user to 2 (to effectively disallow `fork()`ing) you cannot run more than
 394 one Avahi instance on the entire system...
 395
 396 People have been asking to be able to run systemd without `CAP_SYS_ADMIN` and
 397 `CAP_SYS_MKNOD` in the container. This is now supported to some level in
 398 systemd, but we recommend against it (see above). If `CAP_SYS_ADMIN` and
 399 `CAP_SYS_MKNOD` are missing from the container systemd will now gracefully turn
 400 off `PrivateTmp=`, `PrivateNetwork=`, `ProtectHome=`, `ProtectSystem=` and
 401 others, because those capabilities are required to implement these options. The
 402 services using these settings (which include many of systemd's own) will hence
 403 run in a different, less secure environment when the capabilities are missing
 404 than with them around.
 405
 406 With user namespacing in place things get much better. With user namespaces the
 407 `struct user` issue described above goes away, and containers can keep
 408 `CAP_SYS_ADMIN` safely for the user namespace, as capabilities are virtualized
 409 and having capabilities inside a container doesn't mean one also has them
 410 outside.
 411
 412 ## Final Words
 413
 414 If you write software that wants to detect whether it is run in a container,
 415 please check `/proc/1/environ` and look for the `container=` environment
 416 variable. Do not assume the environment variable is inherited down the process
 417 tree. It generally is not. Hence check the environment block of PID 1, not your
 418 own. Note though that this file is only accessible to root. systemd hence early
 419 on also copies the value into `/run/systemd/container`, which is readable for
 420 everybody. However, that's a systemd-specific interface and other init systems
 421 are unlikely to do the same.
 422
 423 Note that it is our intention to make systemd systems work flawlessly and
 424 out-of-the-box in containers. In fact, we are interested to ensure that the same
 425 OS image can be booted on a bare system, in a VM and in a container, and behave
 426 correctly each time. If you notice that some component in systemd does not work
 427 in a container as it should, even though the container manager implements
 428 everything documented above, please contact us.