docs/CONTAINER_INTERFACE.md

   1 ---
   2 title: Container Interface
   3 category: Interfaces
   4 layout: default
   5 SPDX-License-Identifier: LGPL-2.1-or-later
   6 ---
   7
   8 # The Container Interface
   9
  10 Also consult [Writing Virtual Machine or Container
  11 Managers](http://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers).
  12
  13 systemd has a number of interfaces for interacting with container managers,
  14 when systemd is used inside of an OS container. If you work on a container
  15 manager, please consider supporting the following interfaces.
  16
  17 ## Execution Environment
  18
  19 1. If the container manager wants to control the hostname for a container
  20    running systemd it may just set it before invoking systemd, and systemd will
  21    leave it unmodified when there is no hostname configured in `/etc/hostname`
  22    (that file overrides whatever is pre-initialized by the container manager).
  23
  24 2. Make sure to pre-mount `/proc/`, `/sys/`, and `/sys/fs/selinux/` before
  25    invoking systemd, and mount `/proc/sys/`, `/sys/`, and `/sys/fs/selinux/`
  26    read-only in order to prevent the container from altering the host kernel's
  27    configuration settings. (As a special exception, if your container has
  28    network namespaces enabled, feel free to make `/proc/sys/net/` writable).
  29    systemd and various other subsystems (such as the SELinux userspace) have
  30    been modified to behave accordingly when these file systems are read-only.
  31    (It's OK to mount `/sys/` as `tmpfs` btw, and only mount a subset of its
  32    sub-trees from the real `sysfs` to hide `/sys/firmware/`, `/sys/kernel/` and
  33    so on. If you do that, make sure to mark `/sys/` read-only, as that
  34    condition is what systemd looks for, and is what is considered to be the API
  35    in this context.)
  36
  37 3. Pre-mount `/dev/` as (container private) `tmpfs` for the container and bind
  38    mount some suitable TTY to `/dev/console`. Also, make sure to create device
  39    nodes for `/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`,
  40    `/dev/urandom`, `/dev/tty`, `/dev/ptmx` in `/dev/`. It is not necessary to
  41    create `/dev/fd` or `/dev/stdout`, as systemd will do that on its own. Make
  42    sure to set up a `BPF_PROG_TYPE_CGROUP_DEVICE` BPF program — on cgroupv2 —
  43    or the `devices` cgroup controller — on cgroupv1 — so that no other devices
  44    but these may be created in the container. Note that many systemd services
  45    use `PrivateDevices=`, which means that systemd will set up a private
  46    `/dev/` for them for which it needs to be able to create these device nodes.
  47    Dropping `CAP_MKNOD` for containers is hence generally not advisable, but
  48    see below.
  49
  50 4. `systemd-udevd` is not available in containers (and refuses to start), and
  51    hence device dependencies are unavailable. The `systemd-udevd` unit files
  52    will check for `/sys/` being read-only, as an indication whether device
  53    management can work. Therefore make sure to mount `/sys/` read-only in the
  54    container (see above). Various clients of `systemd-udevd` also check the
  55    read-only state of `/sys/`, including PID 1 itself and `systemd-networkd`.
  56
  57 5. If systemd detects it is run in a container it will spawn a single shell on
  58    `/dev/console`, and not care about VTs or multiple gettys on VTs. (But see
  59    `$container_ttys` below.)
  60
  61 6. Either pre-mount all cgroup hierarchies in full into the container, or leave
  62    that to systemd which will do so if they are missing. Note that it is
  63    explicitly *not* OK to just mount a sub-hierarchy into the container as that
  64    is incompatible with `/proc/$PID/cgroup` (which lists full paths). Also the
  65    root-level cgroup directories tend to be quite different from inner
  66    directories, and that distinction matters. It is OK however, to mount the
  67    "upper" parts read-only of the hierarchies, and only allow write-access to
  68    the cgroup sub-tree the container runs in. It's also a good idea to mount
  69    all controller hierarchies with exception of `name=systemd` fully read-only
  70    (this only applies to cgroupv1, of course), to protect the controllers from
  71    alteration from inside the containers. Or to turn this around: only the
  72    cgroup sub-tree of the container itself (on cgroupv2 in the unified
  73    hierarchy, and on cgroupv1 in the `name=systemd` hierarchy) may be writable
  74    to the container.
  75
  76 7. Create the control group root of your container by either running your
  77    container as a service (in case you have one container manager instance per
  78    container instance) or creating one scope unit for each container instance
  79    via systemd's transient unit API (in case you have one container manager
  80    that manages all instances. Either way, make sure to set `Delegate=yes` in
  81    it. This ensures that the unit you created will be part of all cgroup
  82    controllers (or at least the ones systemd understands). The latter may also
  83    be done via `systemd-machined`'s `CreateMachine()` API. Make sure to use the
  84    cgroup path systemd put your process in for all operations of the container.
  85    Do not add new cgroup directories to the top of the tree. This will not only
  86    confuse systemd and the admin, but also prevent your implementation from
  87    being "stackable".
  88
  89 ## Environment Variables
  90
  91 1. To allow systemd (and other programs) to identify that it is executed within
  92    a container, please set the `$container` environment variable for PID 1 in
  93    the container to a short lowercase string identifying your
  94    implementation. With this in place the `ConditionVirtualization=` setting in
  95    unit files will work properly. Example: `container=lxc-libvirt`
  96
  97 2. systemd has special support for allowing container managers to initialize
  98    the UUID for `/etc/machine-id` to some manager supplied value. This is only
  99    enabled if `/etc/machine-id` is empty (i.e. not yet set) at boot time of the
 100    container. The container manager should set `$container_uuid` as environment
 101    variable for the container's PID 1 to the container UUID. (This is similar
 102    to the effect of `qemu`'s `-uuid` switch). Note that you should pass only a
 103    UUID here that is actually unique (i.e. only one running container should
 104    have a specific UUID), and gets changed when a container gets duplicated.
 105    Also note that systemd will try to persistently store the UUID in
 106    `/etc/machine-id` (if writable) when this option is used, hence you should
 107    always pass the same UUID here. Keeping the externally used UUID for a
 108    container and the internal one in sync is hopefully useful to minimize
 109    surprise for the administrator.
 110
 111 3. systemd can automatically spawn login gettys on additional ptys. A container
 112    manager can set the `$container_ttys` environment variable for the
 113    container's PID 1 to tell it on which ptys to spawn gettys. The variable
 114    should take a space separated list of pty names, without the leading `/dev/`
 115    prefix, but with the `pts/` prefix included. Note that despite the
 116    variable's name you may only specify ptys, and not other types of ttys. Also
 117    you need to specify the pty itself, a symlink will not suffice. This is
 118    implemented in
 119    [systemd-getty-generator(8)](https://www.freedesktop.org/software/systemd/man/systemd-getty-generator.html).
 120    Note that this variable should not include the pty that `/dev/console` maps
 121    to if it maps to one (see below). Example: if the container receives
 122    `container_ttys=pts/7 pts/8 pts/14` it will spawn three additional login
 123    gettys on ptys 7, 8, and 14.
 124
 125 4. To allow applications to detect the OS version and other metadata of the host
 126    running the container manager, if this is considered desirable, please parse
 127    the host's `/etc/os-release` and set a `$container_host_<key>=<VALUE>`
 128    environment variable for the ID fields described by the [os-release
 129    interface](https://www.freedesktop.org/software/systemd/man/os-release.html), eg:
 130    `$container_host_id=debian`
 131    `$container_host_build_id=2020-06-15`
 132    `$container_host_variant_id=server`
 133    `$container_host_version_id=10`
 134
 135 5. systemd supports passing immutable binary data blobs with limited size and
 136    restricted access to services via the `LoadCredential=` and `SetCredential=`
 137    settings. The same protocol may be used to pass credentials from the
 138    container manager to systemd itself. The credential data should be placed in
 139    some location (ideally a read-only and non-swappable file system, like
 140    'ramfs'), and the absolute path to this directory exported in the
 141    `$CREDENTIALS_DIRECTORY` environment variable. If the container managers
 142    does this, the credentials passed to the service manager can be propagated
 143    to services via `LoadCredential=` (see ...). The container manager can
 144    choose any path, but `/run/host/credentials` is recommended.
 145
 146 ## Advanced Integration
 147
 148 1. Consider syncing `/etc/localtime` from the host file system into the
 149    container. Make it a relative symlink to the containers's zoneinfo dir, as
 150    usual. Tools rely on being able to determine the timezone setting from the
 151    symlink value, and making it relative looks nice even if people list the
 152    container's `/etc/` from the host.
 153
 154 2. Make the container journal available in the host, by automatically
 155    symlinking the container journal directory into the host journal directory.
 156    More precisely, link `/var/log/journal/<container-machine-id>` of the
 157    container into the same dir of the host. Administrators can then
 158    automatically browse all container journals (correctly interleaved) by
 159    issuing `journalctl -m`. The container machine ID can be determined from
 160    `/etc/machine-id` in the container.
 161
 162 3. If the container manager wants to cleanly shutdown the container, it might
 163    be a good idea to send `SIGRTMIN+3` to its init process. systemd will then
 164    do a clean shutdown. Note however, that since only systemd understands
 165    `SIGRTMIN+3` like this, this might confuse other init systems.
 166
 167 4. To support [Socket Activated
 168    Containers](http://0pointer.de/blog/projects/socket-activated-containers.html)
 169    the container manager should be capable of being run as a systemd
 170    service. It will then receive the sockets starting with FD 3, the number of
 171    passed FDs in `$LISTEN_FDS` and its PID as `$LISTEN_PID`. It should take
 172    these and pass them on to the container's init process, also setting
 173    $LISTEN_FDS and `$LISTEN_PID` (basically, it can just leave the FDs and
 174    `$LISTEN_FDS` untouched, but it needs to adjust `$LISTEN_PID` to the
 175    container init process). That's all that's necessary to make socket
 176    activation work. The protocol to hand sockets from systemd to services is
 177    hence the same as from the container manager to the container systemd. For
 178    further details see the explanations of
 179    [sd_listen_fds(1)](http://0pointer.de/public/systemd-man/sd_listen_fds.html)
 180    and the [blog story for service
 181    developers](http://0pointer.de/blog/projects/socket-activation.html).
 182
 183 5. Container managers should stay away from the cgroup hierarchy outside of the
 184    unit they created for their container. That's private property of systemd,
 185    and no other code should modify it.
 186
 187 6. systemd running inside the container can report when boot-up is complete
 188    using the usual `sd_notify()` protocol that is also used when a service
 189    wants to tell the service manager about readiness. A container manager can
 190    set the `$NOTIFY_SOCKET` environment variable to a suitable socket path to
 191    make use of this functionality. (Also see information about
 192    `/run/host/notify` below.)
 193
 194 ## Networking
 195
 196 1. Inside of a container, if a `veth` link is named `host0`, `systemd-networkd`
 197    running inside of the container will by default run DHCPv4, DHCPv6, and
 198    IPv4LL clients on it. It is thus recommended that container managers that
 199    add a `veth` link to a container name it `host0`, to get an automatically
 200    configured network, with no manual setup.
 201
 202 2. Outside of a container, if a `veth` link is prefixed "ve-", `systemd-networkd`
 203    will by default run DHCPv4 and DHCPv6 servers on it, as well as IPv4LL. It
 204    is thus recommended that container managers that add a `veth` link to a
 205    container name the external side `ve-` + the container name.
 206
 207 3. It is recommended to configure stable MAC addresses for container `veth`
 208    devices, for example hashed out of the container names. That way it is more
 209    likely that DHCP and IPv4LL will acquire stable addresses.
 210
 211 ## The `/run/host/` Hierarchy
 212
 213 Container managers may place certain resources the manager wants to provide to
 214 the container payload below the `/run/host/` hierarchy. This hierarchy should
 215 be mostly immutable (possibly some subdirs might be writable, but the top-level
 216 hierarchy — and probably most subdirs should be read-only to the
 217 container). Note that this hierarchy is used by various container managers, and
 218 care should be taken to avoid naming conflicts. `systemd` (and in particular
 219 `systemd-nspawn`) use the hierarchy for the following resources:
 220
 221 1. The `/run/host/incoming/` directory mount point is configured for `MS_SLAVE`
 222    mount propagation with the host, and is used as intermediary location for
 223    mounts to establish in the container, for the implementation of `machinectl
 224    bind`. Container payload should usually not directly interact with this
 225    directory: it's used by code outside the container to insert mounts inside
 226    it only, and is mostly an internal vehicle to achieve this. Other container
 227    managers that want to implement similar functionality might consider using
 228    the same directory.
 229
 230 2. The `/run/host/inaccessible/` directory may be set up by the container
 231    manager to include six file nodes: `reg`, `dir`, `fifo`, `sock`, `chr`,
 232    `blk`. These nodes correspond with the six types of file nodes Linux knows
 233    (with the exceptions of symlinks). Each node should be of the specific type
 234    and have an all zero access mode, i.e. be inaccessible. The two device node
 235    types should have major and minor of zero (which are unallocated devices on
 236    Linux). These nodes are used as mount source for implementing the
 237    `InaccessiblePath=` setting of unit files, i.e. file nodes to mask this way
 238    are overmounted with these "inaccessible" inodes, guaranteeing that the file
 239    node type does not change this way but the nodes still become
 240    inaccessible. Note that systemd when run as PID 1 in the container payload
 241    will create these nodes on its own if not passed in by the container
 242    manager. However, in that case it likely lacks the privileges to create the
 243    character and block devices nodes (there are fallbacks for this case).
 244
 245 3. The `/run/host/notify` path is a good choice to place the `sd_notify()`
 246    socket in, that may be used for the container's PID 1 to report to the
 247    container manager when boot-up is complete. The path used for this doesn't
 248    matter much as it is communicated via the `$NOTIFY_SOCKET` environment
 249    variable, following the usual protocol for this, however it's suitable, and
 250    recommended place for this socket in case ready notification is desired.
 251
 252 4. The `/run/host/os-release` file contains the `/etc/os-release` file of the
 253    host, i.e. may be used by the container payload to gather limited
 254    information about the host environment, on top of what `uname -a` reports.
 255
 256 5. The `/run/host/container-manager` file may be used to pass the same
 257    information as the `$container` environment variable (see above), i.e. a
 258    short string identifying the container manager implementation. This file
 259    should be newline terminated. Passing this information via this file has the
 260    benefit that payload code can easily access it, even when running
 261    unprivileged without access to the container PID1's environment block.
 262
 263 6. The `/run/host/container-uuid` file may be used to pass the same information
 264    as the `$container_uuid` environment variable (see above). This file should
 265    be newline terminated.
 266
 267 7. The `/run/host/credentials/` directory is a good place to pass credentials
 268    into the container, using the `$CREDENTIALS_DIRECTORY` protocol, see above.
 269
 270 ## What You Shouldn't Do
 271
 272 1. Do not drop `CAP_MKNOD` from the container. `PrivateDevices=` is a commonly
 273    used service setting that provides a service with its own, private, minimal
 274    version of `/dev/`. To set this up systemd in the container needs this
 275    capability. If you take away the capability than all services that set this
 276    flag will cease to work. Use `BPF_PROG_TYPE_CGROUP_DEVICE` BPF programs — on
 277    cgroupv2 — or the `devices` controller — on cgroupv1 — to restrict what
 278    device nodes the container can create instead of taking away the capability
 279    wholesale. (Also see the section about fully unprivileged containers below.)
 280
 281 2. Do not drop `CAP_SYS_ADMIN` from the container. A number of the most
 282    commonly used file system namespacing related settings, such as
 283    `PrivateDevices=`, `ProtectHome=`, `ProtectSystem=`, `MountFlags=`,
 284    `PrivateTmp=`, `ReadWriteDirectories=`, `ReadOnlyDirectories=`,
 285    `InaccessibleDirectories=`, and `MountFlags=` need to be able to open new
 286    mount namespaces and the mount certain file systems into them. You break all
 287    services that make use of these options if you drop the capability. Also
 288    note that logind mounts `XDG_RUNTIME_DIR` as `tmpfs` for all logged in users
 289    and that won't work either if you take away the capability. (Also see
 290    section about fully unprivileged containers below.)
 291
 292 3. Do not cross-link `/dev/kmsg` with `/dev/console`. They are different things,
 293    you cannot link them to each other.
 294
 295 4. Do not pretend that the real VTs are available in the container. The VT
 296    subsystem consists of all the devices `/dev/tty*`, `/dev/vcs*`, `/dev/vcsa*`
 297    plus their `sysfs` counterparts. They speak specific `ioctl()`s and
 298    understand specific escape sequences, that other ptys don't understand.
 299    Hence, it is explicitly not OK to mount a pty to `/dev/tty1`, `/dev/tty2`,
 300    `/dev/tty3`. This is explicitly not supported.
 301
 302 5. Don't pretend that passing arbitrary devices to containers could really work
 303    well. For example, do not pass device nodes for block devices to the
 304    container. Device access (with the exception of network devices) is not
 305    virtualized on Linux. Enumeration and probing of meta information from
 306    `/sys/` and elsewhere is not possible to do correctly in a container. Simply
 307    adding a specific device node to a container's `/dev/` is *not* *enough* to
 308    do the job, as `systemd-udevd` and suchlike are not available at all, and no
 309    devices will appear available or enumerable, inside the container.
 310
 311 6. Don't mount only a sub-tree of the `cgroupfs` into the container. This will not
 312    work as `/proc/$PID/cgroup` lists full paths and cannot be matched up with
 313    the actual `cgroupfs` tree visible, then. (You may "prune" some branches
 314    though, see above.)
 315
 316 7. Do not make `/sys/` writable in the container. If you do,
 317    `systemd-udevd.service` is started to manage your devices — inside the
 318    container, but that will cause conflicts and errors given that the Linux
 319    device model is not virtualized for containers on Linux and thus the
 320    containers and the host would try to manage the same devices, fighting for
 321    ownership. Multiple other subsystems of systemd similarly test for `/sys/`
 322    being writable to decide whether to use `systemd-udevd` or assume that
 323    device management is properly available on the instance. Among them
 324    `systemd-networkd` and `systemd-logind`. The conditionalization on the
 325    read-only state of `/sys/` enables a nice automatism: as soon as `/sys/` and
 326    the Linux device model are changed to be virtualized properly the container
 327    payload can make use of that, simply by marking `/sys/` writable. (Note that
 328    as special exception, the devices in `/sys/class/net/` are virtualized
 329    already, if network namespacing is used. Thus it is OK to mount the relevant
 330    sub-directories of `/sys/` writable, but make sure to leave the root of
 331    `/sys/` read-only.)
 332
 333 8. Do not pass the `CAP_AUDIT_CONTROL`, `CAP_AUDIT_READ`, `CAP_AUDIT_WRITE`
 334    capabilities to the container, in particular not to those making use of user
 335    namespaces. The kernel's audit subsystem is still not virtualized for
 336    containers, and passing these credentials is pointless hence, given the
 337    actual attempt to make use of the audit subsystem will fail. Note that
 338    systemd's audit support is partially conditioned on these capabilities, thus
 339    by dropping them you ensure that you get an entirely clean boot, as systemd
 340    will make no attempt to use it. If you pass the capabilities to the payload
 341    systemd will assume that audit is available and works, and some components
 342    will subsequently fail in various ways. Note that once the kernel learnt
 343    native support for container-virtualized audit, adding the capability to the
 344    container description will automatically make the container payload use it.
 345
 346 ## Fully Unprivileged Container Payload
 347
 348 First things first, to make this clear: Linux containers are not a security
 349 technology right now. There are more holes in the model than in swiss cheese.
 350
 351 For example: if you do not use user namespacing, and share root and other users
 352 between container and host, the `struct user` structures will be shared between
 353 host and container, and hence `RLIMIT_NPROC` and so of the container users
 354 affect the host and other containers, and vice versa. This is a major security
 355 hole, and actually is a real-life problem: since Avahi sets `RLIMIT_NPROC` of
 356 its user to 2 (to effectively disallow `fork()`ing) you cannot run more than
 357 one Avahi instance on the entire system...
 358
 359 People have been asking to be able to run systemd without `CAP_SYS_ADMIN` and
 360 `CAP_SYS_MKNOD` in the container. This is now supported to some level in
 361 systemd, but we recommend against it (see above). If `CAP_SYS_ADMIN` and
 362 `CAP_SYS_MKNOD` are missing from the container systemd will now gracefully turn
 363 off `PrivateTmp=`, `PrivateNetwork=`, `ProtectHome=`, `ProtectSystem=` and
 364 others, because those capabilities are required to implement these options. The
 365 services using these settings (which include many of systemd's own) will hence
 366 run in a different, less secure environment when the capabilities are missing
 367 than with them around.
 368
 369 With user namespacing in place things get much better. With user namespaces the
 370 `struct user` issue described above goes away, and containers can keep
 371 `CAP_SYS_ADMIN` safely for the user namespace, as capabilities are virtualized
 372 and having capabilities inside a container doesn't mean one also has them
 373 outside.
 374
 375 ## Final Words
 376
 377 If you write software that wants to detect whether it is run in a container,
 378 please check `/proc/1/environ` and look for the `container=` environment
 379 variable. Do not assume the environment variable is inherited down the process
 380 tree. It generally is not. Hence check the environment block of PID 1, not your
 381 own. Note though that this file is only accessible to root. systemd hence early
 382 on also copies the value into `/run/systemd/container`, which is readable for
 383 everybody. However, that's a systemd-specific interface and other init systems
 384 are unlikely to do the same.
 385
 386 Note that it is our intention to make systemd systems work flawlessly and
 387 out-of-the-box in containers. In fact we are interested to ensure that the same
 388 OS image can be booted on a bare system, in a VM and in a container, and behave
 389 correctly each time. If you notice that some component in systemd does not work
 390 in a container as it should, even though the container manager implements
 391 everything documented above, please contact us.