## Two Key Design Rules
Much of the philosophy behind these concepts is based on a couple of basic
-design ideas of cgroup v2 (which we however try to adapt as far as we can to
-cgroup v1 too). Specifically two cgroup v2 rules are the most relevant:
+design ideas of cgroup v2. Specifically two cgroup v2 rules are the most relevant:
1. The **no-processes-in-inner-nodes** rule: this means that it's not permitted
to have processes directly attached to a cgroup that also has child cgroups and
be in constant pain as various pieces of software will fight over cgroup
ownership.
-Note that cgroup v1 is currently the most deployed implementation, even though
-it's semantically broken in many ways, and in many cases doesn't actually do
-what people think it does. cgroup v2 is where things are going, and most new
-kernel features in this area are only added to cgroup v2, and not cgroup v1
+Note that cgroup v1 is semantically broken in many ways, and in many cases doesn't
+actually do what people think it does. cgroup v2 is where things are going, and
+new kernel features in this area are only added to cgroup v2, and not cgroup v1
anymore. For example, cgroup v2 provides proper cgroup-empty notifications, has
support for all kinds of per-cgroup BPF magic, supports secure delegation of
cgroup trees to less privileged processes and so on, which all are not
available on cgroup v1.
-## Three Different Tree Setups 🌳
+## Hierarchy and Controller Support
-systemd supports three different modes how cgroups are set up. Specifically:
+systemd supports what's called **unified** hierarchy, which is a simply mode
+that exposes a pure cgroup v2 logic. In this mode `/sys/fs/cgroup/` is the only
+mounted cgroup API file system and all available controllers are exclusively
+exposed through it.
-1. **Unified** — this is the simplest mode, and exposes a pure cgroup v2
-logic. In this mode `/sys/fs/cgroup` is the only mounted cgroup API file system
-and all available controllers are exclusively exposed through it.
+systemd supports a number of controllers (but not all). Specifically, supported
+are: `cpu`, `io`, `memory`, `pids`. It is our intention to natively support all
+cgroup v2 controllers as they are added to the kernel.
+
+The following hierarchies were deprecated in v256, and eventually got removed
+in v258:
-2. **Legacy** — this is the traditional cgroup v1 mode. In this mode the
+* **Legacy** — this is the traditional cgroup v1 mode. In this mode the
various controllers each get their own cgroup file system mounted to
`/sys/fs/cgroup/<controller>/`. On top of that systemd manages its own cgroup
hierarchy for managing purposes as `/sys/fs/cgroup/systemd/`.
-3. **Hybrid** — this is a hybrid between the unified and legacy mode. It's set
+* **Hybrid** — this is a hybrid between the unified and legacy mode. It's set
up mostly like legacy, except that there's also an additional hierarchy
`/sys/fs/cgroup/unified/` that contains the cgroup v2 hierarchy. (Note that in
this mode the unified hierarchy won't have controllers attached, the
too. This mode is a stopgap. Don't bother with this too much unless you have
too much free time.
-To say this clearly, legacy and hybrid modes have no future. If you develop
-software today and don't focus on the unified mode, then you are writing
-software for yesterday, not tomorrow. They are primarily supported for
-compatibility reasons and will not receive new features. Sorry.
-
-Superficially, in legacy and hybrid modes it might appear that the parallel
-cgroup hierarchies for each controller are orthogonal from each other. In
-systemd they are not: the hierarchies of all controllers are always kept in
-sync (at least mostly: sub-trees might be suppressed in certain hierarchies if
-no controller usage is required for them). The fact that systemd keeps these
-hierarchies in sync means that the legacy and hybrid hierarchies are
-conceptually very close to the unified hierarchy. In particular this allows us
-to talk of one specific cgroup and actually mean the same cgroup in all
-available controller hierarchies. E.g. if we talk about the cgroup `/foo/bar/`
-then we actually mean `/sys/fs/cgroup/cpu/foo/bar/` as well as
-`/sys/fs/cgroup/memory/foo/bar/`, `/sys/fs/cgroup/pids/foo/bar/`, and so on.
-Note that in cgroup v2 the controller hierarchies aren't orthogonal, hence
-thinking about them as orthogonal won't help you in the long run anyway.
-
-If you wonder how to detect which of these three modes is currently used, use
-`statfs()` on `/sys/fs/cgroup/`. If it reports `CGROUP2_SUPER_MAGIC` in its
-`.f_type` field, then you are in unified mode. If it reports `TMPFS_MAGIC` then
-you are either in legacy or hybrid mode. To distinguish these two cases, run
-`statfs()` again on `/sys/fs/cgroup/unified/`. If that succeeds and reports
-`CGROUP2_SUPER_MAGIC` you are in hybrid mode, otherwise not.
-From a shell, you can check the `Type` in `stat -f /sys/fs/cgroup` and
-`stat -f /sys/fs/cgroup/unified`.
-
## systemd's Unit Types
The low-level kernel cgroups feature is exposed in systemd in three different
Container managers and suchlike often want to control cgroups directly using
the raw kernel APIs. That's entirely fine and supported, as long as proper
-*delegation* is followed. Delegation is a concept we inherited from cgroup v2,
-but we expose it on cgroup v1 too. Delegation means that some parts of the
-cgroup tree may be managed by different managers than others. As long as it is
-clear which manager manages which part of the tree each one can do within its
-sub-graph of the tree whatever it wants.
+*delegation* is followed. Delegation is a concept we inherited from cgroup v2.
+It means that some parts of the cgroup tree may be managed by different managers
+than others. As long as it is clear which manager manages which part of the tree
+each one can do within its sub-graph of the tree whatever it wants.
Only sub-trees can be delegated (though whoever decides to request a sub-tree
can delegate sub-sub-trees further to somebody else if they like). Delegation
2. If your service makes use of the `User=` functionality, then the sub-tree
will be `chown()`ed to the indicated user so that it can correctly create
- cgroups below it. Note however that systemd will do that only in the unified
- hierarchy (in unified and hybrid mode) as well as on systemd's own private
- hierarchy (in legacy and hybrid mode). It won't pass ownership of the legacy
- controller hierarchies. Delegation to less privileged processes is not safe
- in cgroup v1 (as a limitation of the kernel), hence systemd won't facilitate
- access to it.
+ cgroups below it.
3. Any BPF IP filter programs systemd installs will be installed with
`BPF_F_ALLOW_MULTI` so that your program can install additional ones.
maybe if you dislike D-Bus and systemd that much, the better approach might be
to work on that, and widen your horizon a bit. You are welcome.
-## Controller Support
-
-systemd supports a number of controllers (but not all). Specifically, supported
-are:
-
-* on cgroup v1: `cpu`, `cpuacct`, `blkio`, `memory`, `devices`, `pids`
-* on cgroup v2: `cpu`, `io`, `memory`, `pids`
-
-It is our intention to natively support all cgroup v2 controllers as they are
-added to the kernel. However, regarding cgroup v1: at this point we will not
-add support for any other controllers anymore. This means systemd currently
-does not and will never manage the following controllers on cgroup v1:
-`freezer`, `cpuset`, `net_cls`, `perf_event`, `net_prio`, `hugetlb`. Why not?
-Depending on the case, either their API semantics or implementations aren't
-really usable, or it's very clear they have no future on cgroup v2, and we
-won't add new code for stuff that clearly has no future.
-
-Effectively this means that all those mentioned cgroup v1 controllers are up
-for grabs: systemd won't manage them, and hence won't delegate them to your
-code (however, systemd will still mount their hierarchies, simply because it
-mounts all controller hierarchies it finds available in the kernel). If you
-decide to use them, then that's fine, but systemd won't help you with it (but
-also not interfere with it). To be nice to other tenants it might be wise to
-replicate the cgroup hierarchies of the other controllers in them too however,
-but of course that's between you and those other tenants, and systemd won't
-care. Replicating the cgroup hierarchies in those unsupported controllers would
-mean replicating the full cgroup paths in them, and hence the prefixing
-`.slice` components too, otherwise the hierarchies will start being orthogonal
-after all, and that's not really desirable. One more thing: systemd will clean
-up after you in the hierarchies it manages: if your daemon goes down, its
-cgroups will be removed too. You basically get the guarantee that you start
-with a pristine cgroup sub-tree for your service or scope whenever it is
-started. This is not the case however in the hierarchies systemd doesn't
-manage. This means that your programs should be ready to deal with left-over
-cgroups in them — from previous runs, and be extra careful with them as they
-might still carry settings that might not be valid anymore.
-
-Note a particular asymmetry here: if your systemd version doesn't support a
-specific controller on cgroup v1 you can still make use of it for delegation,
-by directly fiddling with its hierarchy and replicating the cgroup tree there
-as necessary (as suggested above). However, on cgroup v2 this is different:
-separately mounted hierarchies are not available, and delegation has always to
-happen through systemd itself. This means: when you update your kernel and it
-adds a new, so far unseen controller, and you want to use it for delegation,
-then you also need to update systemd to a version that groks it.
-
## systemd as Container Payload
systemd can happily run as a container payload's PID 1. Note that systemd
to get the cgroup for a unit. The method `GetUnitByControlGroup()` may be
used to get the unit for a cgroup.)
-6. âš¡ Think twice before delegating cgroup v1 controllers to less privileged
- containers. It's not safe, you basically allow your containers to freeze the
- system with that and worse. Delegation is a strongpoint of cgroup v2 though,
- and there it's safe to treat delegation boundaries as privilege boundaries.
-
And that's it for now. If you have further questions, refer to the systemd
mailing list.
for `/dev/null`, `/dev/zero`, `/dev/full`, `/dev/random`, `/dev/urandom`,
`/dev/tty`, `/dev/ptmx` in `/dev/`. It is not necessary to create `/dev/fd`
or `/dev/stdout`, as systemd will do that on its own. Make sure to set up a
- `BPF_PROG_TYPE_CGROUP_DEVICE` BPF program — on cgroupv2 — or the `devices`
- cgroup controller — on cgroupv1 — so that no other devices but these may be
- created in the container. Note that many systemd services use
+ `BPF_PROG_TYPE_CGROUP_DEVICE` BPF program on cgroupv2, so that no other devices
+ but these may be created in the container. Note that many systemd services use
`PrivateDevices=`, which means that systemd will set up a private `/dev/`
for them for which it needs to be able to create these device nodes.
Dropping `CAP_MKNOD` for containers is hence generally not advisable, but
root-level cgroup directories tend to be quite different from inner
directories, and that distinction matters. It is OK however, to mount the
"upper" parts read-only of the hierarchies, and only allow write-access to
- the cgroup sub-tree the container runs in. It's also a good idea to mount
- all controller hierarchies with exception of `name=systemd` fully read-only
- (this only applies to cgroupv1, of course), to protect the controllers from
- alteration from inside the containers. Or to turn this around: only the
- cgroup sub-tree of the container itself (on cgroupv2 in the unified
- hierarchy, and on cgroupv1 in the `name=systemd` hierarchy) may be writable
- to the container.
+ the cgroup sub-tree the container runs in.
7. Create the control group root of your container by either running your
container as a service (in case you have one container manager instance per
Starting with version 205 systemd provides a number of interfaces that may be used to create and manage labelled groups of processes for the purpose of monitoring and controlling them and their resource usage.
This is built on top of the Linux kernel Control Groups ("cgroups") facility.
-Previously, the kernel's cgroups API was exposed directly as shared application API, following the rules of the [Pax Control Groups](/PAX_CONTROL_GROUPS) document.
-However, the kernel cgroup interface has been reworked into an API that requires that each individual cgroup is managed by a single writer only.
+Previously, the kernel's cgroups API (cgroup v1) was exposed directly as shared application API. That, however, turned out to be undesirable and semantically broken to manage.
+The superseding cgroup v2 interface therefore requires that each individual cgroup is managed by a single writer only.
With this change the main cgroup tree becomes private property of that userspace component and is no longer a shared resource.
On systemd systems use the systemd APIs as described below. At this time we are not aware of any component that would take the cgroup managing role on Upstart/sysvinit systems, so we cannot help you with this. Sorry.
-### What's the timeframe of this? Do I need to care now?
-
-In the short-term future writing directly to the control group tree from applications should still be OK, as long as the [Pax Control Groups](/PAX_CONTROL_GROUPS) document is followed. In the medium-term future it will still be supported to alter/read individual attributes of cgroups directly, but no longer to create/delete cgroups without using the systemd API. In the longer-term future altering/reading attributes will also be unavailable to userspace applications, unless done via systemd's APIs (either D-Bus based IPC APIs or shared library APIs for _passive_ operations).
-
-It is recommended to use the new systemd APIs described below in any case. Note that the kernel cgroup interface is currently being reworked (available when the "sane_behaviour" kernel option is used). This will change the cgroupfs interface. By using systemd's APIs this change is abstracted away and invisible to applications.
-
## systemd's Resource Control Concepts
Systemd provides three unit types that are useful for the purpose of resource control:
+++ /dev/null
----
-title: Pax Controla Groupiana
-category: Users, Groups and Home Directories
-layout: default
-SPDX-License-Identifier: LGPL-2.1-or-later
----
-
-# Pax Controla Groupiana
-
-_aka "How to behave nicely in the cgroupfs trees"_
-
-**Important Update: Please consult this document only as a historical reference.
-It was written under the assumption that the cgroups tree was a shared resource.
-However, after much discussion this concept has been deemed outdated.
-The cgroups tree can no longer be considered a shared resource.
-Instead, a management daemon of some kind needs to arbitrate access to it, and it needs to actively propagate changes between the entities it manages.
-More specifically, on systemd systems this management daemon is systemd itself, accessible via a number of bus APIs.
-This means instead of dealing directly with the low-level interfaces of the cgroup file system, please use systemd's high-level APIs as a replacement, see the
-[New Control Group Interfaces](/CONTROL_GROUP_INTERFACE)
-for details. They offer similar functionality.**
-
-Are you writing an application interfacing with the cgroups tree?
-The cgroups trees are a shared resource, other applications will use them too.
-Here are a few recommendations how to write your application in a way that minimizes conflicts with other applications.
-If you follow these guidelines applications should not step on any other application's toes and users will be happy.
-
-Before you read these recommendations please make sure you understand cgroups thoroughly,
-and specifically are aware what a controller is, what a named hierarchy is and so on.
-
-## Intended Audience
-
-You should consider these recommendations if you are you working on one of the following:
-
-- You write a system or session manager based on cgroups (like systemd)
-- You write a VM manager based on cgroups (like libvirt)
-- You write a terminal application and want to place every shell in a separate cgroup (like gnome-terminal)
-- You write a web browser and want to place every renderer in a separate cgroup (like Firefox or Chrome)
-- You create a container for some purpose (such as systemd-nspawn)
-- Or you use cgroups for any other purpose and want things to work nicely with other applications.
-
-## General Recommendations
-
-- If you use one of the kernel controllers, do _not_ assume you are the only one who uses them.
- Other programs may manipulate the tree, add cgroups and change group attributes at any time, and they will not inform you about it.
- The kernel provided controller hierarchies are a shared resource, so be nice.
-- If you use a generic named hierarchy with no controller attached, then you may assume it's yours and only yours, and that no other programs interfere with it.
-- If you use a generic named hierarchy with no controller attached, then make sure to name it after your project in order to minimize namespacing conflicts.
- A hierarchy named "name=web" is a bit generic.
- A hierarchy named "name=apache" a much better choice, if you are an Apache developer and need an entire hierarchy all for yourself.
-- Do _not_ assume everybody uses the same library to manipulate the cgroups tree as you are.
- In fact most likely most applications and the user himself will manipulate the tree without any further indirection (i.e. will use naked system calls/shell commands)
-- Never create cgroups at the top of the tree (i.e. with an absolute path).
- If possible find the cgroup your own process was started in and create subgroups only below that group (read /proc/self/cgroup to find it).
- If that's not applicable, then at least place yourself below the cgroup path of PID 1 (read /proc/1/cgroup to find it).
- This is important to ensure that containers work properly (the cgroupfs tree is currently not virtualized for containers!), and solves permission problems, and makes the whole system nicely stackable.
-- A corollary of this: If you spawn subprocesses expect that they will create subcgroups.
- That means when terminating there might be subcgroups below the ones you created and you hence need to recursively remove them too.
- In fact, many of your operations must probably be executed in a recursive fashion.
-- Do not play permission games: if you are an unprivileged user application then it's _not_ your business to ensure you have the right permissions
- (i.e. do not include any setuid code in your app to create groups).
- Instead your system manager (such as systemd),
- should provide you with the right set of permissions on the cgroup you are running in to create subgroups.
- Normally that should mean that depending on administrator configuration, you will or will not get access to create subgroups under the cgroup you are running in and the ability to add PIDs to it.
- If you don't get access to these hierarchies then this might be a decision by the administrator and you should do your best to go on, and fail gracefully.
-- If you create a cgroup, then you are in charge of removing it too after using it.
- Do not remove other program's cgroups.
- Special exception: in some cases it is OK to pre-set attributes on certain cgroups that are primarily managed by another program.
- (Example: in systemd we are fine if you externally pre-create or manipulate service cgroups, for example to make changes to some attributes you cannot control with systemd natively, see below).
- In that case: create the cgroup and set the sticky bit (+t) on the tasks file in it.
- This will then be used as an indication to the primary manager of the group not to remove the cgroup at the end, in order to avoid that your settings are lost.
- This is of course a bit of a misuse of the sticky bit, but given that it serves no other purpose on Linux for normal files, it is an OK use, with a fitting meaning given the name of "sticky bit".
-- If you find a process in a cgroup you are about to remove, and it is not yours, consider leaving the cgroup around.
- I.e. if rmdir returns EEMPTY, ignore this.
-- The cgroup mount point for a specific hierarchy is /sys/fs/cgroup/$CONTROLLER/.
- (Example: /sys/fs/cgroup/cpu for the "cpu" controller).
- In your application you are welcome to rely on these standardized mount points,
- and it is not necessary to dynamically determine the current mount point via /proc/self/mountinfo (but if you do, that's of course fine, too).
- Note that /sys/fs/cgroup/$CONTROLLER/ might actually just be a symlink to some other mount point (see below).
-- If multiple controllers are mounted into the same hierarchy, it is guaranteed that symlinks exist to make sure all jointly mounted controllers are still available under /sys/fs/cgroup/$CONTROLLER/.
- Example: if "cpu" and "cpuacct" are mounted together, then symlinks /sys/fs/cgroup/cpu and /sys/fs/cgroup/cpuacct will point to the joint mountpoint (which could be something like /sys/fs/cgroup/cpu+cpuacct).
-- Your application should not mount the cgroup controller file systems (unless it is your own private named hierarchy).
- This is exclusively a job for the system manager or a system-wide init script such as cgconfig.
- If you work on a system manager or such an init script you must mount the cgroup controllers to /sys/fs/cgroup/$CONTROLLER/ or provide compatibility symlinks.
-- It's a good idea not to fail if a cgroup already exists when you try to create it.
- Ignore EEXIST on mkdir.
-- Avoid renaming cgroups or similar fancier file operations.
-- Expect that other programs might readjust the attributes on your cgroups dynamically during runtime.
-- When creating a cgroup pick a nice a descriptive name that is guessable and no surprise to the admin.
- The admin will thank you for this if he has to read the output of "ps -eo pid,args,cgroups"
-- /sys/fs/cgroup is a tmpfs. If you create your own private named hierarchy then you are welcome to mount it into a subdirectory of this directory.
- This minimizes surprises for the user.
-- /sys/fs/cgroup is a tmpfs, but it's only intended use is to act as place where control group hierarchies can be mounted or symlinked to.
- You should not place any other kind of file in this directory.
- The same way as /dev/shm is for POSIX shared memory segments only -- and nothing else -- this directory is for cgroup hierarchies only.
- Just because something is a tmpfs it doesn't mean you can actually use it for "temporary" files, thank you.
-- Avoid creating orthogonal hierarchies in the various kernel controller hierarchies.
- Please make sure that the controllers contain the same hierarchy or subsets of each other.
-
-## Cooperation with systemd
-
-systemd adheres to the recommendations above and guarantees additional behavior which might be useful for writing applications that cooperate with systemd on cgroup management:
-
-- If a service cgroup already exists, systemd will make use of it and not recreate it.
- (If +t is set on the tasks file it will not remove it when stopping a service, otherwise it will, see above).
- It is hence OK to pre-create cgroups and then let systemd use it, without having systemd remove it afterwards.
-- If a service cgroup already exists, systemd will not override the attributes of the cgroup with the exception of those explicitly configured in the systemd unit files.
- It is hence OK to pre-create cgroups for use in systemd, and pre-apply attributes to it.
-- To avoid that systemd places all services in automatic cgroups in the "cpu" hierarchy change the DefaultControllers= in /etc/systemd/system.conf and set it to the empty string.
-- By default systemd will place services only in automatic cgroups in the "cpu" hierarchy and in its own private tree "name=systemd".
- If you want it to duplicate these trees in other hierarchies add them to DefaultControllers= in /etc/systemd/system.conf
-- To opt-out or opt-in specific services from the automatic tree generation in the kernel controller hierarchies use ControlGroup= in the unit file.
- Use "ControlGroup=cpu:/" to opt-out of cgroup assignment for a service or "ControlGroup=cpu:/foo/bar" to manipulate the cgroup path.
-- Stay away from the name=systemd named hierarchy.
- It's private property of systemd.
- You are welcome to explore it, but it is uncool to modify it from outside systemd.
-Thanks.
| [Journal File Format](/JOURNAL_FILE_FORMAT) | File format | yes | yes | - | maybe | - | no |
| [Journal Export Format](JOURNAL_EXPORT_FORMATS#journal-export-format) | File format | yes | yes | - | yes | - | no |
| [Journal JSON Format](JOURNAL_EXPORT_FORMATS#journal-json-format) | File format | yes | yes | - | yes | - | no |
-| [Cooperation in cgroup tree](/PAX_CONTROL_GROUPS) | Treaty | yes | yes | libvirt | yes | libvirt | no |
| [Password Agents](/PASSWORD_AGENTS) | Socket+Files | yes | yes | - | yes | - | no |
| [udev multi-seat properties](https://www.freedesktop.org/software/systemd/man/sd-login.html) | udev Property | yes | yes | X11, gdm | no | - | no |
| udev session switch ACL properties | udev Property | no | no | - | no | - | no |
projects / thirdparty / systemd.git / commitdiff
