]>
Commit | Line | Data |
---|---|---|
c3e270f4 FB |
1 | --- |
2 | title: Control Group APIs and Delegation | |
3 | --- | |
4 | ||
e30eaff3 LP |
5 | # Control Group APIs and Delegation |
6 | ||
1e46eb59 LP |
7 | *Intended audience: hackers working on userspace subsystems that require direct |
8 | cgroup access, such as container managers and similar.* | |
9 | ||
e30eaff3 LP |
10 | So you are wondering about resource management with systemd, you know Linux |
11 | control groups (cgroups) a bit and are trying to integrate your software with | |
12 | what systemd has to offer there. Here's a bit of documentation about the | |
13 | concepts and interfaces involved with this. | |
14 | ||
15 | What's described here has been part of systemd and documented since v205 | |
5b24525a | 16 | times. However, it has been updated and improved substantially, even |
e30eaff3 LP |
17 | though the concepts stayed mostly the same. This is an attempt to provide more |
18 | comprehensive up-to-date information about all this, particular in light of the | |
19 | poor implementations of the components interfacing with systemd of current | |
20 | container managers. | |
21 | ||
22 | Before you read on, please make sure you read the low-level [kernel | |
23 | documentation about | |
4e1dfa45 | 24 | cgroup v2](https://www.kernel.org/doc/Documentation/cgroup-v2.txt). This |
e30eaff3 LP |
25 | documentation then adds in the higher-level view from systemd. |
26 | ||
27 | This document augments the existing documentation we already have: | |
28 | ||
29 | * [The New Control Group Interfaces](https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/) | |
30 | * [Writing VM and Container Managers](https://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers/) | |
31 | ||
32 | These wiki documents are not as up to date as they should be, currently, but | |
33 | the basic concepts still fully apply. You should read them too, if you do something | |
34 | with cgroups and systemd, in particular as they shine more light on the various | |
35 | D-Bus APIs provided. (That said, sooner or later we should probably fold that | |
36 | wiki documentation into this very document, too.) | |
37 | ||
38 | ## Two Key Design Rules | |
39 | ||
40 | Much of the philosophy behind these concepts is based on a couple of basic | |
4e1dfa45 CD |
41 | design ideas of cgroup v2 (which we however try to adapt as far as we can to |
42 | cgroup v1 too). Specifically two cgroup v2 rules are the most relevant: | |
e30eaff3 LP |
43 | |
44 | 1. The **no-processes-in-inner-nodes** rule: this means that it's not permitted | |
45 | to have processes directly attached to a cgroup that also has child cgroups and | |
46 | vice versa. A cgroup is either an inner node or a leaf node of the tree, and if | |
47 | it's an inner node it may not contain processes directly, and if it's a leaf | |
48 | node then it may not have child cgroups. (Note that there are some minor | |
5b24525a | 49 | exceptions to this rule, though. E.g. the root cgroup is special and allows |
e30eaff3 LP |
50 | both processes and children — which is used in particular to maintain kernel |
51 | threads.) | |
52 | ||
53 | 2. The **single-writer** rule: this means that each cgroup only has a single | |
54 | writer, i.e. a single process managing it. It's OK if different cgroups have | |
55 | different processes managing them. However, only a single process should own a | |
56 | specific cgroup, and when it does that ownership is exclusive, and nothing else | |
57 | should manipulate it at the same time. This rule ensures that various pieces of | |
58 | software don't step on each other's toes constantly. | |
59 | ||
60 | These two rules have various effects. For example, one corollary of this is: if | |
61 | your container manager creates and manages cgroups in the system's root cgroup | |
62 | you violate rule #2, as the root cgroup is managed by systemd and hence off | |
63 | limits to everybody else. | |
64 | ||
4e1dfa45 | 65 | Note that rule #1 is generally enforced by the kernel if cgroup v2 is used: as |
e30eaff3 | 66 | soon as you add a process to a cgroup it is ensured the rule is not |
4e1dfa45 | 67 | violated. On cgroup v1 this rule didn't exist, and hence isn't enforced, even |
e30eaff3 | 68 | though it's a good thing to follow it then too. Rule #2 is not enforced on |
4e1dfa45 | 69 | either cgroup v1 nor cgroup v2 (this is UNIX after all, in the general case |
e30eaff3 LP |
70 | root can do anything, modulo SELinux and friends), but if you ignore it you'll |
71 | be in constant pain as various pieces of software will fight over cgroup | |
72 | ownership. | |
73 | ||
4e1dfa45 | 74 | Note that cgroup v1 is currently the most deployed implementation, even though |
5b24525a | 75 | it's semantically broken in many ways, and in many cases doesn't actually do |
4e1dfa45 CD |
76 | what people think it does. cgroup v2 is where things are going, and most new |
77 | kernel features in this area are only added to cgroup v2, and not cgroup v1 | |
78 | anymore. For example cgroup v2 provides proper cgroup-empty notifications, has | |
5b24525a ZJS |
79 | support for all kinds of per-cgroup BPF magic, supports secure delegation of |
80 | cgroup trees to less privileged processes and so on, which all are not | |
4e1dfa45 | 81 | available on cgroup v1. |
e30eaff3 LP |
82 | |
83 | ## Three Different Tree Setups 🌳 | |
84 | ||
85 | systemd supports three different modes how cgroups are set up. Specifically: | |
86 | ||
4e1dfa45 | 87 | 1. **Unified** — this is the simplest mode, and exposes a pure cgroup v2 |
e30eaff3 LP |
88 | logic. In this mode `/sys/fs/cgroup` is the only mounted cgroup API file system |
89 | and all available controllers are exclusively exposed through it. | |
90 | ||
4e1dfa45 | 91 | 2. **Legacy** — this is the traditional cgroup v1 mode. In this mode the |
e30eaff3 LP |
92 | various controllers each get their own cgroup file system mounted to |
93 | `/sys/fs/cgroup/<controller>/`. On top of that systemd manages its own cgroup | |
94 | hierarchy for managing purposes as `/sys/fs/cgroup/systemd/`. | |
95 | ||
96 | 3. **Hybrid** — this is a hybrid between the unified and legacy mode. It's set | |
97 | up mostly like legacy, except that there's also an additional hierarchy | |
4e1dfa45 | 98 | `/sys/fs/cgroup/unified/` that contains the cgroup v2 hierarchy. (Note that in |
9afd5740 LP |
99 | this mode the unified hierarchy won't have controllers attached, the |
100 | controllers are all mounted as separate hierarchies as in legacy mode, | |
4e1dfa45 | 101 | i.e. `/sys/fs/cgroup/unified/` is purely and exclusively about core cgroup v2 |
9afd5740 | 102 | functionality and not about resource management.) In this mode compatibility |
4e1dfa45 | 103 | with cgroup v1 is retained while some cgroup v2 features are available |
9afd5740 LP |
104 | too. This mode is a stopgap. Don't bother with this too much unless you have |
105 | too much free time. | |
e30eaff3 LP |
106 | |
107 | To say this clearly, legacy and hybrid modes have no future. If you develop | |
108 | software today and don't focus on the unified mode, then you are writing | |
109 | software for yesterday, not tomorrow. They are primarily supported for | |
110 | compatibility reasons and will not receive new features. Sorry. | |
111 | ||
112 | Superficially, in legacy and hybrid modes it might appear that the parallel | |
113 | cgroup hierarchies for each controller are orthogonal from each other. In | |
114 | systemd they are not: the hierarchies of all controllers are always kept in | |
115 | sync (at least mostly: sub-trees might be suppressed in certain hierarchies if | |
116 | no controller usage is required for them). The fact that systemd keeps these | |
117 | hierarchies in sync means that the legacy and hybrid hierarchies are | |
118 | conceptually very close to the unified hierarchy. In particular this allows us | |
5b24525a ZJS |
119 | to talk of one specific cgroup and actually mean the same cgroup in all |
120 | available controller hierarchies. E.g. if we talk about the cgroup `/foo/bar/` | |
121 | then we actually mean `/sys/fs/cgroup/cpu/foo/bar/` as well as | |
122 | `/sys/fs/cgroup/memory/foo/bar/`, `/sys/fs/cgroup/pids/foo/bar/`, and so on. | |
4e1dfa45 | 123 | Note that in cgroup v2 the controller hierarchies aren't orthogonal, hence |
e30eaff3 LP |
124 | thinking about them as orthogonal won't help you in the long run anyway. |
125 | ||
126 | If you wonder how to detect which of these three modes is currently used, use | |
127 | `statfs()` on `/sys/fs/cgroup/`. If it reports `CGROUP2_SUPER_MAGIC` in its | |
128 | `.f_type` field, then you are in unified mode. If it reports `TMPFS_MAGIC` then | |
b2454670 | 129 | you are either in legacy or hybrid mode. To distinguish these two cases, run |
e30eaff3 LP |
130 | `statfs()` again on `/sys/fs/cgroup/unified/`. If that succeeds and reports |
131 | `CGROUP2_SUPER_MAGIC` you are in hybrid mode, otherwise not. | |
132 | ||
133 | ## systemd's Unit Types | |
134 | ||
135 | The low-level kernel cgroups feature is exposed in systemd in three different | |
136 | "unit" types. Specifically: | |
137 | ||
138 | 1. 💼 The `.service` unit type. This unit type is for units encapsulating | |
139 | processes systemd itself starts. Units of these types have cgroups that are | |
140 | the leaves of the cgroup tree the systemd instance manages (though possibly | |
141 | they might contain a sub-tree of their own managed by something else, made | |
142 | possible by the concept of delegation, see below). Service units are usually | |
143 | instantiated based on a unit file on disk that describes the command line to | |
144 | invoke and other properties of the service. However, service units may also | |
145 | be declared and started programmatically at runtime through a D-Bus API | |
146 | (which is called *transient* services). | |
147 | ||
148 | 2. 👓 The `.scope` unit type. This is very similar to `.service`. The main | |
149 | difference: the processes the units of this type encapsulate are forked off | |
150 | by some unrelated manager process, and that manager asked systemd to expose | |
151 | them as a unit. Unlike services, scopes can only be declared and started | |
152 | programmatically, i.e. are always transient. That's because they encapsulate | |
153 | processes forked off by something else, i.e. existing runtime objects, and | |
154 | hence cannot really be defined fully in 'offline' concepts such as unit | |
155 | files. | |
156 | ||
157 | 3. 🔪 The `.slice` unit type. Units of this type do not directly contain any | |
158 | processes. Units of this type are the inner nodes of part of the cgroup tree | |
159 | the systemd instance manages. Much like services, slices can be defined | |
160 | either on disk with unit files or programmatically as transient units. | |
161 | ||
162 | Slices expose the trunk and branches of a tree, and scopes and services are | |
163 | attached to those branches as leaves. The idea is that scopes and services can | |
164 | be moved around though, i.e. assigned to a different slice if needed. | |
165 | ||
166 | The naming of slice units directly maps to the cgroup tree path. This is not | |
167 | the case for service and scope units however. A slice named `foo-bar-baz.slice` | |
168 | maps to a cgroup `/foo.slice/foo-bar.slice/foo-bar-baz.slice/`. A service | |
169 | `quux.service` which is attached to the slice `foo-bar-baz.slice` maps to the | |
170 | cgroup `/foo.slice/foo-bar.slice/foo-bar-baz.slice/quux.service/`. | |
171 | ||
172 | By default systemd sets up four slice units: | |
173 | ||
174 | 1. `-.slice` is the root slice. i.e. the parent of everything else. On the host | |
4e1dfa45 | 175 | system it maps directly to the top-level directory of cgroup v2. |
e30eaff3 LP |
176 | |
177 | 2. `system.slice` is where system services are by default placed, unless | |
178 | configured otherwise. | |
179 | ||
180 | 3. `user.slice` is where user sessions are placed. Each user gets a slice of | |
181 | its own below that. | |
182 | ||
183 | 4. `machines.slice` is where VMs and containers are supposed to be | |
184 | placed. `systemd-nspawn` makes use of this by default, and you're very welcome | |
185 | to place your containers and VMs there too if you hack on managers for those. | |
186 | ||
187 | Users may define any amount of additional slices they like though, the four | |
188 | above are just the defaults. | |
189 | ||
190 | ## Delegation | |
191 | ||
192 | Container managers and suchlike often want to control cgroups directly using | |
193 | the raw kernel APIs. That's entirely fine and supported, as long as proper | |
4e1dfa45 CD |
194 | *delegation* is followed. Delegation is a concept we inherited from cgroup v2, |
195 | but we expose it on cgroup v1 too. Delegation means that some parts of the | |
e30eaff3 LP |
196 | cgroup tree may be managed by different managers than others. As long as it is |
197 | clear which manager manages which part of the tree each one can do within its | |
198 | sub-graph of the tree whatever it wants. | |
199 | ||
200 | Only sub-trees can be delegated (though whoever decides to request a sub-tree | |
5b24525a ZJS |
201 | can delegate sub-sub-trees further to somebody else if they like). Delegation |
202 | takes place at a specific cgroup: in systemd there's a `Delegate=` property you | |
203 | can set for a service or scope unit. If you do, it's the cut-off point for | |
204 | systemd's cgroup management: the unit itself is managed by systemd, i.e. all | |
205 | its attributes are managed exclusively by systemd, however your program may | |
206 | create/remove sub-cgroups inside it freely, and those then become exclusive | |
207 | property of your program, systemd won't touch them — all attributes of *those* | |
208 | sub-cgroups can be manipulated freely and exclusively by your program. | |
e30eaff3 LP |
209 | |
210 | By turning on the `Delegate=` property for a scope or service you get a few | |
211 | guarantees: | |
212 | ||
213 | 1. systemd won't fiddle with your sub-tree of the cgroup tree anymore. It won't | |
214 | change attributes of any cgroups below it, nor will it create or remove any | |
215 | cgroups thereunder, nor migrate processes across the boundaries of that | |
216 | sub-tree as it deems useful anymore. | |
217 | ||
218 | 2. If your service makes use of the `User=` functionality, then the sub-tree | |
219 | will be `chown()`ed to the indicated user so that it can correctly create | |
220 | cgroups below it. Note however that systemd will do that only in the unified | |
221 | hierarchy (in unified and hybrid mode) as well as on systemd's own private | |
222 | hierarchy (in legacy and hybrid mode). It won't pass ownership of the legacy | |
223 | controller hierarchies. Delegation to less privileges processes is not safe | |
4e1dfa45 | 224 | in cgroup v1 (as a limitation of the kernel), hence systemd won't facilitate |
e30eaff3 LP |
225 | access to it. |
226 | ||
227 | 3. Any BPF IP filter programs systemd installs will be installed with | |
228 | `BPF_F_ALLOW_MULTI` so that your program can install additional ones. | |
229 | ||
230 | In unit files the `Delegate=` property is superficially exposed as | |
231 | boolean. However, since v236 it optionally takes a list of controller names | |
232 | instead. If so, delegation is requested for listed controllers | |
1a31d050 | 233 | specifically. Note that this only encodes a request. Depending on various |
e30eaff3 LP |
234 | parameters it might happen that your service actually will get fewer |
235 | controllers delegated (for example, because the controller is not available on | |
236 | the current kernel or was turned off) or more. If no list is specified | |
237 | (i.e. the property simply set to `yes`) then all available controllers are | |
238 | delegated. | |
239 | ||
240 | Let's stress one thing: delegation is available on scope and service units | |
5b24525a ZJS |
241 | only. It's expressly not available on slice units. Why? Because slice units are |
242 | our *inner* nodes of the cgroup trees and we freely attach service and scopes | |
e5988600 | 243 | to them. If we'd allow delegation on slice units then this would mean that |
5b24525a ZJS |
244 | both systemd and your own manager would create/delete cgroups below the slice |
245 | unit and that conflicts with the single-writer rule. | |
e30eaff3 LP |
246 | |
247 | So, if you want to do your own raw cgroups kernel level access, then allocate a | |
248 | scope unit, or a service unit (or just use the service unit you already have | |
249 | for your service code), and turn on delegation for it. | |
250 | ||
e2391ce0 LP |
251 | (OK, here's one caveat: if you turn on delegation for a service, and that |
252 | service has `ExecStartPost=`, `ExecReload=`, `ExecStop=` or `ExecStopPost=` | |
253 | set, then these commands will be executed within the `.control/` sub-cgroup of | |
254 | your service's cgroup. This is necessary because by turning on delegation we | |
255 | have to assume that the cgroup delegated to your service is now an *inner* | |
256 | cgroup, which means that it may not directly contain any processes. Hence, if | |
257 | your service has any of these four settings set, you must be prepared that a | |
258 | `.control/` subcgroup might appear, managed by the service manager. This also | |
259 | means that your service code should have moved itself further down the cgroup | |
260 | tree by the time it notifies the service manager about start-up readiness, so | |
261 | that the service's main cgroup is definitely an inner node by the time the | |
262 | service manager might start `ExecStartPost=`.) | |
263 | ||
e30eaff3 LP |
264 | ## Three Scenarios |
265 | ||
266 | Let's say you write a container manager, and you wonder what to do regarding | |
267 | cgroups for it, as you want your manager to be able to run on systemd systems. | |
268 | ||
269 | You basically have three options: | |
270 | ||
5b24525a ZJS |
271 | 1. 😊 The *integration-is-good* option. For this, you register each container |
272 | you have either as a systemd service (i.e. let systemd invoke the executor | |
273 | binary for you) or a systemd scope (i.e. your manager executes the binary | |
274 | directly, but then tells systemd about it. In this mode the administrator | |
275 | can use the usual systemd resource management and reporting commands | |
276 | individually on those containers. By turning on `Delegate=` for these scopes | |
277 | or services you make it possible to run cgroup-enabled programs in your | |
278 | containers, for example a nested systemd instance. This option has two | |
279 | sub-options: | |
e30eaff3 | 280 | |
5b24525a ZJS |
281 | a. You transiently register the service or scope by directly contacting |
282 | systemd via D-Bus. In this case systemd will just manage the unit for you | |
283 | and nothing else. | |
e30eaff3 LP |
284 | |
285 | b. Instead you register the service or scope through `systemd-machined` | |
286 | (also via D-Bus). This mini-daemon is basically just a proxy for the same | |
287 | operations as in a. The main benefit of this: this way you let the system | |
288 | know that what you are registering is a container, and this opens up | |
289 | certain additional integration points. For example, `journalctl -M` can | |
290 | then be used to directly look into any container's journal logs (should | |
291 | the container run systemd inside), or `systemctl -M` can be used to | |
292 | directly invoke systemd operations inside the containers. Moreover tools | |
293 | like "ps" can then show you to which container a process belongs (`ps -eo | |
294 | pid,comm,machine`), and even gnome-system-monitor supports it. | |
295 | ||
296 | 2. 🙁 The *i-like-islands* option. If all you care about is your own cgroup tree, | |
297 | and you want to have to do as little as possible with systemd and no | |
298 | interest in integration with the rest of the system, then this is a valid | |
299 | option. For this all you have to do is turn on `Delegate=` for your main | |
300 | manager daemon. Then figure out the cgroup systemd placed your daemon in: | |
301 | you can now freely create sub-cgroups beneath it. Don't forget the | |
302 | *no-processes-in-inner-nodes* rule however: you have to move your main | |
303 | daemon process out of that cgroup (and into a sub-cgroup) before you can | |
304 | start further processes in any of your sub-cgroups. | |
305 | ||
306 | 3. 🙁 The *i-like-continents* option. In this option you'd leave your manager | |
307 | daemon where it is, and would not turn on delegation on its unit. However, | |
308 | as first thing you register a new scope unit with systemd, and that scope | |
309 | unit would have `Delegate=` turned on, and then you place all your | |
310 | containers underneath it. From systemd's PoV there'd be two units: your | |
311 | manager service and the big scope that contains all your containers in one. | |
312 | ||
313 | BTW: if for whatever reason you say "I hate D-Bus, I'll never call any D-Bus | |
314 | API, kthxbye", then options #1 and #3 are not available, as they generally | |
315 | involve talking to systemd from your program code, via D-Bus. You still have | |
316 | option #2 in that case however, as you can simply set `Delegate=` in your | |
317 | service's unit file and you are done and have your own sub-tree. In fact, #2 is | |
318 | the one option that allows you to completely ignore systemd's existence: you | |
319 | can entirely generically follow the single rule that you just use the cgroup | |
320 | you are started in, and everything below it, whatever that might be. That said, | |
321 | maybe if you dislike D-Bus and systemd that much, the better approach might be | |
322 | to work on that, and widen your horizon a bit. You are welcome. | |
323 | ||
324 | ## Controller Support | |
325 | ||
326 | systemd supports a number of controllers (but not all). Specifically, supported | |
327 | are: | |
328 | ||
4e1dfa45 CD |
329 | * on cgroup v1: `cpu`, `cpuacct`, `blkio`, `memory`, `devices`, `pids` |
330 | * on cgroup v2: `cpu`, `io`, `memory`, `pids` | |
e30eaff3 | 331 | |
4e1dfa45 CD |
332 | It is our intention to natively support all cgroup v2 controllers as they are |
333 | added to the kernel. However, regarding cgroup v1: at this point we will not | |
5b24525a | 334 | add support for any other controllers anymore. This means systemd currently |
4e1dfa45 | 335 | does not and will never manage the following controllers on cgroup v1: |
e30eaff3 LP |
336 | `freezer`, `cpuset`, `net_cls`, `perf_event`, `net_prio`, `hugetlb`. Why not? |
337 | Depending on the case, either their API semantics or implementations aren't | |
4e1dfa45 | 338 | really usable, or it's very clear they have no future on cgroup v2, and we |
e30eaff3 LP |
339 | won't add new code for stuff that clearly has no future. |
340 | ||
4e1dfa45 | 341 | Effectively this means that all those mentioned cgroup v1 controllers are up |
e30eaff3 LP |
342 | for grabs: systemd won't manage them, and hence won't delegate them to your |
343 | code (however, systemd will still mount their hierarchies, simply because it | |
344 | mounts all controller hierarchies it finds available in the kernel). If you | |
345 | decide to use them, then that's fine, but systemd won't help you with it (but | |
346 | also not interfere with it). To be nice to other tenants it might be wise to | |
347 | replicate the cgroup hierarchies of the other controllers in them too however, | |
348 | but of course that's between you and those other tenants, and systemd won't | |
349 | care. Replicating the cgroup hierarchies in those unsupported controllers would | |
350 | mean replicating the full cgroup paths in them, and hence the prefixing | |
351 | `.slice` components too, otherwise the hierarchies will start being orthogonal | |
352 | after all, and that's not really desirable. On more thing: systemd will clean | |
353 | up after you in the hierarchies it manages: if your daemon goes down, its | |
354 | cgroups will be removed too. You basically get the guarantee that you start | |
355 | with a pristine cgroup sub-tree for your service or scope whenever it is | |
356 | started. This is not the case however in the hierarchies systemd doesn't | |
357 | manage. This means that your programs should be ready to deal with left-over | |
358 | cgroups in them — from previous runs, and be extra careful with them as they | |
359 | might still carry settings that might not be valid anymore. | |
360 | ||
361 | Note a particular asymmetry here: if your systemd version doesn't support a | |
4e1dfa45 | 362 | specific controller on cgroup v1 you can still make use of it for delegation, |
e30eaff3 | 363 | by directly fiddling with its hierarchy and replicating the cgroup tree there |
4e1dfa45 | 364 | as necessary (as suggested above). However, on cgroup v2 this is different: |
e30eaff3 LP |
365 | separately mounted hierarchies are not available, and delegation has always to |
366 | happen through systemd itself. This means: when you update your kernel and it | |
367 | adds a new, so far unseen controller, and you want to use it for delegation, | |
368 | then you also need to update systemd to a version that groks it. | |
369 | ||
370 | ## systemd as Container Payload | |
371 | ||
372 | systemd can happily run as a container payload's PID 1. Note that systemd | |
373 | unconditionally needs write access to the cgroup tree however, hence you need | |
374 | to delegate a sub-tree to it. Note that there's nothing too special you have to | |
375 | do beyond that: just invoke systemd as PID 1 inside the root of the delegated | |
376 | cgroup sub-tree, and it will figure out the rest: it will determine the cgroup | |
377 | it is running in and take possession of it. It won't interfere with any cgroup | |
378 | outside of the sub-tree it was invoked in. Use of `CLONE_NEWCGROUP` is hence | |
379 | optional (but of course wise). | |
380 | ||
381 | Note one particular asymmetry here though: systemd will try to take possession | |
382 | of the root cgroup you pass to it *in* *full*, i.e. it will not only | |
e5988600 | 383 | create/remove child cgroups below it, it will also attempt to manage the |
e30eaff3 LP |
384 | attributes of it. OTOH as mentioned above, when delegating a cgroup tree to |
385 | somebody else it only passes the rights to create/remove sub-cgroups, but will | |
386 | insist on managing the delegated cgroup tree's top-level attributes. Or in | |
387 | other words: systemd is *greedy* when accepting delegated cgroup trees and also | |
388 | *greedy* when delegating them to others: it insists on managing attributes on | |
389 | the specific cgroup in both cases. A container manager that is itself a payload | |
390 | of a host systemd which wants to run a systemd as its own container payload | |
391 | instead hence needs to insert an extra level in the hierarchy in between, so | |
392 | that the systemd on the host and the one in the container won't fight for the | |
393 | attributes. That said, you likely should do that anyway, due to the | |
394 | no-processes-in-inner-cgroups rule, see below. | |
395 | ||
396 | When systemd runs as container payload it will make use of all hierarchies it | |
397 | has write access to. For legacy mode you need to make at least | |
398 | `/sys/fs/cgroup/systemd/` available, all other hierarchies are optional. For | |
399 | hybrid mode you need to add `/sys/fs/cgroup/unified/`. Finally, for fully | |
400 | unified you (of course, I guess) need to provide only `/sys/fs/cgroup/` itself. | |
401 | ||
402 | ## Some Dos | |
403 | ||
404 | 1. ⚡ If you go for implementation option 1a or 1b (as in the list above), then | |
405 | each of your containers will have its own systemd-managed unit and hence | |
406 | cgroup with possibly further sub-cgroups below. Typically the first process | |
407 | running in that unit will be some kind of executor program, which will in | |
408 | turn fork off the payload processes of the container. In this case don't | |
409 | forget that there are two levels of delegation involved: first, systemd | |
410 | delegates a group sub-tree to your executor. And then your executor should | |
411 | delegate a sub-tree further down to the container payload. Oh, and because | |
412 | of the no-process-in-inner-nodes rule, your executor needs to migrate itself | |
413 | to a sub-cgroup of the cgroup it got delegated, too. Most likely you hence | |
414 | want a two-pronged approach: below the cgroup you got started in, you want | |
415 | one cgroup maybe called `supervisor/` where your manager runs in and then | |
416 | for each container a sibling cgroup of that maybe called `payload-xyz/`. | |
417 | ||
418 | 2. ⚡ Don't forget that the cgroups you create have to have names that are | |
419 | suitable as UNIX file names, and that they live in the same namespace as the | |
420 | various kernel attribute files. Hence, when you want to allow the user | |
421 | arbitrary naming, you might need to escape some of the names (for example, | |
422 | you really don't want to create a cgroup named `tasks`, just because the | |
423 | user created a container by that name, because `tasks` after all is a magic | |
4e1dfa45 | 424 | attribute in cgroup v1, and your `mkdir()` will hence fail with `EEXIST`. In |
e30eaff3 LP |
425 | systemd we do escaping by prefixing names that might collide with a kernel |
426 | attribute name with an underscore. You might want to do the same, but this | |
427 | is really up to you how you do it. Just do it, and be careful. | |
428 | ||
429 | ## Some Don'ts | |
430 | ||
431 | 1. 🚫 Never create your own cgroups below arbitrary cgroups systemd manages, i.e | |
432 | cgroups you haven't set `Delegate=` in. Specifically: 🔥 don't create your | |
433 | own cgroups below the root cgroup 🔥. That's owned by systemd, and you will | |
434 | step on systemd's toes if you ignore that, and systemd will step on | |
435 | yours. Get your own delegated sub-tree, you may create as many cgroups there | |
436 | as you like. Seriously, if you create cgroups directly in the cgroup root, | |
437 | then all you do is ask for trouble. | |
438 | ||
439 | 2. 🚫 Don't attempt to set `Delegate=` in slice units, and in particular not in | |
440 | `-.slice`. It's not supported, and will generate an error. | |
441 | ||
442 | 3. 🚫 Never *write* to any of the attributes of a cgroup systemd created for | |
443 | you. It's systemd's private property. You are welcome to manipulate the | |
444 | attributes of cgroups you created in your own delegated sub-tree, but the | |
445 | cgroup tree of systemd itself is out of limits for you. It's fine to *read* | |
446 | from any attribute you like however. That's totally OK and welcome. | |
447 | ||
d11623e9 LP |
448 | 4. 🚫 When not using `CLONE_NEWCGROUP` when delegating a sub-tree to a |
449 | container payload running systemd, then don't get the idea that you can bind | |
450 | mount only a sub-tree of the host's cgroup tree into the container. Part of | |
451 | the cgroup API is that `/proc/$PID/cgroup` reports the cgroup path of every | |
e30eaff3 LP |
452 | process, and hence any path below `/sys/fs/cgroup/` needs to match what |
453 | `/proc/$PID/cgroup` of the payload processes reports. What you can do safely | |
d11623e9 LP |
454 | however, is mount the upper parts of the cgroup tree read-only (or even |
455 | replace the middle bits with an intermediary `tmpfs` — but be careful not to | |
456 | break the `statfs()` detection logic discussed above), as long as the path | |
457 | to the delegated sub-tree remains accessible as-is. | |
e30eaff3 | 458 | |
3ee9b2f6 LP |
459 | 5. ⚡ Currently, the algorithm for mapping between slice/scope/service unit |
460 | naming and their cgroup paths is not considered public API of systemd, and | |
461 | may change in future versions. This means: it's best to avoid implementing a | |
462 | local logic of translating cgroup paths to slice/scope/service names in your | |
463 | program, or vice versa — it's likely going to break sooner or later. Use the | |
464 | appropriate D-Bus API calls for that instead, so that systemd translates | |
465 | this for you. (Specifically: each Unit object has a `ControlGroup` property | |
466 | to get the cgroup for a unit. The method `GetUnitByControlGroup()` may be | |
467 | used to get the unit for a cgroup.) | |
468 | ||
4e1dfa45 | 469 | 6. ⚡ Think twice before delegating cgroup v1 controllers to less privileged |
e30eaff3 | 470 | containers. It's not safe, you basically allow your containers to freeze the |
4e1dfa45 | 471 | system with that and worse. Delegation is a strongpoint of cgroup v2 though, |
e30eaff3 LP |
472 | and there it's safe to treat delegation boundaries as privilege boundaries. |
473 | ||
474 | And that's it for now. If you have further questions, refer to the systemd | |
475 | mailing list. | |
476 | ||
477 | — Berlin, 2018-04-20 |