]>
Commit | Line | Data |
---|---|---|
e30eaff3 LP |
1 | # Control Group APIs and Delegation |
2 | ||
1e46eb59 LP |
3 | *Intended audience: hackers working on userspace subsystems that require direct |
4 | cgroup access, such as container managers and similar.* | |
5 | ||
e30eaff3 LP |
6 | So you are wondering about resource management with systemd, you know Linux |
7 | control groups (cgroups) a bit and are trying to integrate your software with | |
8 | what systemd has to offer there. Here's a bit of documentation about the | |
9 | concepts and interfaces involved with this. | |
10 | ||
11 | What's described here has been part of systemd and documented since v205 | |
5b24525a | 12 | times. However, it has been updated and improved substantially, even |
e30eaff3 LP |
13 | though the concepts stayed mostly the same. This is an attempt to provide more |
14 | comprehensive up-to-date information about all this, particular in light of the | |
15 | poor implementations of the components interfacing with systemd of current | |
16 | container managers. | |
17 | ||
18 | Before you read on, please make sure you read the low-level [kernel | |
19 | documentation about | |
20 | cgroupsv2](https://www.kernel.org/doc/Documentation/cgroup-v2.txt). This | |
21 | documentation then adds in the higher-level view from systemd. | |
22 | ||
23 | This document augments the existing documentation we already have: | |
24 | ||
25 | * [The New Control Group Interfaces](https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/) | |
26 | * [Writing VM and Container Managers](https://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers/) | |
27 | ||
28 | These wiki documents are not as up to date as they should be, currently, but | |
29 | the basic concepts still fully apply. You should read them too, if you do something | |
30 | with cgroups and systemd, in particular as they shine more light on the various | |
31 | D-Bus APIs provided. (That said, sooner or later we should probably fold that | |
32 | wiki documentation into this very document, too.) | |
33 | ||
34 | ## Two Key Design Rules | |
35 | ||
36 | Much of the philosophy behind these concepts is based on a couple of basic | |
37 | design ideas of cgroupsv2 (which we however try to adapt as far as we can to | |
38 | cgroupsv1 too). Specifically two cgroupsv2 rules are the most relevant: | |
39 | ||
40 | 1. The **no-processes-in-inner-nodes** rule: this means that it's not permitted | |
41 | to have processes directly attached to a cgroup that also has child cgroups and | |
42 | vice versa. A cgroup is either an inner node or a leaf node of the tree, and if | |
43 | it's an inner node it may not contain processes directly, and if it's a leaf | |
44 | node then it may not have child cgroups. (Note that there are some minor | |
5b24525a | 45 | exceptions to this rule, though. E.g. the root cgroup is special and allows |
e30eaff3 LP |
46 | both processes and children — which is used in particular to maintain kernel |
47 | threads.) | |
48 | ||
49 | 2. The **single-writer** rule: this means that each cgroup only has a single | |
50 | writer, i.e. a single process managing it. It's OK if different cgroups have | |
51 | different processes managing them. However, only a single process should own a | |
52 | specific cgroup, and when it does that ownership is exclusive, and nothing else | |
53 | should manipulate it at the same time. This rule ensures that various pieces of | |
54 | software don't step on each other's toes constantly. | |
55 | ||
56 | These two rules have various effects. For example, one corollary of this is: if | |
57 | your container manager creates and manages cgroups in the system's root cgroup | |
58 | you violate rule #2, as the root cgroup is managed by systemd and hence off | |
59 | limits to everybody else. | |
60 | ||
61 | Note that rule #1 is generally enforced by the kernel if cgroupsv2 is used: as | |
62 | soon as you add a process to a cgroup it is ensured the rule is not | |
63 | violated. On cgroupsv1 this rule didn't exist, and hence isn't enforced, even | |
64 | though it's a good thing to follow it then too. Rule #2 is not enforced on | |
65 | either cgroupsv1 nor cgroupsv2 (this is UNIX after all, in the general case | |
66 | root can do anything, modulo SELinux and friends), but if you ignore it you'll | |
67 | be in constant pain as various pieces of software will fight over cgroup | |
68 | ownership. | |
69 | ||
5b24525a ZJS |
70 | Note that cgroupsv1 is currently the most deployed implementation, even though |
71 | it's semantically broken in many ways, and in many cases doesn't actually do | |
72 | what people think it does. cgroupsv2 is where things are going, and most new | |
73 | kernel features in this area are only added to cgroupsv2, and not cgroupsv1 | |
74 | anymore. For example cgroupsv2 provides proper cgroup-empty notifications, has | |
75 | support for all kinds of per-cgroup BPF magic, supports secure delegation of | |
76 | cgroup trees to less privileged processes and so on, which all are not | |
77 | available on cgroupsv1. | |
e30eaff3 LP |
78 | |
79 | ## Three Different Tree Setups 🌳 | |
80 | ||
81 | systemd supports three different modes how cgroups are set up. Specifically: | |
82 | ||
83 | 1. **Unified** — this is the simplest mode, and exposes a pure cgroupsv2 | |
84 | logic. In this mode `/sys/fs/cgroup` is the only mounted cgroup API file system | |
85 | and all available controllers are exclusively exposed through it. | |
86 | ||
87 | 2. **Legacy** — this is the traditional cgroupsv1 mode. In this mode the | |
88 | various controllers each get their own cgroup file system mounted to | |
89 | `/sys/fs/cgroup/<controller>/`. On top of that systemd manages its own cgroup | |
90 | hierarchy for managing purposes as `/sys/fs/cgroup/systemd/`. | |
91 | ||
92 | 3. **Hybrid** — this is a hybrid between the unified and legacy mode. It's set | |
93 | up mostly like legacy, except that there's also an additional hierarchy | |
9afd5740 LP |
94 | `/sys/fs/cgroup/unified/` that contains the cgroupsv2 hierarchy. (Note that in |
95 | this mode the unified hierarchy won't have controllers attached, the | |
96 | controllers are all mounted as separate hierarchies as in legacy mode, | |
97 | i.e. `/sys/fs/cgroup/unified/` is purely and exclusively about core cgroupsv2 | |
98 | functionality and not about resource management.) In this mode compatibility | |
99 | with cgroupsv1 is retained while some cgroupsv2 features are available | |
100 | too. This mode is a stopgap. Don't bother with this too much unless you have | |
101 | too much free time. | |
e30eaff3 LP |
102 | |
103 | To say this clearly, legacy and hybrid modes have no future. If you develop | |
104 | software today and don't focus on the unified mode, then you are writing | |
105 | software for yesterday, not tomorrow. They are primarily supported for | |
106 | compatibility reasons and will not receive new features. Sorry. | |
107 | ||
108 | Superficially, in legacy and hybrid modes it might appear that the parallel | |
109 | cgroup hierarchies for each controller are orthogonal from each other. In | |
110 | systemd they are not: the hierarchies of all controllers are always kept in | |
111 | sync (at least mostly: sub-trees might be suppressed in certain hierarchies if | |
112 | no controller usage is required for them). The fact that systemd keeps these | |
113 | hierarchies in sync means that the legacy and hybrid hierarchies are | |
114 | conceptually very close to the unified hierarchy. In particular this allows us | |
5b24525a ZJS |
115 | to talk of one specific cgroup and actually mean the same cgroup in all |
116 | available controller hierarchies. E.g. if we talk about the cgroup `/foo/bar/` | |
117 | then we actually mean `/sys/fs/cgroup/cpu/foo/bar/` as well as | |
118 | `/sys/fs/cgroup/memory/foo/bar/`, `/sys/fs/cgroup/pids/foo/bar/`, and so on. | |
119 | Note that in cgroupsv2 the controller hierarchies aren't orthogonal, hence | |
e30eaff3 LP |
120 | thinking about them as orthogonal won't help you in the long run anyway. |
121 | ||
122 | If you wonder how to detect which of these three modes is currently used, use | |
123 | `statfs()` on `/sys/fs/cgroup/`. If it reports `CGROUP2_SUPER_MAGIC` in its | |
124 | `.f_type` field, then you are in unified mode. If it reports `TMPFS_MAGIC` then | |
125 | you are either in legacy or hybrid mode. To distuingish these two cases, run | |
126 | `statfs()` again on `/sys/fs/cgroup/unified/`. If that succeeds and reports | |
127 | `CGROUP2_SUPER_MAGIC` you are in hybrid mode, otherwise not. | |
128 | ||
129 | ## systemd's Unit Types | |
130 | ||
131 | The low-level kernel cgroups feature is exposed in systemd in three different | |
132 | "unit" types. Specifically: | |
133 | ||
134 | 1. 💼 The `.service` unit type. This unit type is for units encapsulating | |
135 | processes systemd itself starts. Units of these types have cgroups that are | |
136 | the leaves of the cgroup tree the systemd instance manages (though possibly | |
137 | they might contain a sub-tree of their own managed by something else, made | |
138 | possible by the concept of delegation, see below). Service units are usually | |
139 | instantiated based on a unit file on disk that describes the command line to | |
140 | invoke and other properties of the service. However, service units may also | |
141 | be declared and started programmatically at runtime through a D-Bus API | |
142 | (which is called *transient* services). | |
143 | ||
144 | 2. 👓 The `.scope` unit type. This is very similar to `.service`. The main | |
145 | difference: the processes the units of this type encapsulate are forked off | |
146 | by some unrelated manager process, and that manager asked systemd to expose | |
147 | them as a unit. Unlike services, scopes can only be declared and started | |
148 | programmatically, i.e. are always transient. That's because they encapsulate | |
149 | processes forked off by something else, i.e. existing runtime objects, and | |
150 | hence cannot really be defined fully in 'offline' concepts such as unit | |
151 | files. | |
152 | ||
153 | 3. 🔪 The `.slice` unit type. Units of this type do not directly contain any | |
154 | processes. Units of this type are the inner nodes of part of the cgroup tree | |
155 | the systemd instance manages. Much like services, slices can be defined | |
156 | either on disk with unit files or programmatically as transient units. | |
157 | ||
158 | Slices expose the trunk and branches of a tree, and scopes and services are | |
159 | attached to those branches as leaves. The idea is that scopes and services can | |
160 | be moved around though, i.e. assigned to a different slice if needed. | |
161 | ||
162 | The naming of slice units directly maps to the cgroup tree path. This is not | |
163 | the case for service and scope units however. A slice named `foo-bar-baz.slice` | |
164 | maps to a cgroup `/foo.slice/foo-bar.slice/foo-bar-baz.slice/`. A service | |
165 | `quux.service` which is attached to the slice `foo-bar-baz.slice` maps to the | |
166 | cgroup `/foo.slice/foo-bar.slice/foo-bar-baz.slice/quux.service/`. | |
167 | ||
168 | By default systemd sets up four slice units: | |
169 | ||
170 | 1. `-.slice` is the root slice. i.e. the parent of everything else. On the host | |
171 | system it maps directly to the top-level directory of cgroupsv2. | |
172 | ||
173 | 2. `system.slice` is where system services are by default placed, unless | |
174 | configured otherwise. | |
175 | ||
176 | 3. `user.slice` is where user sessions are placed. Each user gets a slice of | |
177 | its own below that. | |
178 | ||
179 | 4. `machines.slice` is where VMs and containers are supposed to be | |
180 | placed. `systemd-nspawn` makes use of this by default, and you're very welcome | |
181 | to place your containers and VMs there too if you hack on managers for those. | |
182 | ||
183 | Users may define any amount of additional slices they like though, the four | |
184 | above are just the defaults. | |
185 | ||
186 | ## Delegation | |
187 | ||
188 | Container managers and suchlike often want to control cgroups directly using | |
189 | the raw kernel APIs. That's entirely fine and supported, as long as proper | |
190 | *delegation* is followed. Delegation is a concept we inherited from cgroupsv2, | |
191 | but we expose it on cgroupsv1 too. Delegation means that some parts of the | |
192 | cgroup tree may be managed by different managers than others. As long as it is | |
193 | clear which manager manages which part of the tree each one can do within its | |
194 | sub-graph of the tree whatever it wants. | |
195 | ||
196 | Only sub-trees can be delegated (though whoever decides to request a sub-tree | |
5b24525a ZJS |
197 | can delegate sub-sub-trees further to somebody else if they like). Delegation |
198 | takes place at a specific cgroup: in systemd there's a `Delegate=` property you | |
199 | can set for a service or scope unit. If you do, it's the cut-off point for | |
200 | systemd's cgroup management: the unit itself is managed by systemd, i.e. all | |
201 | its attributes are managed exclusively by systemd, however your program may | |
202 | create/remove sub-cgroups inside it freely, and those then become exclusive | |
203 | property of your program, systemd won't touch them — all attributes of *those* | |
204 | sub-cgroups can be manipulated freely and exclusively by your program. | |
e30eaff3 LP |
205 | |
206 | By turning on the `Delegate=` property for a scope or service you get a few | |
207 | guarantees: | |
208 | ||
209 | 1. systemd won't fiddle with your sub-tree of the cgroup tree anymore. It won't | |
210 | change attributes of any cgroups below it, nor will it create or remove any | |
211 | cgroups thereunder, nor migrate processes across the boundaries of that | |
212 | sub-tree as it deems useful anymore. | |
213 | ||
214 | 2. If your service makes use of the `User=` functionality, then the sub-tree | |
215 | will be `chown()`ed to the indicated user so that it can correctly create | |
216 | cgroups below it. Note however that systemd will do that only in the unified | |
217 | hierarchy (in unified and hybrid mode) as well as on systemd's own private | |
218 | hierarchy (in legacy and hybrid mode). It won't pass ownership of the legacy | |
219 | controller hierarchies. Delegation to less privileges processes is not safe | |
220 | in cgroupsv1 (as a limitation of the kernel), hence systemd won't facilitate | |
221 | access to it. | |
222 | ||
223 | 3. Any BPF IP filter programs systemd installs will be installed with | |
224 | `BPF_F_ALLOW_MULTI` so that your program can install additional ones. | |
225 | ||
226 | In unit files the `Delegate=` property is superficially exposed as | |
227 | boolean. However, since v236 it optionally takes a list of controller names | |
228 | instead. If so, delegation is requested for listed controllers | |
229 | specifically. Note hat this only encodes a request. Depending on various | |
230 | parameters it might happen that your service actually will get fewer | |
231 | controllers delegated (for example, because the controller is not available on | |
232 | the current kernel or was turned off) or more. If no list is specified | |
233 | (i.e. the property simply set to `yes`) then all available controllers are | |
234 | delegated. | |
235 | ||
236 | Let's stress one thing: delegation is available on scope and service units | |
5b24525a ZJS |
237 | only. It's expressly not available on slice units. Why? Because slice units are |
238 | our *inner* nodes of the cgroup trees and we freely attach service and scopes | |
e5988600 | 239 | to them. If we'd allow delegation on slice units then this would mean that |
5b24525a ZJS |
240 | both systemd and your own manager would create/delete cgroups below the slice |
241 | unit and that conflicts with the single-writer rule. | |
e30eaff3 LP |
242 | |
243 | So, if you want to do your own raw cgroups kernel level access, then allocate a | |
244 | scope unit, or a service unit (or just use the service unit you already have | |
245 | for your service code), and turn on delegation for it. | |
246 | ||
247 | ## Three Scenarios | |
248 | ||
249 | Let's say you write a container manager, and you wonder what to do regarding | |
250 | cgroups for it, as you want your manager to be able to run on systemd systems. | |
251 | ||
252 | You basically have three options: | |
253 | ||
5b24525a ZJS |
254 | 1. 😊 The *integration-is-good* option. For this, you register each container |
255 | you have either as a systemd service (i.e. let systemd invoke the executor | |
256 | binary for you) or a systemd scope (i.e. your manager executes the binary | |
257 | directly, but then tells systemd about it. In this mode the administrator | |
258 | can use the usual systemd resource management and reporting commands | |
259 | individually on those containers. By turning on `Delegate=` for these scopes | |
260 | or services you make it possible to run cgroup-enabled programs in your | |
261 | containers, for example a nested systemd instance. This option has two | |
262 | sub-options: | |
e30eaff3 | 263 | |
5b24525a ZJS |
264 | a. You transiently register the service or scope by directly contacting |
265 | systemd via D-Bus. In this case systemd will just manage the unit for you | |
266 | and nothing else. | |
e30eaff3 LP |
267 | |
268 | b. Instead you register the service or scope through `systemd-machined` | |
269 | (also via D-Bus). This mini-daemon is basically just a proxy for the same | |
270 | operations as in a. The main benefit of this: this way you let the system | |
271 | know that what you are registering is a container, and this opens up | |
272 | certain additional integration points. For example, `journalctl -M` can | |
273 | then be used to directly look into any container's journal logs (should | |
274 | the container run systemd inside), or `systemctl -M` can be used to | |
275 | directly invoke systemd operations inside the containers. Moreover tools | |
276 | like "ps" can then show you to which container a process belongs (`ps -eo | |
277 | pid,comm,machine`), and even gnome-system-monitor supports it. | |
278 | ||
279 | 2. 🙁 The *i-like-islands* option. If all you care about is your own cgroup tree, | |
280 | and you want to have to do as little as possible with systemd and no | |
281 | interest in integration with the rest of the system, then this is a valid | |
282 | option. For this all you have to do is turn on `Delegate=` for your main | |
283 | manager daemon. Then figure out the cgroup systemd placed your daemon in: | |
284 | you can now freely create sub-cgroups beneath it. Don't forget the | |
285 | *no-processes-in-inner-nodes* rule however: you have to move your main | |
286 | daemon process out of that cgroup (and into a sub-cgroup) before you can | |
287 | start further processes in any of your sub-cgroups. | |
288 | ||
289 | 3. 🙁 The *i-like-continents* option. In this option you'd leave your manager | |
290 | daemon where it is, and would not turn on delegation on its unit. However, | |
291 | as first thing you register a new scope unit with systemd, and that scope | |
292 | unit would have `Delegate=` turned on, and then you place all your | |
293 | containers underneath it. From systemd's PoV there'd be two units: your | |
294 | manager service and the big scope that contains all your containers in one. | |
295 | ||
296 | BTW: if for whatever reason you say "I hate D-Bus, I'll never call any D-Bus | |
297 | API, kthxbye", then options #1 and #3 are not available, as they generally | |
298 | involve talking to systemd from your program code, via D-Bus. You still have | |
299 | option #2 in that case however, as you can simply set `Delegate=` in your | |
300 | service's unit file and you are done and have your own sub-tree. In fact, #2 is | |
301 | the one option that allows you to completely ignore systemd's existence: you | |
302 | can entirely generically follow the single rule that you just use the cgroup | |
303 | you are started in, and everything below it, whatever that might be. That said, | |
304 | maybe if you dislike D-Bus and systemd that much, the better approach might be | |
305 | to work on that, and widen your horizon a bit. You are welcome. | |
306 | ||
307 | ## Controller Support | |
308 | ||
309 | systemd supports a number of controllers (but not all). Specifically, supported | |
310 | are: | |
311 | ||
312 | * on cgroupsv1: `cpu`, `cpuacct`, `blkio`, `memory`, `devices`, `pids` | |
313 | * on cgroupsv2: `cpu`, `io`, `memory`, `pids` | |
314 | ||
5b24525a ZJS |
315 | It is our intention to natively support all cgroupsv2 controllers as they are |
316 | added to the kernel. However, regarding cgroupsv1: at this point we will not | |
317 | add support for any other controllers anymore. This means systemd currently | |
e30eaff3 LP |
318 | does not and will never manage the following controllers on cgroupsv1: |
319 | `freezer`, `cpuset`, `net_cls`, `perf_event`, `net_prio`, `hugetlb`. Why not? | |
320 | Depending on the case, either their API semantics or implementations aren't | |
321 | really usable, or it's very clear they have no future on cgroupsv2, and we | |
322 | won't add new code for stuff that clearly has no future. | |
323 | ||
324 | Effectively this means that all those mentioned cgroupsv1 controllers are up | |
325 | for grabs: systemd won't manage them, and hence won't delegate them to your | |
326 | code (however, systemd will still mount their hierarchies, simply because it | |
327 | mounts all controller hierarchies it finds available in the kernel). If you | |
328 | decide to use them, then that's fine, but systemd won't help you with it (but | |
329 | also not interfere with it). To be nice to other tenants it might be wise to | |
330 | replicate the cgroup hierarchies of the other controllers in them too however, | |
331 | but of course that's between you and those other tenants, and systemd won't | |
332 | care. Replicating the cgroup hierarchies in those unsupported controllers would | |
333 | mean replicating the full cgroup paths in them, and hence the prefixing | |
334 | `.slice` components too, otherwise the hierarchies will start being orthogonal | |
335 | after all, and that's not really desirable. On more thing: systemd will clean | |
336 | up after you in the hierarchies it manages: if your daemon goes down, its | |
337 | cgroups will be removed too. You basically get the guarantee that you start | |
338 | with a pristine cgroup sub-tree for your service or scope whenever it is | |
339 | started. This is not the case however in the hierarchies systemd doesn't | |
340 | manage. This means that your programs should be ready to deal with left-over | |
341 | cgroups in them — from previous runs, and be extra careful with them as they | |
342 | might still carry settings that might not be valid anymore. | |
343 | ||
344 | Note a particular asymmetry here: if your systemd version doesn't support a | |
345 | specific controller on cgroupsv1 you can still make use of it for delegation, | |
346 | by directly fiddling with its hierarchy and replicating the cgroup tree there | |
347 | as necessary (as suggested above). However, on cgroupsv2 this is different: | |
348 | separately mounted hierarchies are not available, and delegation has always to | |
349 | happen through systemd itself. This means: when you update your kernel and it | |
350 | adds a new, so far unseen controller, and you want to use it for delegation, | |
351 | then you also need to update systemd to a version that groks it. | |
352 | ||
353 | ## systemd as Container Payload | |
354 | ||
355 | systemd can happily run as a container payload's PID 1. Note that systemd | |
356 | unconditionally needs write access to the cgroup tree however, hence you need | |
357 | to delegate a sub-tree to it. Note that there's nothing too special you have to | |
358 | do beyond that: just invoke systemd as PID 1 inside the root of the delegated | |
359 | cgroup sub-tree, and it will figure out the rest: it will determine the cgroup | |
360 | it is running in and take possession of it. It won't interfere with any cgroup | |
361 | outside of the sub-tree it was invoked in. Use of `CLONE_NEWCGROUP` is hence | |
362 | optional (but of course wise). | |
363 | ||
364 | Note one particular asymmetry here though: systemd will try to take possession | |
365 | of the root cgroup you pass to it *in* *full*, i.e. it will not only | |
e5988600 | 366 | create/remove child cgroups below it, it will also attempt to manage the |
e30eaff3 LP |
367 | attributes of it. OTOH as mentioned above, when delegating a cgroup tree to |
368 | somebody else it only passes the rights to create/remove sub-cgroups, but will | |
369 | insist on managing the delegated cgroup tree's top-level attributes. Or in | |
370 | other words: systemd is *greedy* when accepting delegated cgroup trees and also | |
371 | *greedy* when delegating them to others: it insists on managing attributes on | |
372 | the specific cgroup in both cases. A container manager that is itself a payload | |
373 | of a host systemd which wants to run a systemd as its own container payload | |
374 | instead hence needs to insert an extra level in the hierarchy in between, so | |
375 | that the systemd on the host and the one in the container won't fight for the | |
376 | attributes. That said, you likely should do that anyway, due to the | |
377 | no-processes-in-inner-cgroups rule, see below. | |
378 | ||
379 | When systemd runs as container payload it will make use of all hierarchies it | |
380 | has write access to. For legacy mode you need to make at least | |
381 | `/sys/fs/cgroup/systemd/` available, all other hierarchies are optional. For | |
382 | hybrid mode you need to add `/sys/fs/cgroup/unified/`. Finally, for fully | |
383 | unified you (of course, I guess) need to provide only `/sys/fs/cgroup/` itself. | |
384 | ||
385 | ## Some Dos | |
386 | ||
387 | 1. ⚡ If you go for implementation option 1a or 1b (as in the list above), then | |
388 | each of your containers will have its own systemd-managed unit and hence | |
389 | cgroup with possibly further sub-cgroups below. Typically the first process | |
390 | running in that unit will be some kind of executor program, which will in | |
391 | turn fork off the payload processes of the container. In this case don't | |
392 | forget that there are two levels of delegation involved: first, systemd | |
393 | delegates a group sub-tree to your executor. And then your executor should | |
394 | delegate a sub-tree further down to the container payload. Oh, and because | |
395 | of the no-process-in-inner-nodes rule, your executor needs to migrate itself | |
396 | to a sub-cgroup of the cgroup it got delegated, too. Most likely you hence | |
397 | want a two-pronged approach: below the cgroup you got started in, you want | |
398 | one cgroup maybe called `supervisor/` where your manager runs in and then | |
399 | for each container a sibling cgroup of that maybe called `payload-xyz/`. | |
400 | ||
401 | 2. ⚡ Don't forget that the cgroups you create have to have names that are | |
402 | suitable as UNIX file names, and that they live in the same namespace as the | |
403 | various kernel attribute files. Hence, when you want to allow the user | |
404 | arbitrary naming, you might need to escape some of the names (for example, | |
405 | you really don't want to create a cgroup named `tasks`, just because the | |
406 | user created a container by that name, because `tasks` after all is a magic | |
407 | attribute in cgroupsv1, and your `mkdir()` will hence fail with `EEXIST`. In | |
408 | systemd we do escaping by prefixing names that might collide with a kernel | |
409 | attribute name with an underscore. You might want to do the same, but this | |
410 | is really up to you how you do it. Just do it, and be careful. | |
411 | ||
412 | ## Some Don'ts | |
413 | ||
414 | 1. 🚫 Never create your own cgroups below arbitrary cgroups systemd manages, i.e | |
415 | cgroups you haven't set `Delegate=` in. Specifically: 🔥 don't create your | |
416 | own cgroups below the root cgroup 🔥. That's owned by systemd, and you will | |
417 | step on systemd's toes if you ignore that, and systemd will step on | |
418 | yours. Get your own delegated sub-tree, you may create as many cgroups there | |
419 | as you like. Seriously, if you create cgroups directly in the cgroup root, | |
420 | then all you do is ask for trouble. | |
421 | ||
422 | 2. 🚫 Don't attempt to set `Delegate=` in slice units, and in particular not in | |
423 | `-.slice`. It's not supported, and will generate an error. | |
424 | ||
425 | 3. 🚫 Never *write* to any of the attributes of a cgroup systemd created for | |
426 | you. It's systemd's private property. You are welcome to manipulate the | |
427 | attributes of cgroups you created in your own delegated sub-tree, but the | |
428 | cgroup tree of systemd itself is out of limits for you. It's fine to *read* | |
429 | from any attribute you like however. That's totally OK and welcome. | |
430 | ||
d11623e9 LP |
431 | 4. 🚫 When not using `CLONE_NEWCGROUP` when delegating a sub-tree to a |
432 | container payload running systemd, then don't get the idea that you can bind | |
433 | mount only a sub-tree of the host's cgroup tree into the container. Part of | |
434 | the cgroup API is that `/proc/$PID/cgroup` reports the cgroup path of every | |
e30eaff3 LP |
435 | process, and hence any path below `/sys/fs/cgroup/` needs to match what |
436 | `/proc/$PID/cgroup` of the payload processes reports. What you can do safely | |
d11623e9 LP |
437 | however, is mount the upper parts of the cgroup tree read-only (or even |
438 | replace the middle bits with an intermediary `tmpfs` — but be careful not to | |
439 | break the `statfs()` detection logic discussed above), as long as the path | |
440 | to the delegated sub-tree remains accessible as-is. | |
e30eaff3 | 441 | |
3ee9b2f6 LP |
442 | 5. ⚡ Currently, the algorithm for mapping between slice/scope/service unit |
443 | naming and their cgroup paths is not considered public API of systemd, and | |
444 | may change in future versions. This means: it's best to avoid implementing a | |
445 | local logic of translating cgroup paths to slice/scope/service names in your | |
446 | program, or vice versa — it's likely going to break sooner or later. Use the | |
447 | appropriate D-Bus API calls for that instead, so that systemd translates | |
448 | this for you. (Specifically: each Unit object has a `ControlGroup` property | |
449 | to get the cgroup for a unit. The method `GetUnitByControlGroup()` may be | |
450 | used to get the unit for a cgroup.) | |
451 | ||
452 | 6. ⚡ Think twice before delegating cgroupsv1 controllers to less privileged | |
e30eaff3 LP |
453 | containers. It's not safe, you basically allow your containers to freeze the |
454 | system with that and worse. Delegation is a strongpoint of cgroupsv2 though, | |
455 | and there it's safe to treat delegation boundaries as privilege boundaries. | |
456 | ||
457 | And that's it for now. If you have further questions, refer to the systemd | |
458 | mailing list. | |
459 | ||
460 | — Berlin, 2018-04-20 |