]>
Commit | Line | Data |
---|---|---|
c3e270f4 FB |
1 | --- |
2 | title: Control Group APIs and Delegation | |
4cdca0af | 3 | category: Interfaces |
b41a3f66 | 4 | layout: default |
c3e270f4 FB |
5 | --- |
6 | ||
e30eaff3 LP |
7 | # Control Group APIs and Delegation |
8 | ||
1e46eb59 LP |
9 | *Intended audience: hackers working on userspace subsystems that require direct |
10 | cgroup access, such as container managers and similar.* | |
11 | ||
e30eaff3 LP |
12 | So you are wondering about resource management with systemd, you know Linux |
13 | control groups (cgroups) a bit and are trying to integrate your software with | |
14 | what systemd has to offer there. Here's a bit of documentation about the | |
15 | concepts and interfaces involved with this. | |
16 | ||
17 | What's described here has been part of systemd and documented since v205 | |
5b24525a | 18 | times. However, it has been updated and improved substantially, even |
e30eaff3 LP |
19 | though the concepts stayed mostly the same. This is an attempt to provide more |
20 | comprehensive up-to-date information about all this, particular in light of the | |
21 | poor implementations of the components interfacing with systemd of current | |
22 | container managers. | |
23 | ||
bb6d563a ZJS |
24 | Before you read on, please make sure you read the low-level kernel |
25 | documentation about the | |
26 | [unified cgroup hierarchy](https://www.kernel.org/doc/html/latest/admin-guide/cgroup-v2.html). | |
27 | This document then adds in the higher-level view from systemd. | |
e30eaff3 LP |
28 | |
29 | This document augments the existing documentation we already have: | |
30 | ||
31 | * [The New Control Group Interfaces](https://www.freedesktop.org/wiki/Software/systemd/ControlGroupInterface/) | |
32 | * [Writing VM and Container Managers](https://www.freedesktop.org/wiki/Software/systemd/writing-vm-managers/) | |
33 | ||
34 | These wiki documents are not as up to date as they should be, currently, but | |
35 | the basic concepts still fully apply. You should read them too, if you do something | |
36 | with cgroups and systemd, in particular as they shine more light on the various | |
37 | D-Bus APIs provided. (That said, sooner or later we should probably fold that | |
38 | wiki documentation into this very document, too.) | |
39 | ||
40 | ## Two Key Design Rules | |
41 | ||
42 | Much of the philosophy behind these concepts is based on a couple of basic | |
4e1dfa45 CD |
43 | design ideas of cgroup v2 (which we however try to adapt as far as we can to |
44 | cgroup v1 too). Specifically two cgroup v2 rules are the most relevant: | |
e30eaff3 LP |
45 | |
46 | 1. The **no-processes-in-inner-nodes** rule: this means that it's not permitted | |
47 | to have processes directly attached to a cgroup that also has child cgroups and | |
48 | vice versa. A cgroup is either an inner node or a leaf node of the tree, and if | |
49 | it's an inner node it may not contain processes directly, and if it's a leaf | |
50 | node then it may not have child cgroups. (Note that there are some minor | |
5b24525a | 51 | exceptions to this rule, though. E.g. the root cgroup is special and allows |
e30eaff3 LP |
52 | both processes and children — which is used in particular to maintain kernel |
53 | threads.) | |
54 | ||
55 | 2. The **single-writer** rule: this means that each cgroup only has a single | |
56 | writer, i.e. a single process managing it. It's OK if different cgroups have | |
57 | different processes managing them. However, only a single process should own a | |
58 | specific cgroup, and when it does that ownership is exclusive, and nothing else | |
59 | should manipulate it at the same time. This rule ensures that various pieces of | |
60 | software don't step on each other's toes constantly. | |
61 | ||
62 | These two rules have various effects. For example, one corollary of this is: if | |
63 | your container manager creates and manages cgroups in the system's root cgroup | |
64 | you violate rule #2, as the root cgroup is managed by systemd and hence off | |
65 | limits to everybody else. | |
66 | ||
4e1dfa45 | 67 | Note that rule #1 is generally enforced by the kernel if cgroup v2 is used: as |
e30eaff3 | 68 | soon as you add a process to a cgroup it is ensured the rule is not |
4e1dfa45 | 69 | violated. On cgroup v1 this rule didn't exist, and hence isn't enforced, even |
e30eaff3 | 70 | though it's a good thing to follow it then too. Rule #2 is not enforced on |
4e1dfa45 | 71 | either cgroup v1 nor cgroup v2 (this is UNIX after all, in the general case |
e30eaff3 LP |
72 | root can do anything, modulo SELinux and friends), but if you ignore it you'll |
73 | be in constant pain as various pieces of software will fight over cgroup | |
74 | ownership. | |
75 | ||
4e1dfa45 | 76 | Note that cgroup v1 is currently the most deployed implementation, even though |
5b24525a | 77 | it's semantically broken in many ways, and in many cases doesn't actually do |
4e1dfa45 CD |
78 | what people think it does. cgroup v2 is where things are going, and most new |
79 | kernel features in this area are only added to cgroup v2, and not cgroup v1 | |
80 | anymore. For example cgroup v2 provides proper cgroup-empty notifications, has | |
5b24525a ZJS |
81 | support for all kinds of per-cgroup BPF magic, supports secure delegation of |
82 | cgroup trees to less privileged processes and so on, which all are not | |
4e1dfa45 | 83 | available on cgroup v1. |
e30eaff3 LP |
84 | |
85 | ## Three Different Tree Setups 🌳 | |
86 | ||
87 | systemd supports three different modes how cgroups are set up. Specifically: | |
88 | ||
4e1dfa45 | 89 | 1. **Unified** — this is the simplest mode, and exposes a pure cgroup v2 |
e30eaff3 LP |
90 | logic. In this mode `/sys/fs/cgroup` is the only mounted cgroup API file system |
91 | and all available controllers are exclusively exposed through it. | |
92 | ||
4e1dfa45 | 93 | 2. **Legacy** — this is the traditional cgroup v1 mode. In this mode the |
e30eaff3 LP |
94 | various controllers each get their own cgroup file system mounted to |
95 | `/sys/fs/cgroup/<controller>/`. On top of that systemd manages its own cgroup | |
96 | hierarchy for managing purposes as `/sys/fs/cgroup/systemd/`. | |
97 | ||
98 | 3. **Hybrid** — this is a hybrid between the unified and legacy mode. It's set | |
99 | up mostly like legacy, except that there's also an additional hierarchy | |
4e1dfa45 | 100 | `/sys/fs/cgroup/unified/` that contains the cgroup v2 hierarchy. (Note that in |
9afd5740 LP |
101 | this mode the unified hierarchy won't have controllers attached, the |
102 | controllers are all mounted as separate hierarchies as in legacy mode, | |
4e1dfa45 | 103 | i.e. `/sys/fs/cgroup/unified/` is purely and exclusively about core cgroup v2 |
9afd5740 | 104 | functionality and not about resource management.) In this mode compatibility |
4e1dfa45 | 105 | with cgroup v1 is retained while some cgroup v2 features are available |
9afd5740 LP |
106 | too. This mode is a stopgap. Don't bother with this too much unless you have |
107 | too much free time. | |
e30eaff3 LP |
108 | |
109 | To say this clearly, legacy and hybrid modes have no future. If you develop | |
110 | software today and don't focus on the unified mode, then you are writing | |
111 | software for yesterday, not tomorrow. They are primarily supported for | |
112 | compatibility reasons and will not receive new features. Sorry. | |
113 | ||
114 | Superficially, in legacy and hybrid modes it might appear that the parallel | |
115 | cgroup hierarchies for each controller are orthogonal from each other. In | |
116 | systemd they are not: the hierarchies of all controllers are always kept in | |
117 | sync (at least mostly: sub-trees might be suppressed in certain hierarchies if | |
118 | no controller usage is required for them). The fact that systemd keeps these | |
119 | hierarchies in sync means that the legacy and hybrid hierarchies are | |
120 | conceptually very close to the unified hierarchy. In particular this allows us | |
5b24525a ZJS |
121 | to talk of one specific cgroup and actually mean the same cgroup in all |
122 | available controller hierarchies. E.g. if we talk about the cgroup `/foo/bar/` | |
123 | then we actually mean `/sys/fs/cgroup/cpu/foo/bar/` as well as | |
124 | `/sys/fs/cgroup/memory/foo/bar/`, `/sys/fs/cgroup/pids/foo/bar/`, and so on. | |
4e1dfa45 | 125 | Note that in cgroup v2 the controller hierarchies aren't orthogonal, hence |
e30eaff3 LP |
126 | thinking about them as orthogonal won't help you in the long run anyway. |
127 | ||
128 | If you wonder how to detect which of these three modes is currently used, use | |
129 | `statfs()` on `/sys/fs/cgroup/`. If it reports `CGROUP2_SUPER_MAGIC` in its | |
130 | `.f_type` field, then you are in unified mode. If it reports `TMPFS_MAGIC` then | |
b2454670 | 131 | you are either in legacy or hybrid mode. To distinguish these two cases, run |
e30eaff3 LP |
132 | `statfs()` again on `/sys/fs/cgroup/unified/`. If that succeeds and reports |
133 | `CGROUP2_SUPER_MAGIC` you are in hybrid mode, otherwise not. | |
134 | ||
135 | ## systemd's Unit Types | |
136 | ||
137 | The low-level kernel cgroups feature is exposed in systemd in three different | |
138 | "unit" types. Specifically: | |
139 | ||
140 | 1. 💼 The `.service` unit type. This unit type is for units encapsulating | |
141 | processes systemd itself starts. Units of these types have cgroups that are | |
142 | the leaves of the cgroup tree the systemd instance manages (though possibly | |
143 | they might contain a sub-tree of their own managed by something else, made | |
144 | possible by the concept of delegation, see below). Service units are usually | |
145 | instantiated based on a unit file on disk that describes the command line to | |
146 | invoke and other properties of the service. However, service units may also | |
147 | be declared and started programmatically at runtime through a D-Bus API | |
148 | (which is called *transient* services). | |
149 | ||
150 | 2. 👓 The `.scope` unit type. This is very similar to `.service`. The main | |
151 | difference: the processes the units of this type encapsulate are forked off | |
152 | by some unrelated manager process, and that manager asked systemd to expose | |
153 | them as a unit. Unlike services, scopes can only be declared and started | |
154 | programmatically, i.e. are always transient. That's because they encapsulate | |
155 | processes forked off by something else, i.e. existing runtime objects, and | |
156 | hence cannot really be defined fully in 'offline' concepts such as unit | |
157 | files. | |
158 | ||
159 | 3. 🔪 The `.slice` unit type. Units of this type do not directly contain any | |
160 | processes. Units of this type are the inner nodes of part of the cgroup tree | |
161 | the systemd instance manages. Much like services, slices can be defined | |
162 | either on disk with unit files or programmatically as transient units. | |
163 | ||
164 | Slices expose the trunk and branches of a tree, and scopes and services are | |
165 | attached to those branches as leaves. The idea is that scopes and services can | |
166 | be moved around though, i.e. assigned to a different slice if needed. | |
167 | ||
168 | The naming of slice units directly maps to the cgroup tree path. This is not | |
169 | the case for service and scope units however. A slice named `foo-bar-baz.slice` | |
170 | maps to a cgroup `/foo.slice/foo-bar.slice/foo-bar-baz.slice/`. A service | |
171 | `quux.service` which is attached to the slice `foo-bar-baz.slice` maps to the | |
172 | cgroup `/foo.slice/foo-bar.slice/foo-bar-baz.slice/quux.service/`. | |
173 | ||
174 | By default systemd sets up four slice units: | |
175 | ||
176 | 1. `-.slice` is the root slice. i.e. the parent of everything else. On the host | |
4e1dfa45 | 177 | system it maps directly to the top-level directory of cgroup v2. |
e30eaff3 LP |
178 | |
179 | 2. `system.slice` is where system services are by default placed, unless | |
180 | configured otherwise. | |
181 | ||
182 | 3. `user.slice` is where user sessions are placed. Each user gets a slice of | |
183 | its own below that. | |
184 | ||
185 | 4. `machines.slice` is where VMs and containers are supposed to be | |
186 | placed. `systemd-nspawn` makes use of this by default, and you're very welcome | |
187 | to place your containers and VMs there too if you hack on managers for those. | |
188 | ||
189 | Users may define any amount of additional slices they like though, the four | |
190 | above are just the defaults. | |
191 | ||
192 | ## Delegation | |
193 | ||
194 | Container managers and suchlike often want to control cgroups directly using | |
195 | the raw kernel APIs. That's entirely fine and supported, as long as proper | |
4e1dfa45 CD |
196 | *delegation* is followed. Delegation is a concept we inherited from cgroup v2, |
197 | but we expose it on cgroup v1 too. Delegation means that some parts of the | |
e30eaff3 LP |
198 | cgroup tree may be managed by different managers than others. As long as it is |
199 | clear which manager manages which part of the tree each one can do within its | |
200 | sub-graph of the tree whatever it wants. | |
201 | ||
202 | Only sub-trees can be delegated (though whoever decides to request a sub-tree | |
5b24525a ZJS |
203 | can delegate sub-sub-trees further to somebody else if they like). Delegation |
204 | takes place at a specific cgroup: in systemd there's a `Delegate=` property you | |
205 | can set for a service or scope unit. If you do, it's the cut-off point for | |
206 | systemd's cgroup management: the unit itself is managed by systemd, i.e. all | |
207 | its attributes are managed exclusively by systemd, however your program may | |
208 | create/remove sub-cgroups inside it freely, and those then become exclusive | |
209 | property of your program, systemd won't touch them — all attributes of *those* | |
210 | sub-cgroups can be manipulated freely and exclusively by your program. | |
e30eaff3 LP |
211 | |
212 | By turning on the `Delegate=` property for a scope or service you get a few | |
213 | guarantees: | |
214 | ||
215 | 1. systemd won't fiddle with your sub-tree of the cgroup tree anymore. It won't | |
216 | change attributes of any cgroups below it, nor will it create or remove any | |
217 | cgroups thereunder, nor migrate processes across the boundaries of that | |
218 | sub-tree as it deems useful anymore. | |
219 | ||
220 | 2. If your service makes use of the `User=` functionality, then the sub-tree | |
221 | will be `chown()`ed to the indicated user so that it can correctly create | |
222 | cgroups below it. Note however that systemd will do that only in the unified | |
223 | hierarchy (in unified and hybrid mode) as well as on systemd's own private | |
224 | hierarchy (in legacy and hybrid mode). It won't pass ownership of the legacy | |
225 | controller hierarchies. Delegation to less privileges processes is not safe | |
4e1dfa45 | 226 | in cgroup v1 (as a limitation of the kernel), hence systemd won't facilitate |
e30eaff3 LP |
227 | access to it. |
228 | ||
229 | 3. Any BPF IP filter programs systemd installs will be installed with | |
230 | `BPF_F_ALLOW_MULTI` so that your program can install additional ones. | |
231 | ||
232 | In unit files the `Delegate=` property is superficially exposed as | |
233 | boolean. However, since v236 it optionally takes a list of controller names | |
234 | instead. If so, delegation is requested for listed controllers | |
1a31d050 | 235 | specifically. Note that this only encodes a request. Depending on various |
e30eaff3 LP |
236 | parameters it might happen that your service actually will get fewer |
237 | controllers delegated (for example, because the controller is not available on | |
238 | the current kernel or was turned off) or more. If no list is specified | |
239 | (i.e. the property simply set to `yes`) then all available controllers are | |
240 | delegated. | |
241 | ||
242 | Let's stress one thing: delegation is available on scope and service units | |
5b24525a ZJS |
243 | only. It's expressly not available on slice units. Why? Because slice units are |
244 | our *inner* nodes of the cgroup trees and we freely attach service and scopes | |
e5988600 | 245 | to them. If we'd allow delegation on slice units then this would mean that |
5b24525a ZJS |
246 | both systemd and your own manager would create/delete cgroups below the slice |
247 | unit and that conflicts with the single-writer rule. | |
e30eaff3 LP |
248 | |
249 | So, if you want to do your own raw cgroups kernel level access, then allocate a | |
250 | scope unit, or a service unit (or just use the service unit you already have | |
251 | for your service code), and turn on delegation for it. | |
252 | ||
e2391ce0 LP |
253 | (OK, here's one caveat: if you turn on delegation for a service, and that |
254 | service has `ExecStartPost=`, `ExecReload=`, `ExecStop=` or `ExecStopPost=` | |
255 | set, then these commands will be executed within the `.control/` sub-cgroup of | |
256 | your service's cgroup. This is necessary because by turning on delegation we | |
257 | have to assume that the cgroup delegated to your service is now an *inner* | |
258 | cgroup, which means that it may not directly contain any processes. Hence, if | |
259 | your service has any of these four settings set, you must be prepared that a | |
260 | `.control/` subcgroup might appear, managed by the service manager. This also | |
261 | means that your service code should have moved itself further down the cgroup | |
262 | tree by the time it notifies the service manager about start-up readiness, so | |
263 | that the service's main cgroup is definitely an inner node by the time the | |
264 | service manager might start `ExecStartPost=`.) | |
265 | ||
e30eaff3 LP |
266 | ## Three Scenarios |
267 | ||
268 | Let's say you write a container manager, and you wonder what to do regarding | |
269 | cgroups for it, as you want your manager to be able to run on systemd systems. | |
270 | ||
271 | You basically have three options: | |
272 | ||
5b24525a ZJS |
273 | 1. 😊 The *integration-is-good* option. For this, you register each container |
274 | you have either as a systemd service (i.e. let systemd invoke the executor | |
275 | binary for you) or a systemd scope (i.e. your manager executes the binary | |
276 | directly, but then tells systemd about it. In this mode the administrator | |
277 | can use the usual systemd resource management and reporting commands | |
278 | individually on those containers. By turning on `Delegate=` for these scopes | |
279 | or services you make it possible to run cgroup-enabled programs in your | |
280 | containers, for example a nested systemd instance. This option has two | |
281 | sub-options: | |
e30eaff3 | 282 | |
5b24525a ZJS |
283 | a. You transiently register the service or scope by directly contacting |
284 | systemd via D-Bus. In this case systemd will just manage the unit for you | |
285 | and nothing else. | |
e30eaff3 LP |
286 | |
287 | b. Instead you register the service or scope through `systemd-machined` | |
288 | (also via D-Bus). This mini-daemon is basically just a proxy for the same | |
289 | operations as in a. The main benefit of this: this way you let the system | |
290 | know that what you are registering is a container, and this opens up | |
291 | certain additional integration points. For example, `journalctl -M` can | |
292 | then be used to directly look into any container's journal logs (should | |
293 | the container run systemd inside), or `systemctl -M` can be used to | |
294 | directly invoke systemd operations inside the containers. Moreover tools | |
295 | like "ps" can then show you to which container a process belongs (`ps -eo | |
296 | pid,comm,machine`), and even gnome-system-monitor supports it. | |
297 | ||
298 | 2. 🙁 The *i-like-islands* option. If all you care about is your own cgroup tree, | |
299 | and you want to have to do as little as possible with systemd and no | |
300 | interest in integration with the rest of the system, then this is a valid | |
301 | option. For this all you have to do is turn on `Delegate=` for your main | |
302 | manager daemon. Then figure out the cgroup systemd placed your daemon in: | |
303 | you can now freely create sub-cgroups beneath it. Don't forget the | |
304 | *no-processes-in-inner-nodes* rule however: you have to move your main | |
305 | daemon process out of that cgroup (and into a sub-cgroup) before you can | |
306 | start further processes in any of your sub-cgroups. | |
307 | ||
308 | 3. 🙁 The *i-like-continents* option. In this option you'd leave your manager | |
309 | daemon where it is, and would not turn on delegation on its unit. However, | |
310 | as first thing you register a new scope unit with systemd, and that scope | |
311 | unit would have `Delegate=` turned on, and then you place all your | |
312 | containers underneath it. From systemd's PoV there'd be two units: your | |
313 | manager service and the big scope that contains all your containers in one. | |
314 | ||
315 | BTW: if for whatever reason you say "I hate D-Bus, I'll never call any D-Bus | |
316 | API, kthxbye", then options #1 and #3 are not available, as they generally | |
317 | involve talking to systemd from your program code, via D-Bus. You still have | |
318 | option #2 in that case however, as you can simply set `Delegate=` in your | |
319 | service's unit file and you are done and have your own sub-tree. In fact, #2 is | |
320 | the one option that allows you to completely ignore systemd's existence: you | |
321 | can entirely generically follow the single rule that you just use the cgroup | |
322 | you are started in, and everything below it, whatever that might be. That said, | |
323 | maybe if you dislike D-Bus and systemd that much, the better approach might be | |
324 | to work on that, and widen your horizon a bit. You are welcome. | |
325 | ||
326 | ## Controller Support | |
327 | ||
328 | systemd supports a number of controllers (but not all). Specifically, supported | |
329 | are: | |
330 | ||
4e1dfa45 CD |
331 | * on cgroup v1: `cpu`, `cpuacct`, `blkio`, `memory`, `devices`, `pids` |
332 | * on cgroup v2: `cpu`, `io`, `memory`, `pids` | |
e30eaff3 | 333 | |
4e1dfa45 CD |
334 | It is our intention to natively support all cgroup v2 controllers as they are |
335 | added to the kernel. However, regarding cgroup v1: at this point we will not | |
5b24525a | 336 | add support for any other controllers anymore. This means systemd currently |
4e1dfa45 | 337 | does not and will never manage the following controllers on cgroup v1: |
e30eaff3 LP |
338 | `freezer`, `cpuset`, `net_cls`, `perf_event`, `net_prio`, `hugetlb`. Why not? |
339 | Depending on the case, either their API semantics or implementations aren't | |
4e1dfa45 | 340 | really usable, or it's very clear they have no future on cgroup v2, and we |
e30eaff3 LP |
341 | won't add new code for stuff that clearly has no future. |
342 | ||
4e1dfa45 | 343 | Effectively this means that all those mentioned cgroup v1 controllers are up |
e30eaff3 LP |
344 | for grabs: systemd won't manage them, and hence won't delegate them to your |
345 | code (however, systemd will still mount their hierarchies, simply because it | |
346 | mounts all controller hierarchies it finds available in the kernel). If you | |
347 | decide to use them, then that's fine, but systemd won't help you with it (but | |
348 | also not interfere with it). To be nice to other tenants it might be wise to | |
349 | replicate the cgroup hierarchies of the other controllers in them too however, | |
350 | but of course that's between you and those other tenants, and systemd won't | |
351 | care. Replicating the cgroup hierarchies in those unsupported controllers would | |
352 | mean replicating the full cgroup paths in them, and hence the prefixing | |
353 | `.slice` components too, otherwise the hierarchies will start being orthogonal | |
354 | after all, and that's not really desirable. On more thing: systemd will clean | |
355 | up after you in the hierarchies it manages: if your daemon goes down, its | |
356 | cgroups will be removed too. You basically get the guarantee that you start | |
357 | with a pristine cgroup sub-tree for your service or scope whenever it is | |
358 | started. This is not the case however in the hierarchies systemd doesn't | |
359 | manage. This means that your programs should be ready to deal with left-over | |
360 | cgroups in them — from previous runs, and be extra careful with them as they | |
361 | might still carry settings that might not be valid anymore. | |
362 | ||
363 | Note a particular asymmetry here: if your systemd version doesn't support a | |
4e1dfa45 | 364 | specific controller on cgroup v1 you can still make use of it for delegation, |
e30eaff3 | 365 | by directly fiddling with its hierarchy and replicating the cgroup tree there |
4e1dfa45 | 366 | as necessary (as suggested above). However, on cgroup v2 this is different: |
e30eaff3 LP |
367 | separately mounted hierarchies are not available, and delegation has always to |
368 | happen through systemd itself. This means: when you update your kernel and it | |
369 | adds a new, so far unseen controller, and you want to use it for delegation, | |
370 | then you also need to update systemd to a version that groks it. | |
371 | ||
372 | ## systemd as Container Payload | |
373 | ||
374 | systemd can happily run as a container payload's PID 1. Note that systemd | |
375 | unconditionally needs write access to the cgroup tree however, hence you need | |
376 | to delegate a sub-tree to it. Note that there's nothing too special you have to | |
377 | do beyond that: just invoke systemd as PID 1 inside the root of the delegated | |
378 | cgroup sub-tree, and it will figure out the rest: it will determine the cgroup | |
379 | it is running in and take possession of it. It won't interfere with any cgroup | |
380 | outside of the sub-tree it was invoked in. Use of `CLONE_NEWCGROUP` is hence | |
381 | optional (but of course wise). | |
382 | ||
383 | Note one particular asymmetry here though: systemd will try to take possession | |
384 | of the root cgroup you pass to it *in* *full*, i.e. it will not only | |
e5988600 | 385 | create/remove child cgroups below it, it will also attempt to manage the |
e30eaff3 LP |
386 | attributes of it. OTOH as mentioned above, when delegating a cgroup tree to |
387 | somebody else it only passes the rights to create/remove sub-cgroups, but will | |
388 | insist on managing the delegated cgroup tree's top-level attributes. Or in | |
389 | other words: systemd is *greedy* when accepting delegated cgroup trees and also | |
390 | *greedy* when delegating them to others: it insists on managing attributes on | |
391 | the specific cgroup in both cases. A container manager that is itself a payload | |
392 | of a host systemd which wants to run a systemd as its own container payload | |
393 | instead hence needs to insert an extra level in the hierarchy in between, so | |
394 | that the systemd on the host and the one in the container won't fight for the | |
395 | attributes. That said, you likely should do that anyway, due to the | |
396 | no-processes-in-inner-cgroups rule, see below. | |
397 | ||
398 | When systemd runs as container payload it will make use of all hierarchies it | |
399 | has write access to. For legacy mode you need to make at least | |
400 | `/sys/fs/cgroup/systemd/` available, all other hierarchies are optional. For | |
401 | hybrid mode you need to add `/sys/fs/cgroup/unified/`. Finally, for fully | |
402 | unified you (of course, I guess) need to provide only `/sys/fs/cgroup/` itself. | |
403 | ||
404 | ## Some Dos | |
405 | ||
406 | 1. ⚡ If you go for implementation option 1a or 1b (as in the list above), then | |
407 | each of your containers will have its own systemd-managed unit and hence | |
408 | cgroup with possibly further sub-cgroups below. Typically the first process | |
409 | running in that unit will be some kind of executor program, which will in | |
410 | turn fork off the payload processes of the container. In this case don't | |
411 | forget that there are two levels of delegation involved: first, systemd | |
412 | delegates a group sub-tree to your executor. And then your executor should | |
413 | delegate a sub-tree further down to the container payload. Oh, and because | |
414 | of the no-process-in-inner-nodes rule, your executor needs to migrate itself | |
415 | to a sub-cgroup of the cgroup it got delegated, too. Most likely you hence | |
416 | want a two-pronged approach: below the cgroup you got started in, you want | |
417 | one cgroup maybe called `supervisor/` where your manager runs in and then | |
418 | for each container a sibling cgroup of that maybe called `payload-xyz/`. | |
419 | ||
420 | 2. ⚡ Don't forget that the cgroups you create have to have names that are | |
421 | suitable as UNIX file names, and that they live in the same namespace as the | |
422 | various kernel attribute files. Hence, when you want to allow the user | |
423 | arbitrary naming, you might need to escape some of the names (for example, | |
424 | you really don't want to create a cgroup named `tasks`, just because the | |
425 | user created a container by that name, because `tasks` after all is a magic | |
4e1dfa45 | 426 | attribute in cgroup v1, and your `mkdir()` will hence fail with `EEXIST`. In |
e30eaff3 LP |
427 | systemd we do escaping by prefixing names that might collide with a kernel |
428 | attribute name with an underscore. You might want to do the same, but this | |
429 | is really up to you how you do it. Just do it, and be careful. | |
430 | ||
431 | ## Some Don'ts | |
432 | ||
433 | 1. 🚫 Never create your own cgroups below arbitrary cgroups systemd manages, i.e | |
434 | cgroups you haven't set `Delegate=` in. Specifically: 🔥 don't create your | |
435 | own cgroups below the root cgroup 🔥. That's owned by systemd, and you will | |
436 | step on systemd's toes if you ignore that, and systemd will step on | |
437 | yours. Get your own delegated sub-tree, you may create as many cgroups there | |
438 | as you like. Seriously, if you create cgroups directly in the cgroup root, | |
439 | then all you do is ask for trouble. | |
440 | ||
441 | 2. 🚫 Don't attempt to set `Delegate=` in slice units, and in particular not in | |
442 | `-.slice`. It's not supported, and will generate an error. | |
443 | ||
444 | 3. 🚫 Never *write* to any of the attributes of a cgroup systemd created for | |
445 | you. It's systemd's private property. You are welcome to manipulate the | |
446 | attributes of cgroups you created in your own delegated sub-tree, but the | |
447 | cgroup tree of systemd itself is out of limits for you. It's fine to *read* | |
448 | from any attribute you like however. That's totally OK and welcome. | |
449 | ||
d11623e9 LP |
450 | 4. 🚫 When not using `CLONE_NEWCGROUP` when delegating a sub-tree to a |
451 | container payload running systemd, then don't get the idea that you can bind | |
452 | mount only a sub-tree of the host's cgroup tree into the container. Part of | |
453 | the cgroup API is that `/proc/$PID/cgroup` reports the cgroup path of every | |
e30eaff3 LP |
454 | process, and hence any path below `/sys/fs/cgroup/` needs to match what |
455 | `/proc/$PID/cgroup` of the payload processes reports. What you can do safely | |
d11623e9 LP |
456 | however, is mount the upper parts of the cgroup tree read-only (or even |
457 | replace the middle bits with an intermediary `tmpfs` — but be careful not to | |
458 | break the `statfs()` detection logic discussed above), as long as the path | |
459 | to the delegated sub-tree remains accessible as-is. | |
e30eaff3 | 460 | |
3ee9b2f6 LP |
461 | 5. ⚡ Currently, the algorithm for mapping between slice/scope/service unit |
462 | naming and their cgroup paths is not considered public API of systemd, and | |
463 | may change in future versions. This means: it's best to avoid implementing a | |
464 | local logic of translating cgroup paths to slice/scope/service names in your | |
465 | program, or vice versa — it's likely going to break sooner or later. Use the | |
466 | appropriate D-Bus API calls for that instead, so that systemd translates | |
467 | this for you. (Specifically: each Unit object has a `ControlGroup` property | |
468 | to get the cgroup for a unit. The method `GetUnitByControlGroup()` may be | |
469 | used to get the unit for a cgroup.) | |
470 | ||
4e1dfa45 | 471 | 6. ⚡ Think twice before delegating cgroup v1 controllers to less privileged |
e30eaff3 | 472 | containers. It's not safe, you basically allow your containers to freeze the |
4e1dfa45 | 473 | system with that and worse. Delegation is a strongpoint of cgroup v2 though, |
e30eaff3 LP |
474 | and there it's safe to treat delegation boundaries as privilege boundaries. |
475 | ||
476 | And that's it for now. If you have further questions, refer to the systemd | |
477 | mailing list. | |
478 | ||
479 | — Berlin, 2018-04-20 |