From: Tom Hromatka Date: Wed, 10 May 2023 20:37:14 +0000 (-0600) Subject: samples: Add systemd command line example X-Git-Tag: v3.1.0~67 X-Git-Url: http://git.ipfire.org/cgi-bin/gitweb.cgi?a=commitdiff_plain;h=bdbcd2ceae0c367f6ea6d05dbad13ba7058adcf2;p=thirdparty%2Flibcgroup.git samples: Add systemd command line example Signed-off-by: Tom Hromatka Reviewed-by: Kamalesh Babulal --- diff --git a/samples/cmdline/systemd-with-idle-process.md b/samples/cmdline/systemd-with-idle-process.md new file mode 100644 index 00000000..8e50e04b --- /dev/null +++ b/samples/cmdline/systemd-with-idle-process.md @@ -0,0 +1,219 @@ +_SPDX-License-Identifier: LGPL-2.1-only_ + +_Copyright (c) 2023 Oracle and/or its affiliates._ + +_Author: Tom Hromatka <>_ + + +# Creating a Systemd Scope and Child Hierarchy via Libcgroup Command Line + +The goal of this document is to outline the steps required to create +a systemd scope and a child cgroup hierarchy using the libcgroup +command line tools. + +The following steps are encapsulated in a +[libcgroup automated test](../../tests/ftests/086-sudo-systemd_cmdline_example.py). + +## Requirements: +1. Create a cgroup hierarchy that obeys the + [single-writer rule](https://systemd.io/CGROUP_DELEGATION/) +1. The hierarchy should be cgroup v2 +1. The hierarchy should be of the form + ``` + database.scope + ├── high-priority + ├── low-priority + └── medium-priority + ``` +1. The memory controller should be enabled for the entire tree + + 1. The `high-priority` cgroup should be guaranteed at least 1GB of RAM + 1. The `low-priority` cgroup should be hard-limited to 2GB of RAM + 1. The `medium-prioirty` cgroup should have a soft memory limit of 3 GB + +1. The cpu controller should be enabled for the entire tree + + 1. The `high-priority` cgroup should be able to consume 60% of CPU cycles + 1. The `medium-priority` cgroup should be able to consume 30% of CPU cycles + 1. The `low-priority` cgroup should be able to consume 10% of CPU cycles + +## Steps + +1. Create a delegated + [scope](https://www.freedesktop.org/software/systemd/man/systemd.scope.html) + cgroup + ``` + sudo cgcreate -c -S -g cpu,memory:mycompany.slice/database.scope + ``` + + This will create a + [transient](https://github.com/systemd/systemd/blob/main/docs/TRANSIENT-SETTINGS.md), + [delegated](https://systemd.io/CGROUP_DELEGATION/) scope. The `-c` flag + instructs libcgroup to create a systemd scope; libcgroup then instructs + systemd that this hierarchy is delegated, i.e. it is to be managed by + another process and _not_ by systemd. The `-S` flag notifies libcgroup + that we want `mycompany.scope/database.slice` to be the default base + path in libcgroup; this will significantly help in reducing typing in + follow-on commands. The `-g` flag tells libcgroup to create a cgroup + named `mycompany.slice/database.scope` and enable the cpu and memory + controllers within it. + +
+ Problems during this step? + + Systemd should automatically remove scopes with no active processes + running within them. So, the first step would be to kill any processes + in the scope, wait to see if systemd removes the scope, and then try the + `cgcreate` operation again. + + - Remove all processes in the scope + ``` + $ for PID in $(cgget -nvb -r cgroup.procs mycompany.slice/database.scope); do sudo kill -9 $PID;done + ``` + + The above command could be simplified as + `for PID in $(cgget -nv -r cgroup.procs /); do sudo kill -9 $PID;done`, + but introduces some risk. If there is a typo or the default scope path + isn't set, then unconditionally killing processes in `/` could be + catastrophic. + + Sometimes systemd's internal list of scopes gets out of sync with the + filesystem. You can purge the `database.scope` from its list by running + the following commands + - Remove `database.scope` from systemd's internal list + ``` + sudo systemctl kill database.scope + sudo systemctl stop database.scope + ``` +
+ +1. Create the child cgroups + ``` + $ sudo cgcreate -g cpu,memory:mycompany.slice/database.scope/high-priority + cgcreate: can't create cgroup mycompany.slice/database.scope/high-priority: Operation not supported + ``` + + But... but... I did everything right. Why can't I create the + `high-priority` child cgroup? This operation failed due to the + [no-processes-in-inner-nodes](https://systemd.io/CGROUP_DELEGATION/) rule. + Since a process, `libcgroup_systemd_idle_thread`, resides in + `database.scope`, we are subject to the no-process-in-inner-nodes rule. + The kernel will let us create a child cgroup, but it will fail when we try + to enable controllers in `database.scope`'s `cgroup.subtree_control` file. + There are a few different ways to solve this failure; the easiest is + probably to create a temporary cgroup under `database.scope` and move the + `libcgroup_systemd_idle_thread` to this temporary cgroup. This allows + `database.scope` to operate as a legal inner node, and we can then create + the entire hierarchy. + + 1. Temporarily disable the cpu and memory controllers at the scope level + ``` + sudo cgset -r cgroup.subtree_control="-cpu -memory" / + ``` + + Since we informed libcgroup that `mycompany.slice/database.scope` is the + default path, we can use `/`. Otherwise, we would have had to specify + the entire path. This pattern continues throughout this example. + + 1. Create a temporary cgroup and move the idle process + ``` + sudo cgcreate -g :tmp + sudo cgclassify -g :tmp $(cgget -nv -r cgroup.procs /) + ``` + + 1. Re-enable the cpu and memory controllers at the scope level + ``` + sudo cgset -r cgroup.subtree_control="+cpu +memory" / + ``` + + Now we can finally get back to creating our child cgroups + ``` + sudo cgcreate -g cpu,memory:high-priority -g cpu,memory:medium-priority -g cpu,memory:low-priority + ``` + +1. Configure the cgroups per the requirements + 1. The `high-priority` cgroup should be guaranteed at least 1GB of RAM + ``` + sudo cgset -r memory.low=1G high-priority + ``` + + 1. The `low-priority` cgroup should be hard-limited to 2GB of RAM + ``` + sudo cgset -r memory.max=2G low-priority + ``` + + 1. The `medium-prioirty` cgroup should have a soft memory limit of 3 GB + ``` + sudo cgset -r memory.high=3G medium-priority + ``` + + 1. The `high-priority` cgroup should be able to consume 60% of CPU cycles + ``` + sudo cgset -r cpu.weight=600 high-priority + ``` + + Note that I've (somewhat arbitrarily) chosen a total `cpu.weight` within + `database.scope` to be 1000. Thus, to meet the 60% requirement, we need + to allocate 600 shares to the `high-priority` cgroup. + + 1. The `medium-priority` cgroup should be able to consume 30% of CPU cycles + ``` + sudo cgset -r cpu.weight=300 medium-priority + ``` + + 1. The `low-priority` cgroup should be able to consume 10% of CPU cycles + ``` + sudo cgset -r cpu.weight=100 low-priority + ``` + +1. Start up the application + + - If the application is already running, then you can use `cgclassify` to + move the process(es) to the appropriate cgroups + + - To start a fresh application, it is recommended to use `cgexec` to place + the application in the desired cgroup + + - Finally, consider using `cgrulesengd` to automatically move processes to + the correct cgroups + +1. Clean up + + 1. Now that there are other processes running within the scope, we can + remove the `libcgroup_systemd_idle_thread` + ``` + $ for PID in $(cgget -nv -r cgroup.procs tmp); do sudo kill -9 $PID;done + ``` + + 1. Delete the `tmp` cgroup + ``` + sudo cgdelete -g :tmp + ``` + +1. Verify the cgroups were configured per the requirements + ``` + $ cgget -r cpu.weight -r memory.low -r memory.high -r memory.max high-priority medium-priority low-priority + high-priority: + cpu.weight: 600 + memory.low: 1073741824 + memory.high: max + memory.max: max + + medium-priority: + cpu.weight: 300 + memory.low: 0 + memory.high: 3221225472 + memory.max: max + + low-priority: + cpu.weight: 100 + memory.low: 0 + memory.high: max + memory.max: 2147483648 + ``` + +1. Summary + +This document outlines the steps for creating a delegated systemd scope and +configuring its child cgroups on a cgroup v2 system. Systemd and libcgroup +provide powerful tools to simplify these steps.