doc: Add housekeeping documentation

Signed-off-by: Frederic Weisbecker <frederic@kernel.org>
Acked-by: Waiman Long <longman@redhat.com>
Cc: Marco Crivellari <marco.crivellari@suse.com>
Cc: Michal Hocko <mhocko@suse.com>
Cc: Peter Zijlstra <peterz@infradead.org>
Cc: Tejun Heo <tj@kernel.org>
Cc: Thomas Gleixner <tglx@linutronix.de>
Cc: Waiman Long <longman@redhat.com>

diff --git a/Documentation/core-api/housekeeping.rst b/Documentation/core-api/housekeeping.rst
new file mode 100644 (file)
index 0000000..e541730
--- /dev/null
+++ b/Documentation/core-api/housekeeping.rst
@@ -0,0 +1,111 @@
+======================================
+Housekeeping
+======================================
+
+
+CPU Isolation moves away kernel work that may otherwise run on any CPU.
+The purpose of its related features is to reduce the OS jitter that some
+extreme workloads can't stand, such as in some DPDK usecases.
+
+The kernel work moved away by CPU isolation is commonly described as
+"housekeeping" because it includes ground work that performs cleanups,
+statistics maintainance and actions relying on them, memory release,
+various deferrals etc...
+
+Sometimes housekeeping is just some unbound work (unbound workqueues,
+unbound timers, ...) that gets easily assigned to non-isolated CPUs.
+But sometimes housekeeping is tied to a specific CPU and requires
+elaborated tricks to be offloaded to non-isolated CPUs (RCU_NOCB, remote
+scheduler tick, etc...).
+
+Thus, a housekeeping CPU can be considered as the reverse of an isolated
+CPU. It is simply a CPU that can execute housekeeping work. There must
+always be at least one online housekeeping CPU at any time. The CPUs that
+are not        isolated are automatically assigned as housekeeping.
+
+Housekeeping is currently divided in four features described
+by the ``enum hk_type type``:
+
+1.     HK_TYPE_DOMAIN matches the work moved away by scheduler domain
+       isolation performed through ``isolcpus=domain`` boot parameter or
+       isolated cpuset partitions in cgroup v2. This includes scheduler
+       load balancing, unbound workqueues and timers.
+
+2.     HK_TYPE_KERNEL_NOISE matches the work moved away by tick isolation
+       performed through ``nohz_full=`` or ``isolcpus=nohz`` boot
+       parameters. This includes remote scheduler tick, vmstat and lockup
+       watchdog.
+
+3.     HK_TYPE_MANAGED_IRQ matches the IRQ handlers moved away by managed
+       IRQ isolation performed through ``isolcpus=managed_irq``.
+
+4.     HK_TYPE_DOMAIN_BOOT matches the work moved away by scheduler domain
+       isolation performed through ``isolcpus=domain`` only. It is similar
+       to HK_TYPE_DOMAIN except it ignores the isolation performed by
+       cpusets.
+
+
+Housekeeping cpumasks
+=================================
+
+Housekeeping cpumasks include the CPUs that can execute the work moved
+away by the matching isolation feature. These cpumasks are returned by
+the following function::
+
+       const struct cpumask *housekeeping_cpumask(enum hk_type type)
+
+By default, if neither ``nohz_full=``, nor ``isolcpus``, nor cpuset's
+isolated partitions are used, which covers most usecases, this function
+returns the cpu_possible_mask.
+
+Otherwise the function returns the cpumask complement of the isolation
+feature. For example:
+
+With isolcpus=domain,7 the following will return a mask with all possible
+CPUs except 7::
+
+       housekeeping_cpumask(HK_TYPE_DOMAIN)
+
+Similarly with nohz_full=5,6 the following will return a mask with all
+possible CPUs except 5,6::
+
+       housekeeping_cpumask(HK_TYPE_KERNEL_NOISE)
+
+
+Synchronization against cpusets
+=================================
+
+Cpuset can modify the HK_TYPE_DOMAIN housekeeping cpumask while creating,
+modifying or deleting an isolated partition.
+
+The users of HK_TYPE_DOMAIN cpumask must then make sure to synchronize
+properly against cpuset in order to make sure that:
+
+1.     The cpumask snapshot stays coherent.
+
+2.     No housekeeping work is queued on a newly made isolated CPU.
+
+3.     Pending housekeeping work that was queued to a non isolated
+       CPU which just turned isolated through cpuset must be flushed
+       before the related created/modified isolated partition is made
+       available to userspace.
+
+This synchronization is maintained by an RCU based scheme. The cpuset update
+side waits for an RCU grace period after updating the HK_TYPE_DOMAIN
+cpumask and before flushing pending works. On the read side, care must be
+taken to gather the housekeeping target election and the work enqueue within
+the same RCU read side critical section.
+
+A typical layout example would look like this on the update side
+(``housekeeping_update()``)::
+
+       rcu_assign_pointer(housekeeping_cpumasks[type], trial);
+       synchronize_rcu();
+       flush_workqueue(example_workqueue);
+
+And then on the read side::
+
+       rcu_read_lock();
+       cpu = housekeeping_any_cpu(HK_TYPE_DOMAIN);
+       queue_work_on(cpu, example_workqueue, work);
+       rcu_read_unlock();

diff --git a/Documentation/core-api/index.rst b/Documentation/core-api/index.rst
index 5eb0fbbbc323c380138cb53a4b263ad90b61ca91..79fe7735692ead8c73e59fa4345b77d2abeb12e2 100644 (file)
--- a/Documentation/core-api/index.rst
+++ b/Documentation/core-api/index.rst
@@ -25,6 +25,7 @@ it.
    symbol-namespaces
    asm-annotations
    real-time/index
+   housekeeping.rst
 
 Data structures and low-level utilities
 =======================================