Intended audience: hackers working on userspace subsystems that require direct cgroup access, such as container managers and similar.
So you are wondering about resource management with systemd, you know Linux control groups (cgroups) a bit and are trying to integrate your software with what systemd has to offer there. Here’s a bit of documentation about the concepts and interfaces involved with this.
What’s described here has been part of systemd and documented since v205 times. However, it has been updated and improved substantially, even though the concepts stayed mostly the same. This is an attempt to provide more comprehensive up-to-date information about all this, particular in light of the poor implementations of the components interfacing with systemd of current container managers.
Before you read on, please make sure you read the low-level kernel documentation about the unified cgroup hierarchy. This document then adds in the higher-level view from systemd.
This document augments the existing documentation we already have:
These wiki documents are not as up to date as they should be, currently, but the basic concepts still fully apply. You should read them too, if you do something with cgroups and systemd, in particular as they shine more light on the various D-Bus APIs provided. (That said, sooner or later we should probably fold that wiki documentation into this very document, too.)
Much of the philosophy behind these concepts is based on a couple of basic design ideas of cgroup v2 (which we however try to adapt as far as we can to cgroup v1 too). Specifically two cgroup v2 rules are the most relevant:
The no-processes-in-inner-nodes rule: this means that it’s not permitted to have processes directly attached to a cgroup that also has child cgroups and vice versa. A cgroup is either an inner node or a leaf node of the tree, and if it’s an inner node it may not contain processes directly, and if it’s a leaf node then it may not have child cgroups. (Note that there are some minor exceptions to this rule, though. E.g. the root cgroup is special and allows both processes and children — which is used in particular to maintain kernel threads.)
The single-writer rule: this means that each cgroup only has a single writer, i.e. a single process managing it. It’s OK if different cgroups have different processes managing them. However, only a single process should own a specific cgroup, and when it does that ownership is exclusive, and nothing else should manipulate it at the same time. This rule ensures that various pieces of software don’t step on each other’s toes constantly.
These two rules have various effects. For example, one corollary of this is: if your container manager creates and manages cgroups in the system’s root cgroup you violate rule #2, as the root cgroup is managed by systemd and hence off limits to everybody else.
Note that rule #1 is generally enforced by the kernel if cgroup v2 is used: as soon as you add a process to a cgroup it is ensured the rule is not violated. On cgroup v1 this rule didn’t exist, and hence isn’t enforced, even though it’s a good thing to follow it then too. Rule #2 is not enforced on either cgroup v1 nor cgroup v2 (this is UNIX after all, in the general case root can do anything, modulo SELinux and friends), but if you ignore it you’ll be in constant pain as various pieces of software will fight over cgroup ownership.
Note that cgroup v1 is currently the most deployed implementation, even though it’s semantically broken in many ways, and in many cases doesn’t actually do what people think it does. cgroup v2 is where things are going, and most new kernel features in this area are only added to cgroup v2, and not cgroup v1 anymore. For example, cgroup v2 provides proper cgroup-empty notifications, has support for all kinds of per-cgroup BPF magic, supports secure delegation of cgroup trees to less privileged processes and so on, which all are not available on cgroup v1.
systemd supports three different modes how cgroups are set up. Specifically:
Unified — this is the simplest mode, and exposes a pure cgroup v2
logic. In this mode /sys/fs/cgroup is the only mounted cgroup API file system
and all available controllers are exclusively exposed through it.
Legacy — this is the traditional cgroup v1 mode. In this mode the
various controllers each get their own cgroup file system mounted to
/sys/fs/cgroup/<controller>/. On top of that systemd manages its own cgroup
hierarchy for managing purposes as /sys/fs/cgroup/systemd/.
Hybrid — this is a hybrid between the unified and legacy mode. It’s set
up mostly like legacy, except that there’s also an additional hierarchy
/sys/fs/cgroup/unified/ that contains the cgroup v2 hierarchy. (Note that in
this mode the unified hierarchy won’t have controllers attached, the
controllers are all mounted as separate hierarchies as in legacy mode,
i.e. /sys/fs/cgroup/unified/ is purely and exclusively about core cgroup v2
functionality and not about resource management.) In this mode compatibility
with cgroup v1 is retained while some cgroup v2 features are available
too. This mode is a stopgap. Don’t bother with this too much unless you have
too much free time.
To say this clearly, legacy and hybrid modes have no future. If you develop software today and don’t focus on the unified mode, then you are writing software for yesterday, not tomorrow. They are primarily supported for compatibility reasons and will not receive new features. Sorry.
Superficially, in legacy and hybrid modes it might appear that the parallel
cgroup hierarchies for each controller are orthogonal from each other. In
systemd they are not: the hierarchies of all controllers are always kept in
sync (at least mostly: sub-trees might be suppressed in certain hierarchies if
no controller usage is required for them). The fact that systemd keeps these
hierarchies in sync means that the legacy and hybrid hierarchies are
conceptually very close to the unified hierarchy. In particular this allows us
to talk of one specific cgroup and actually mean the same cgroup in all
available controller hierarchies. E.g. if we talk about the cgroup /foo/bar/
then we actually mean /sys/fs/cgroup/cpu/foo/bar/ as well as
/sys/fs/cgroup/memory/foo/bar/, /sys/fs/cgroup/pids/foo/bar/, and so on.
Note that in cgroup v2 the controller hierarchies aren’t orthogonal, hence
thinking about them as orthogonal won’t help you in the long run anyway.
If you wonder how to detect which of these three modes is currently used, use
statfs() on /sys/fs/cgroup/. If it reports CGROUP2_SUPER_MAGIC in its
.f_type field, then you are in unified mode. If it reports TMPFS_MAGIC then
you are either in legacy or hybrid mode. To distinguish these two cases, run
statfs() again on /sys/fs/cgroup/unified/. If that succeeds and reports
CGROUP2_SUPER_MAGIC you are in hybrid mode, otherwise not.
From a shell, you can check the Type in stat -f /sys/fs/cgroup and
stat -f /sys/fs/cgroup/unified.
The low-level kernel cgroups feature is exposed in systemd in three different “unit” types. Specifically:
💼 The .service unit type. This unit type is for units encapsulating
processes systemd itself starts. Units of these types have cgroups that are
the leaves of the cgroup tree the systemd instance manages (though possibly
they might contain a sub-tree of their own managed by something else, made
possible by the concept of delegation, see below). Service units are usually
instantiated based on a unit file on disk that describes the command line to
invoke and other properties of the service. However, service units may also
be declared and started programmatically at runtime through a D-Bus API
(which is called transient services).
👓 The .scope unit type. This is very similar to .service. The main
difference: the processes the units of this type encapsulate are forked off
by some unrelated manager process, and that manager asked systemd to expose
them as a unit. Unlike services, scopes can only be declared and started
programmatically, i.e. are always transient. That’s because they encapsulate
processes forked off by something else, i.e. existing runtime objects, and
hence cannot really be defined fully in ‘offline’ concepts such as unit
files.
🔪 The .slice unit type. Units of this type do not directly contain any
processes. Units of this type are the inner nodes of part of the cgroup tree
the systemd instance manages. Much like services, slices can be defined
either on disk with unit files or programmatically as transient units.
Slices expose the trunk and branches of a tree, and scopes and services are attached to those branches as leaves. The idea is that scopes and services can be moved around though, i.e. assigned to a different slice if needed.
The naming of slice units directly maps to the cgroup tree path. This is not
the case for service and scope units however. A slice named foo-bar-baz.slice
maps to a cgroup /foo.slice/foo-bar.slice/foo-bar-baz.slice/. A service
quux.service which is attached to the slice foo-bar-baz.slice maps to the
cgroup /foo.slice/foo-bar.slice/foo-bar-baz.slice/quux.service/.
By default systemd sets up four slice units:
-.slice is the root slice. i.e. the parent of everything else. On the host
system it maps directly to the top-level directory of cgroup v2.
system.slice is where system services are by default placed, unless
configured otherwise.
user.slice is where user sessions are placed. Each user gets a slice of
its own below that.
machines.slice is where VMs and containers are supposed to be
placed. systemd-nspawn makes use of this by default, and you’re very welcome
to place your containers and VMs there too if you hack on managers for those.
Users may define any amount of additional slices they like though, the four above are just the defaults.
Container managers and suchlike often want to control cgroups directly using the raw kernel APIs. That’s entirely fine and supported, as long as proper delegation is followed. Delegation is a concept we inherited from cgroup v2, but we expose it on cgroup v1 too. Delegation means that some parts of the cgroup tree may be managed by different managers than others. As long as it is clear which manager manages which part of the tree each one can do within its sub-graph of the tree whatever it wants.
Only sub-trees can be delegated (though whoever decides to request a sub-tree
can delegate sub-sub-trees further to somebody else if they like). Delegation
takes place at a specific cgroup: in systemd there’s a Delegate= property you
can set for a service or scope unit. If you do, it’s the cut-off point for
systemd’s cgroup management: the unit itself is managed by systemd, i.e. all
its attributes are managed exclusively by systemd, however your program may
create/remove sub-cgroups inside it freely, and those then become exclusive
property of your program, systemd won’t touch them — all attributes of those
sub-cgroups can be manipulated freely and exclusively by your program.
By turning on the Delegate= property for a scope or service you get a few
guarantees:
systemd won’t fiddle with your sub-tree of the cgroup tree anymore. It won’t change attributes of any cgroups below it, nor will it create or remove any cgroups thereunder, nor migrate processes across the boundaries of that sub-tree as it deems useful anymore.
If your service makes use of the User= functionality, then the sub-tree
will be chown()ed to the indicated user so that it can correctly create
cgroups below it. Note however that systemd will do that only in the unified
hierarchy (in unified and hybrid mode) as well as on systemd’s own private
hierarchy (in legacy and hybrid mode). It won’t pass ownership of the legacy
controller hierarchies. Delegation to less privileged processes is not safe
in cgroup v1 (as a limitation of the kernel), hence systemd won’t facilitate
access to it.
Any BPF IP filter programs systemd installs will be installed with
BPF_F_ALLOW_MULTI so that your program can install additional ones.
In unit files the Delegate= property is superficially exposed as
boolean. However, since v236 it optionally takes a list of controller names
instead. If so, delegation is requested for listed controllers
specifically. Note that this only encodes a request. Depending on various
parameters it might happen that your service actually will get fewer
controllers delegated (for example, because the controller is not available on
the current kernel or was turned off) or more. If no list is specified
(i.e. the property simply set to yes) then all available controllers are
delegated.
Let’s stress one thing: delegation is available on scope and service units only. It’s expressly not available on slice units. Why? Because slice units are our inner nodes of the cgroup trees and we freely attach services and scopes to them. If we’d allow delegation on slice units then this would mean that both systemd and your own manager would create/delete cgroups below the slice unit and that conflicts with the single-writer rule.
So, if you want to do your own raw cgroups kernel level access, then allocate a scope unit, or a service unit (or just use the service unit you already have for your service code), and turn on delegation for it.
The service manager sets the user.delegate extended attribute (readable via
getxattr(2) and related calls) to the character 1 on cgroup directories
where delegation is enabled (and removes it on those cgroups where it is
not). This may be used by service programs to determine whether a cgroup tree
was delegated to them. Note that this is only supported on kernels 5.6 and
newer in combination with systemd 251 and newer.
(OK, here’s one caveat: if you turn on delegation for a service, and that
service has ExecStartPost=, ExecReload=, ExecStop= or ExecStopPost=
set, then these commands will be executed within the .control/ sub-cgroup of
your service’s cgroup. This is necessary because by turning on delegation we
have to assume that the cgroup delegated to your service is now an inner
cgroup, which means that it may not directly contain any processes. Hence, if
your service has any of these four settings set, you must be prepared that a
.control/ subcgroup might appear, managed by the service manager. This also
means that your service code should have moved itself further down the cgroup
tree by the time it notifies the service manager about start-up readiness, so
that the service’s main cgroup is definitely an inner node by the time the
service manager might start ExecStartPost=. Starting with systemd 254 you may
also use DelegateSubgroup= to let the service manager put your initial
service process into a subgroup right away.)
(Also note, if you intend to use “threaded” cgroups — as added in Linux 4.14 —,
then you should do that two levels down from the main service cgroup your
turned delegation on for. Why that? You need one level so that systemd can
properly create the .control subgroup, as described above. But that one
cannot be threaded, since that would mean .control has to be threaded too —
this is a requirement of threaded cgroups: either a cgroup and all its siblings
are threaded or none –, but systemd expects it to be a regular cgroup. Thus you
have to nest a second cgroup beneath it which then can be threaded.)
Let’s say you write a container manager, and you wonder what to do regarding cgroups for it, as you want your manager to be able to run on systemd systems.
You basically have three options:
😊 The integration-is-good option. For this, you register each container
you have either as a systemd service (i.e. let systemd invoke the executor
binary for you) or a systemd scope (i.e. your manager executes the binary
directly, but then tells systemd about it. In this mode the administrator
can use the usual systemd resource management and reporting commands
individually on those containers. By turning on Delegate= for these scopes
or services you make it possible to run cgroup-enabled programs in your
containers, for example a nested systemd instance. This option has two
sub-options:
a. You transiently register the service or scope by directly contacting systemd via D-Bus. In this case systemd will just manage the unit for you and nothing else.
b. Instead you register the service or scope through systemd-machined
(also via D-Bus). This mini-daemon is basically just a proxy for the same
operations as in a. The main benefit of this: this way you let the system
know that what you are registering is a container, and this opens up
certain additional integration points. For example, journalctl -M can
then be used to directly look into any container’s journal logs (should
the container run systemd inside), or systemctl -M can be used to
directly invoke systemd operations inside the containers. Moreover tools
like “ps” can then show you to which container a process belongs (ps -eo
pid,comm,machine), and even gnome-system-monitor supports it.
🙁 The i-like-islands option. If all you care about is your own cgroup tree,
and you want to have to do as little as possible with systemd and no
interest in integration with the rest of the system, then this is a valid
option. For this all you have to do is turn on Delegate= for your main
manager daemon. Then figure out the cgroup systemd placed your daemon in:
you can now freely create sub-cgroups beneath it. Don’t forget the
no-processes-in-inner-nodes rule however: you have to move your main
daemon process out of that cgroup (and into a sub-cgroup) before you can
start further processes in any of your sub-cgroups.
🙁 The i-like-continents option. In this option you’d leave your manager
daemon where it is, and would not turn on delegation on its unit. However,
as you start your first managed process (a container, for example) you would
register a new scope unit with systemd, and that scope unit would have
Delegate= turned on, and it would contain the PID of this process; all
your managed processes subsequently created should also be moved into this
scope. From systemd’s PoV there’d be two units: your manager service and the
big scope that contains all your managed processes in one.
BTW: if for whatever reason you say “I hate D-Bus, I’ll never call any D-Bus
API, kthxbye”, then options #1 and #3 are not available, as they generally
involve talking to systemd from your program code, via D-Bus. You still have
option #2 in that case however, as you can simply set Delegate= in your
service’s unit file and you are done and have your own sub-tree. In fact, #2 is
the one option that allows you to completely ignore systemd’s existence: you
can entirely generically follow the single rule that you just use the cgroup
you are started in, and everything below it, whatever that might be. That said,
maybe if you dislike D-Bus and systemd that much, the better approach might be
to work on that, and widen your horizon a bit. You are welcome.
systemd supports a number of controllers (but not all). Specifically, supported are:
cpu, cpuacct, blkio, memory, devices, pidscpu, io, memory, pidsIt is our intention to natively support all cgroup v2 controllers as they are
added to the kernel. However, regarding cgroup v1: at this point we will not
add support for any other controllers anymore. This means systemd currently
does not and will never manage the following controllers on cgroup v1:
freezer, cpuset, net_cls, perf_event, net_prio, hugetlb. Why not?
Depending on the case, either their API semantics or implementations aren’t
really usable, or it’s very clear they have no future on cgroup v2, and we
won’t add new code for stuff that clearly has no future.
Effectively this means that all those mentioned cgroup v1 controllers are up
for grabs: systemd won’t manage them, and hence won’t delegate them to your
code (however, systemd will still mount their hierarchies, simply because it
mounts all controller hierarchies it finds available in the kernel). If you
decide to use them, then that’s fine, but systemd won’t help you with it (but
also not interfere with it). To be nice to other tenants it might be wise to
replicate the cgroup hierarchies of the other controllers in them too however,
but of course that’s between you and those other tenants, and systemd won’t
care. Replicating the cgroup hierarchies in those unsupported controllers would
mean replicating the full cgroup paths in them, and hence the prefixing
.slice components too, otherwise the hierarchies will start being orthogonal
after all, and that’s not really desirable. One more thing: systemd will clean
up after you in the hierarchies it manages: if your daemon goes down, its
cgroups will be removed too. You basically get the guarantee that you start
with a pristine cgroup sub-tree for your service or scope whenever it is
started. This is not the case however in the hierarchies systemd doesn’t
manage. This means that your programs should be ready to deal with left-over
cgroups in them — from previous runs, and be extra careful with them as they
might still carry settings that might not be valid anymore.
Note a particular asymmetry here: if your systemd version doesn’t support a specific controller on cgroup v1 you can still make use of it for delegation, by directly fiddling with its hierarchy and replicating the cgroup tree there as necessary (as suggested above). However, on cgroup v2 this is different: separately mounted hierarchies are not available, and delegation has always to happen through systemd itself. This means: when you update your kernel and it adds a new, so far unseen controller, and you want to use it for delegation, then you also need to update systemd to a version that groks it.
systemd can happily run as a container payload’s PID 1. Note that systemd
unconditionally needs write access to the cgroup tree however, hence you need
to delegate a sub-tree to it. Note that there’s nothing too special you have to
do beyond that: just invoke systemd as PID 1 inside the root of the delegated
cgroup sub-tree, and it will figure out the rest: it will determine the cgroup
it is running in and take possession of it. It won’t interfere with any cgroup
outside of the sub-tree it was invoked in. Use of CLONE_NEWCGROUP is hence
optional (but of course wise).
Note one particular asymmetry here though: systemd will try to take possession of the root cgroup you pass to it in full, i.e. it will not only create/remove child cgroups below it, it will also attempt to manage the attributes of it. OTOH as mentioned above, when delegating a cgroup tree to somebody else it only passes the rights to create/remove sub-cgroups, but will insist on managing the delegated cgroup tree’s top-level attributes. Or in other words: systemd is greedy when accepting delegated cgroup trees and also greedy when delegating them to others: it insists on managing attributes on the specific cgroup in both cases. A container manager that is itself a payload of a host systemd which wants to run a systemd as its own container payload instead hence needs to insert an extra level in the hierarchy in between, so that the systemd on the host and the one in the container won’t fight for the attributes. That said, you likely should do that anyway, due to the no-processes-in-inner-cgroups rule, see below.
When systemd runs as container payload it will make use of all hierarchies it
has write access to. For legacy mode you need to make at least
/sys/fs/cgroup/systemd/ available, all other hierarchies are optional. For
hybrid mode you need to add /sys/fs/cgroup/unified/. Finally, for fully
unified you (of course, I guess) need to provide only /sys/fs/cgroup/ itself.
⚡ If you go for implementation option 1a or 1b (as in the list above), then
each of your containers will have its own systemd-managed unit and hence
cgroup with possibly further sub-cgroups below. Typically the first process
running in that unit will be some kind of executor program, which will in
turn fork off the payload processes of the container. In this case don’t
forget that there are two levels of delegation involved: first, systemd
delegates a group sub-tree to your executor. And then your executor should
delegate a sub-tree further down to the container payload. Oh, and because
of the no-process-in-inner-nodes rule, your executor needs to migrate itself
to a sub-cgroup of the cgroup it got delegated, too. Most likely you hence
want a two-pronged approach: below the cgroup you got started in, you want
one cgroup maybe called supervisor/ where your manager runs in and then
for each container a sibling cgroup of that maybe called payload-xyz/.
⚡ Don’t forget that the cgroups you create have to have names that are
suitable as UNIX file names, and that they live in the same namespace as the
various kernel attribute files. Hence, when you want to allow the user
arbitrary naming, you might need to escape some of the names (for example,
you really don’t want to create a cgroup named tasks, just because the
user created a container by that name, because tasks after all is a magic
attribute in cgroup v1, and your mkdir() will hence fail with EEXIST. In
systemd we do escaping by prefixing names that might collide with a kernel
attribute name with an underscore. You might want to do the same, but this
is really up to you how you do it. Just do it, and be careful.
🚫 Never create your own cgroups below arbitrary cgroups systemd manages, i.e
cgroups you haven’t set Delegate= in. Specifically: 🔥 don’t create your
own cgroups below the root cgroup 🔥. That’s owned by systemd, and you will
step on systemd’s toes if you ignore that, and systemd will step on
yours. Get your own delegated sub-tree, you may create as many cgroups there
as you like. Seriously, if you create cgroups directly in the cgroup root,
then all you do is ask for trouble.
🚫 Don’t attempt to set Delegate= in slice units, and in particular not in
-.slice. It’s not supported, and will generate an error.
🚫 Never write to any of the attributes of a cgroup systemd created for you. It’s systemd’s private property. You are welcome to manipulate the attributes of cgroups you created in your own delegated sub-tree, but the cgroup tree of systemd itself is out of limits for you. It’s fine to read from any attribute you like however. That’s totally OK and welcome.
🚫 When not using CLONE_NEWCGROUP when delegating a sub-tree to a
container payload running systemd, then don’t get the idea that you can bind
mount only a sub-tree of the host’s cgroup tree into the container. Part of
the cgroup API is that /proc/$PID/cgroup reports the cgroup path of every
process, and hence any path below /sys/fs/cgroup/ needs to match what
/proc/$PID/cgroup of the payload processes reports. What you can do safely
however, is mount the upper parts of the cgroup tree read-only (or even
replace the middle bits with an intermediary tmpfs — but be careful not to
break the statfs() detection logic discussed above), as long as the path
to the delegated sub-tree remains accessible as-is.
⚡ Currently, the algorithm for mapping between slice/scope/service unit
naming and their cgroup paths is not considered public API of systemd, and
may change in future versions. This means: it’s best to avoid implementing a
local logic of translating cgroup paths to slice/scope/service names in your
program, or vice versa — it’s likely going to break sooner or later. Use the
appropriate D-Bus API calls for that instead, so that systemd translates
this for you. (Specifically: each Unit object has a ControlGroup property
to get the cgroup for a unit. The method GetUnitByControlGroup() may be
used to get the unit for a cgroup.)
⚡ Think twice before delegating cgroup v1 controllers to less privileged containers. It’s not safe, you basically allow your containers to freeze the system with that and worse. Delegation is a strongpoint of cgroup v2 though, and there it’s safe to treat delegation boundaries as privilege boundaries.
And that’s it for now. If you have further questions, refer to the systemd mailing list.
— Berlin, 2018-04-20