Skip to content
Faber Faber

Cgroups and resource accounting

A cgroup (control group) is a node in a kernel tree of processes. Each node can cap the memory, process count and CPU time of everything beneath it, and counts what they use. Faber uses cgroup v2 for every per-task limit except CPU time and for every statistic in a result. Resource limits lists the limits and the outcomes they produce; this page shows how they are built.

The hierarchy

Section titled “The hierarchy”

Faber manages the cgroup it was started in, read from /proc/self/cgroup. With Docker’s --cgroupns=host and the host’s /sys/fs/cgroup mounted, that is the container’s own cgroup, and nothing outside it is touched. Faber refuses to start if the path it reads does not contain its own process, which is what a private cgroup namespace with the host’s tree mounted looks like.

  • <container cgroup> +cpu +memory +pids For example /sys/fs/cgroup/system.slice/docker-<id>.scope.
    • faber-init The service itself. Faber moves the container's processes here on first start: a cgroup that enables controllers for its children cannot hold processes.
    • faber memory.max (MEMORY_MAX + allowance) × slotsmemory.swap.max 0pids.max (PIDS_MAX + 3) × slots Every sandbox together.
      • req-<pid>-<16 random> memory.max MEMORY_MAX × width + allowancepids.max (PIDS_MAX + 2) × width + 1 One per running request. Its cgroup.kill ends exactly this request.
        • supervisor memory.max allowance The jailer, the supervisors and the inits, with oom_score_adj -999.
        • task-<pid>-<16 random> memory.max MEMORY_MAXmemory.swap.max 0pids.max PIDS_MAXcpu.max CPU_MAX One per running task, removed as soon as it ends.
        • task-… The other tasks of a parallel step.
      • req-… Other running requests.
  • Created by Docker
  • Faber's processes, or limits over them
  • A task's process tree
Faber enables the cpu, memory and pids controllers for the children of the container cgroup, of faber and of every req-* cgroup. Only task cgroups set cpu.max.

How the limits are sized

Section titled “How the limits are sized”

Each request needs memory for more than its tasks. Faber calls the rest the supervisor allowance:

allowance = 64 MiB + 2 × min(REQUEST_OUTPUT_LIMIT_BYTES, 1 GiB) + 288 MiB
  • 64 MiB for the jailer’s and the supervisors’ own memory;
  • twice the request’s output budget, capped at 1 GiB, because the output is collected and then serialized;
  • 288 MiB for /faber and /tmp: 2 × 128 MiB of data plus 32768 inodes at 1 KiB each.

The workspace is part of it because tmpfs pages are charged to the task that wrote them and move to the request cgroup when that task’s cgroup is removed. With the defaults the allowance is 384 MiB, and:

Cgroupmemory.maxpids.maxOne task per stepA step of 8 tasks
task-*MEMORY_MAXPIDS_MAX256 MiB, 64256 MiB, 64
supervisorallowancethe request’s384 MiB384 MiB
req-*MEMORY_MAX × width + allowance(PIDS_MAX + 2) × width + 1640 MiB, 672432 MiB, 529
faber(MEMORY_MAX + allowance) × MAX_CONCURRENCY(PIDS_MAX + 3) × MAX_CONCURRENCY6400 MiB, 670the same

The width is the number of tasks in the request’s widest step. The + 2 per task is its supervisor and init, the + 1 per request its jailer, and the service-wide + 3 per slot is all three, since every request holds at least one slot.

These numbers fit together. A request holds as many slots as its width, so the request cgroups of all running requests add up to no more than the faber cgroup’s limit, and a request’s own limit covers each of its tasks at its full MEMORY_MAX at once. A running request should therefore never be stopped by the faber limit; the container’s or host’s memory limit can still be lower, which is what sizing is about.

Task slots

Section titled “Task slots”

MAX_CONCURRENCY is a semaphore of task slots in the API server. A request takes as many as its widest step has tasks, all at once or not at all, and keeps them until its controller returns, which is after its sandbox is gone.

  • Held by a request
  • Free
  • Arriving request
Faber never queues. A request that does not fit is told so at once, with Retry-After, and can be retried here or sent to another instance.

A client that disconnects does not give its slots back early: the permit lives on the controller’s thread, which returns only once the sandbox has been killed and removed. A request whose every step is a cache hit takes no slots at all.

Telling OOM kills apart

Section titled “Telling OOM kills apart”

When a cgroup reaches its memory.max, the kernel’s OOM killer picks a victim among the processes beneath it (when the host runs out, among all of them). Within a request, the supervisor processes have oom_score_adj -999 and the tasks 0, so a task goes first. After the task has exited, its supervisor reads two counters of the task cgroup:

  • oom_kill in memory.events: how many of the task’s processes the OOM killer killed, for whatever reason.
  • oom in memory.events.local: how often the task cgroup’s own memory.max was reached. (Kernels without the .local file give the same count in memory.events; a task cgroup has no children, so the two agree.)
  • Outside Faber
  • Faber and the kernel (trusted)
  • Submitted code (untrusted)
out_of_memory is the workload’s fault and expected for hostile code. A rising count of ancestor_out_of_memory means the instance has more slots than memory: lower MAX_CONCURRENCY or raise the container’s limit.

Both counts must be non-zero for out_of_memory; oom_kill alone gives ancestor_out_of_memory. A timeout or an output-limit kill takes precedence over both; outcomes has the full order.

Processes

Section titled “Processes”

pids.max counts every process and thread of the task’s tree. When it is reached, fork and clone fail with EAGAIN and the kernel counts the refusal in pids.events (max). That count becomes pids_limit_hit_count, and the outcome pids_limit unless something earlier in the order applies. A program may handle the failed fork and exit normally, so check exit_code too.

Faber does not set RLIMIT_NPROC: an equal limit would refuse the fork before the cgroup does, and the result would no longer say pids_limit.

CPU and wall time

Section titled “CPU and wall time”

Three limits, enforced by three different mechanisms:

LimitMechanismWhat happensOutcome
CPU_MAX (50000 100000: 50 ms in every 100 ms)cpu.max of the task cgroupThe whole tree is throttled for the rest of the period. Counted in cpu_nr_throttled and cpu_throttled_usecnone, the task just runs slower
CPU_TIME_LIMIT_SECS (5 s, whole seconds)RLIMIT_CPU of each processThe kernel signals the process that used it upsignaled
WALL_TIMEOUT_MS (5 s, cut to the request’s remaining time)The supervisor’s clock, checked every 10 mscgroup.kill on the task cgroup; after 500 ms the direct child is killed outright if it is still theretimed_out

With the defaults a task cannot use more than 2.5 s of CPU before its 5 s wall timeout, so RLIMIT_CPU only matters when CPU_MAX or WALL_TIMEOUT_MS is raised. RLIMIT_RTPRIO and RLIMIT_NICE are 0: real-time scheduling, which cpu.max does not throttle, is refused.

What the statistics are

Section titled “What the statistics are”

The supervisor reads these files of the task cgroup after the task has exited and before it removes the cgroup:

Result fieldSource
memory_peak_bytesmemory.peak
cpu_usage_usec, cpu_nr_throttled, cpu_throttled_useccpu.stat: usage_usec, nr_throttled, throttled_usec
pids_peakpids.peak
oom_kill_countmemory.events: oom_kill
pids_limit_hit_countpids.events: max
cleanup_succeededWhether the task cgroup could be removed

execution_time_ms is the supervisor’s own clock, from before the task’s cgroup is created to after it is removed.

Where this lives in the code

Section titled “Where this lives in the code”